You are on page 1of 83

AlliedWareTM OS How To | Configure Microsoft Windows 2003 Network Load Balancing Clustering with Allied Telesis Switches

Introduction
This is a guide to Network Load Balancing (NLB) clustering options with Allied Telesis managed layer 3 devices. NLB is one of the clustering technologies available from Microsoft. It provides high availability for services such as HTTP and FTP, by grouping identical servers into a cluster and sharing the network load between all currently-available servers in the cluster. This How To Note uses scenarios that have been tested in the lab and verified to work. It also deals exclusively with nodes that have dual network cards on the cluster side for bandwidth redundancy and/or aggregation. This document describes best-practice implementations of NLB clustering on Allied Telesis switches. You should read through this document and decide which setup best suits your requirements. Please pay attention to the details, because some traps in setting these scenarios up can cause unexpected results.

C613-16073-00 REV B

www.alliedtelesis.com

Introduction

What information will you find in this document?
This document is divided into the following sections:
z z

"Network Load Balancing clustering modes" on page 3 "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6 "Note about capturing traffic on the clustered servers" on page 7 "How to quickly set up Windows 2003 NLB clustering" on page 7 "Switch Fault Tolerance, two switches and VRRP" on page 8 "Switch Fault Tolerance with a SwitchBlade" on page 17 "Adaptive Load Balancing, teamed NICs and a SwitchBlade" on page 19 "Adaptive Load Balancing, two switches and VRRP" on page 22 "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29 "Static Link Aggregation with a SwitchBlade" on page 34 "How Allied Telesis recommends NLB be implemented at layer 3" on page 40 "Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster" on page 43 "Appendix 2: IGMP snooping port membership information in switch memory tables" on page 81

z z z z z z z z z z z

Which product and software version does this information apply to?
The specified solutions only apply to AT-9800 series and SwitchBlade 4000 series, except for the solution in "How Allied Telesis recommends NLB be implemented at layer 3" on page 40. In that solution, the layer 2 switches can be any Allied Telesis switch and the layer 3 device can be any AR400 or AR700 router. It applies to AlliedWare versions since 2.7.5. The switch or router implementation varies slightly with the software version—see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6.

Page 2 | AlliedWare™ OS How To Note: NLB Clustering

Network Load Balancing clustering modes

Network Load Balancing clustering modes
There are two modes for Network Load Balancing with Windows 2003: unicast and multicast. Multicast mode has a further option of IGMP Multicasting. This section discusses how these modes work. Note that clustering only works, in multicast or unicast mode, if all packets sent to the cluster's IP address are sent to all nodes in the cluster. This means that the cluster-side switch must flood traffic to all ports. To reduce traffic congestion, the cluster should ideally be put in its own VLAN. This recommendation applies regardless of which NLB cluster mode you use. This document focuses on non-IGMP multicast network load balancing in a layer 3 environment. However, IGMP multicasting appears to be an excellent option for layer 2 networks, because traffic to the servers is not seen on other ports unless they send an IGMP report for that group. Traffic to the servers can be protected from prying eyes by using IGMP filtering and static IGMP entries (see How To Configure IGMP for Multicasting on Routers and Managed Layer 3 Switches from www.alliedtelesis.com/resources/literature/howto.aspx). In a layer 3 environment you can be less concerned about the security of traffic because your servers should be in their own VLAN, i.e. not in the same segment as clients. In the section "How Allied Telesis recommends NLB be implemented at layer 3" on page 40, we use multicast mode with IGMP selected.

Unicast mode
In unicast mode, all hosts in the cluster share a single unicast “cluster” MAC address, to go with the cluster IP address. This mode has the disadvantage that it stops cluster hosts from using their own “burned-in” MAC addresses, so hosts cannot contact each other unless you install a second NIC card on each host and use that for intra-cluster communication. The cluster operation forces the switch to flood all packets that are destined for the cluster, by stopping the switch from learning the cluster MAC address. The switch cannot learn the cluster MAC address because it never appears in the source field of the Ethernet header. Instead, each NIC uses a special unicast MAC address in the source field of the Ethernet header. The cluster MAC address must never be found in the source field of the Ethernet header, because otherwise the switch will learn the cluster MAC address, stop flooding packets to all ports, and only one node in the cluster will receive traffic for the cluster's IP. In the two-node cluster that this document deals with, both nodes will use the same cluster MAC address when answering ARP requests for the cluster's address, which is 172.16.0.127.

Page 3 | AlliedWare™ OS How To Note: NLB Clustering

Network Load Balancing clustering modes

Multicast mode
In multicast mode, hosts in the cluster use their real “burned-in” MAC address in the source field of the Ethernet header. However, they answer ARP requests for the cluster IP address with a multicast MAC in the ARP packet's payload, while the Ethernet header on the ARP reply still has the real MAC address. This allows cluster hosts to contact each other. The following tcpdump output shows the multicast MAC:
14:02:57.814560 arp who-has 172.16.0.127 (Broadcast) tell 172.16.0.1 14:02:57.814910 arp reply 172.16.0.127 is-at 03:bf:ac:10:00:7f

Because the ARP reply is a multicast MAC address, Allied Telesis switches will by default ignore it, so that MAC address is never found in the software IP ARP table. The solution to this depends on the software version—see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6. Because this document deals with dual NICs on each node, multicast is often the only NLB mode that these scenarios will work with. The reason is due to a limitation in the network card drivers, which in many of the setups will simply not recognise the unicast MAC address in the destination field of the Ethernet header. In other words the NICs will not process packets destined to the virtual MAC address despite it being the same source MAC they are sending packets from. In both multicast and unicast mode the mechanism to cause flooding is as such: ARP requests for the cluster IP are answered with the virtual cluster MAC address in the ARP packet's payload. The node must send all packets, including ARP requests and replies, with a different source MAC in the Ethernet header. This prevents the switch from entering the virtual cluster MAC into the forwarding database (FDB). The end result is that traffic destined for the virtual MAC will be flooded to all ports in the VLAN associated with the cluster IP. Note: According to Microsoft if clustering is working properly, one node cannot ping another node in the same cluster. This explains why cluster traffic (like MS NLB heartbeats) generated by each node and teaming packets (like Intel ANS Probes) generated by the NIC driver are broadcasted.

Page 4 | AlliedWare™ OS How To Note: NLB Clustering

.... default (1) Fast Leave . instead traffic only goes to NLB ports....0... The cluster will answer ARP requests for the cluster IP address with a multicast MAC in the ARP packet's payload. As far as an Allied Telesis switch is concerned. The switch's IGMP snooping table (the show igmpsnooping command) reveals this information: IGMP Snooping -----------------------------------------------------------------------Status . No group memberships........... 239...... Group.. All IP traffic destined for 172... The cluster node responds with its burned in MAC in the Ethernet source field.66 16:07:03.0....127 is then sent with an ethernet destination of 01:00:5e:7f:00:7f..16....255....0.....Network Load Balancing clustering modes Multicast mode with the IGMP option selected In multicast mode with the IGMP option selected. This mode uses IGMP to prevent the switch from flooding all ports... Enabled Disabled All-groups ports .. This mode also causes the cluster to use its real “burned in” MAC address in the source field of the ethernet header.... None Vlan Name (vlan id) .... Page 5 | AlliedWare™ OS How To Note: NLB Clustering .127 is-at 01:00:5e:7f:00:7f The Windows client will happily add this IP / MAC combination to its internal ARP table (on a Windows or Linux PC.. which still identifies it as an IP-multicast MAC address: 16:07:03.. Off Group List ... the servers send IGMP reports for a group that corresponds to the MAC address they are putting in their ARP responses. cluster hosts can also contact each other.. the clustered servers are seen as IGMP clients and the IGMP multicast data (stream) is sent by the workstations trying to access the cluster.. Microsoft's IGMP multicasting mode cleverly turns the concept of IGMP clients and hosts on its head....6 Group. while the Ethernet header on the ARP reply still has the real MAC address..16.. To ensure that the switches do correctly forward the multicast data.254 Ports 1......643330 arp reply 172..16.. Off Group List .....16. vlan2 (2) Fast Leave .....127 tell 172....0.255...643160 arp who-has 172.. use “arp -a” to see all ARP entries). Entry timeout 234 secs Entry timeout 235 secs You can also use the memory tables for information about port membership—see "Appendix 2: IGMP snooping port membership information in switch memory tables" on page 81.6 Vlan Name (vlan id) . 239.... Note though.. Reports are sent frequently by the NLB servers so there is no concern about IGMP entries timing out on the switch..127 Ports 1... that the MAC address is slightly different to the previously discussed multicast mode—the MAC address starts with the bytes 01:00:5e....255.0..

you need to enter at least one extra command to get multicast mode to work.6 and 276-01.Accepting ARP entries that have a multicast MAC address and a unicast IP address Accepting ARP entries that have a multicast MAC address and a unicast IP address Allied Telesis switches ignore multicast MAC addresses by default. you have to also specify a port number. you need to enter the command enable ip macdisparity. the layer 3 switches have multiple ports connected to the cluster in all configurations except the solution in "How Allied Telesis recommends NLB be implemented at layer 3" on page 40. using the command add ip arp. you need to enter the multicast MAC as a static entry. If the Allied Telesis device is layer 3 switching between the clients and the servers.8.7. Summary The following table shows these differences. depending on the software version. You do not need to enter the multicast MAC as a static entry. There are three sets of behaviour: z with the earliest versions. In this How To Note. This means that you cannot use other switches with configurations that have multiple ports connected to the server cluster. SwitchBlade and AT-9800 switches ignore the port number and flood the entry to the VLAN.8. using the command add ip arp with later 2.1 Yes Yes No No Yes Yes No No No No Yes Yes Yes Yes No Yes Yes Yes No No No Yes Yes No No No No No Yes Yes enable ip mscluster enable ip macdisparity Page 6 | AlliedWare™ OS How To Note: NLB Clustering .6 276-01 ≥ 276-02 ≥ 2. you need to enter the command enable ip mscluster and then enter the multicast MAC as a static entry. This means that you can use SwitchBlade and AT-9800 switches in all configurations in this How To Note.6 versions and with version 2. z z Ports in ARP entries When you add a static ARP entry for a VLAN.1 and later.7. Other switches do not flood the entry to the VLAN. You can specify any port in the static ARP entry—the number is irrelevant.7. Commands required: add ip arp AT -9800 series: sb-275 to sb275-04 sb-276 sb276-01 sb276-02 ≥ sb-281 SwitchBlade: ≥ sb-275a Other layer 3 devices: 2. with all sb275a versions and with versions 2.

However. The following sections include the switch configurations for a number of NLB options: z z z z z z z "Switch Fault Tolerance. but also teamed network cards. when debugging a cluster. teamed NICs and a SwitchBlade" on page 19 "Adaptive Load Balancing. two switches and VRRP" on page 8 "Switch Fault Tolerance with a SwitchBlade" on page 17 "Adaptive Load Balancing. So. ping would succeed when Ethereal was capturing traffic on a node. For that reason. and produce erroneous results. The appendix will get you started and after you have had some success you must do more research into cluster policies and design to make your cluster operate efficiently. or Windows cluster software. Page 7 | AlliedWare™ OS How To Note: NLB Clustering . be advised that this implementation is not guaranteed to offer the best security or design for your organisation. The ideas are not overly complex but the mechanisms behind it can take time to understand. we conclude that packet sniffers conflict with the network interface driver. For example. inter-cluster node (server-to-server) pings should not work. two switches and VRRP" on page 22 "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29 "Static Link Aggregation with a SwitchBlade" on page 34 "How Allied Telesis recommends NLB be implemented at layer 3" on page 40 This is followed by "Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster" on page 43. How to quickly set up Windows 2003 NLB clustering Setting up a Windows 2003 Network Load Balancing cluster requires considerable time for research. The results cannot be trusted. you should not do packet sniffing on the servers themselves. In our setup.Note about capturing traffic on the clustered servers Note about capturing traffic on the clustered servers The setups in this How To Note not only use network load balancing.

0/24 AT-9816GB-2 Clumpy2 PRIVATE INTERFACE clumpy1 .0.0/24 10.16. if one switch fails.0. At any given time.16.0.42 02-02-ac-10-00-7f virtual-172.Switch Fault Tolerance.0.0.0.16.127 02-bf-ac-10-00-7f VRRP on AT -9816GBs VLAN100-172.0.0/24 Cluster 172.0.41 02-01-ac-10-00-7f clumpy2 .127/24 Page 8 | AlliedWare™ OS How To Note: NLB Clustering . The AT-9924SP also serves as a media adapter so that the UTP NIC can connect to the fibre AT-9816GB.0/24 Cluster 172.0/24 Tagged VLAN100 & VLAN200 10.168.0.0.0.0/24 L2 Switch Workstation L2 Switch Private 192. two switches and VRRP Switch Fault Tolerance (SFT) provides a failover relationship between two adapters when each adapter is connected to a separate switch. Network Diagram VLAN100 VLAN200 Private 192.16.0.0. one adapter is alive while the other is in standby mode.168.192.42 CLUSTER INTERFACE clumpy1 .254/24 VLAN200 .172.172.0.16.16.10. Using SFT.0.0/24 AT-9816GB-1 Cluster 172. then the two switches will maintain node and cluster connectivity without network downtime if each server is attached to both switches.16. Equipment z z z z z z Intel Pro/1000 MT Dual Port NIC Linux PC as the workstation 2 x 32 bit PCs with Windows 2003 Enterprise Edition AT-9924SP as L2 switch connected to workstation 2 x AT-9816GB AT-8748SL as L2 switch on private cluster LAN If only two servers are being used in the cluster then the AT-8748SL may be left out and the nodes connected directly to each other with a crossover cable.0.254/24 clumpy1 to port 1 on AT-9816GB-1 port 1on AT-9816GB-2 clumpy2 to port 9 on AT-9816GB-1 port 9 on AT-9816GB-2 WORKSTATION 10.16.0/24 10.168.41 clumpy2 . two switches and VRRP Switch Fault Tolerance.192.0/24 Clumpy1 Cluster 172.0.0.0.168.

16.0.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged delete vlan="1" port=2 # # IP configuration # enable ip add ip int=vlan100 ip=172. Unicast VRRP switch configurations AT -9800 #1 # # SYSTEM configuration # set system name="AT-9816GB-1" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.255. We cannot guarantee that unicast will work at all.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.0.0.0 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172. we will give a configuration with NLB in unicast mode.1 mask=255.0.255.1 mask=255.Switch Fault Tolerance.254 add vrrp=200 monitoredinterface=vlan200 Page 9 | AlliedWare™ OS How To Note: NLB Clustering .16.255.0 add ip int=vlan200 ip=10. Note that multicast mode is favoured over unicast mode. because of Windows 2003 and NIC driver variance.255.0. two switches and VRRP Unicast Mode First.0. This configuration uses 2 server clusters with 1 private and 2 teamed NICs.

0.16.0 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.0.0 add ip int=vlan200 ip=10. two switches and VRRP AT -9800 #2 # # SYSTEM configuration # set system name="AT-9816GB-2" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.0.0.2 mask=255.16.0.2 mask=255.255.0.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged # # IP configuration # enable ip add ip int=vlan100 ip=172.255.255.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.255.Switch Fault Tolerance.254 add vrrp=200 monitoredinterface=vlan200 AT -9924SP # # SYSTEM configuration # set system name="AT-9924SP" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=1-8 create vlan="vlan200" vid=200 add vlan="200" port=9-12 create vlan="vlan300" vid=300 add vlan="300" port=13-16 create vlan="vlan400" vid=400 add vlan="400" port=17-20 create vlan="vlan500" vid=500 add vlan="500" port=21-24 Page 10 | AlliedWare™ OS How To Note: NLB Clustering .

0 00-00-cd-09-73-50 0 1 200 0 0 0 1 The clustered IP's MAC has been learnt and the AT-9816GB will make routing decisions to 172. 02-01 tells us that it is a clustering MAC address for the node with priority 1.0.0.0.16.0. if we look at the switch's FDB: Manager top> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 8 --------------------------------------------------------------------------VLAN MAC Address Port Status -------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-00-5e-00-01-64 CPU static 200 00-00-5e-00-01-c8 CPU static 1 dynamic 100 02-01-ac-10-00-7f 9 dynamic 100 02-02-ac-10-00-7f 200 00-50-fc-ee-f5-13 3 dynamic The table above illustrates that the MAC address 02-bf-ac-10-00-7f does not appear in the switch chip's table.16. two switches and VRRP Table Entries Here is an example of the entries that should appear in the switch FDB and ARP tables when unicast mode is used with NLB clustering.0. Manager top> sh swi tab=ip CXE IP route table: Index Seg Network Nexthop ToProc VRP VID Hits PV Prio Perm ----------------------------------------------------------------------------0 100 0 0 0 1 0 101 172.255 00-00-cd-09-73-50 0 1 200 0 0 0 1 8 110 172.0.1 00-00-cd-09-73-50 0 0 100 0 0 0 1 5 102 172.0.127 02-bf-ac-10-00-7f 0 0 100 0 0 0 1 0 100 0 0 0 1 3 101 172. Page 11 | AlliedWare™ OS How To Note: NLB Clustering .41 02-bf-ac-10-00-7f 0 4 102 172. however.16.16. 02-02 tells us that it is a clustering MAC address for the node with priority 2.0.0.0 00-00-cd-09-73-50 0 0 100 0 0 0 1 9 110 10.0.16. Notice that the two highlighted MAC entries are very similar but not the same as the cluster's MAC address.41/42/127 based on the MAC address 02-bf-ac-10-00-7f.Switch Fault Tolerance.16.1 00-00-cd-09-73-50 0 1 200 0 0 0 1 7 102 10.42 02-bf-ac-10-00-7f 0 1 101 10.0.255 00-00-cd-09-73-50 0 0 100 0 0 0 1 6 102 10.16.0. so a packet with a destination MAC address of 02-bf-ac-10-00-7f must be flooded out all ports in the VLAN.0.0.127 00-50-fc-ee-f5-13 0 1 200 0 0 0 1 2 101 172.0.

In the following configurations. as shown by the underlined text above. For the most recent software versions. Page 12 | AlliedWare™ OS How To Note: NLB Clustering .16.0. you can see the specific multicast address to use in the NLB Manager (see "Set up the NLB Cluster" on page 66). you need to use the command enable ip macdisparity. If you need a static ARP entry.0 add ip arp=172.0.1 mask=255. Multicast VRRP switch configurations AT -9800 #1 # # SYSTEM configuration # set system name="AT-9816GB-1" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.255.254 add vrrp=200 monitoredinterface=vlan200 Note: The MAC address is now slightly different. For more recent software versions.0.0.0 add ip int=vlan200 ip=10. two switches and VRRP Multicast Mode This is the same setup but this time with NLB clustering in multicast mode.16.0.255. you need to also use the command enable ip mscluster. see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6. but do not need the static ARP entry.Switch Fault Tolerance.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.1 mask=255. we have added a static ARP entry.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged delete vlan="1" port=2 # # IP configuration # enable ip add ip int=vlan100 ip=172.0.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.255. For details.0.16.255.

16.254 add vrrp=200 monitoredinterface=vlan200 AT -9924SP # # SYSTEM configuration # set system name="AT-9924SP" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=1-8 create vlan="vlan200" vid=200 add vlan="200" port=9-12 create vlan="vlan300" vid=300 add vlan="300" port=13-16 create vlan="vlan400" vid=400 add vlan="400" port=17-20 create vlan="vlan500" vid=500 add vlan="500" port=21-24 Page 13 | AlliedWare™ OS How To Note: NLB Clustering .0.255.0.2 mask=255. two switches and VRRP AT -9800 #2 # SYSTEM configuration # set system name="AT-9816GB-2" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.0 add ip int=vlan200 ip=10.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1# # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.Switch Fault Tolerance.16.0 add ip arp=172.255.0.0.0.255.0.255.0.16.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged # IP configuration # enable ip add ip int=vlan100 ip=172.2 mask=255.

Manager AT-9816GB-1> sh swi fdb Switch Forwarding Database (hardware) ----------------------------------------------------------------------------Total Number of Entries = 7 -------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-00-5e-00-01-64 CPU static 200 00-00-5e-00-01-c8 CPU static 100 00-04-23-b5-bb-6f 1 dynamic 100 00-04-23-ab-37-02 2 dynamic Manager AT-9816GB-1> sh ip arp Interface IP Address Physical Address ARP Type Status ----------------------------------------------------------------------------vlan200(3) 10.255. Note that in this case the individual server IPs (172.0.127.16.0.16. This has been statically added.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(1) 172.Switch Fault Tolerance.255.0. there is an entry in the software table for the cluster: IP 172.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(1) 172.127 03-bf-ac-10-00-7f vlan100 172.255 ff-ff-ff-ff-ff-ff Other active vlan100 255.16.41-42) appear in the ARP table with different MACs and these MACs also appear in the FDB.127 00-50-fc-ee-f5-13 Dynamic active vlan200 10.0.16.0.16.0. There is no entry for the MAC 03-bf-ac-10-00-7f in the forwarding database so traffic will be flooded to all ports in the VLAN that is attached to 172.0.255 ff-ff-ff-ff-ff-ff Other active Page 14 | AlliedWare™ OS How To Note: NLB Clustering .41 00-04-23-b5-bb-6f Dynamic active vlan100(2) 172.0.16.0.0/24.0.16. two switches and VRRP Table Entries The following tables are the contents of the switch tables for this scenario.

221659 IP 10.16.520693 IP <nop.16.nop.0.40195 > 172.40195: S 3107871426:3107871426(0) ack 593168139 win 65535 <mss 1460.42) as seen by clumpy1 (172.127.0.Switch Fault Tolerance.219277 IP 10.40195:. This is not the fault of the switch.0.221777 IP 172.0.40194 > 148937179 0> 10.timestamp 15:52:56.0.0.0. Both of these captures are taken from the perspective of clumpy1.16.nop.0.40194 > 148937179 230102> 172.40195 > 172.127.0.127.16.timestamp 148937549 0.40195 > 172.0.http: P 0:273(273) ack 1 win 5840 172.0.0.16. both servers will respond to pings to the cluster address 172. Interestingly.520006 IP <nop. ack 275 win 65262 <nop.The active port that clumpy1 was connected to on the switch was mirrored and the mirror port was then sniffed by a linux server with tcpdump version 3.127.timestamp 148937549 0> 15:53:00. The first capture shows the packet flow when clumpy1 answers the HTTP request (in accordance with the defined clustering rules). the following traffic flows should apply.127 and served by clumpy1 (172.timestamp 148937549 0> 15:53:00.220717 IP 172.nop.0.127.127.0.519534 IP <nop.nop.127.http: F 274:274(0) ack 394 win 6432 <nop.0.wscale 0> 15:53:00.0.127.16.0.timestamp 15:52:56.sackOK.0.8.0.41): 15:52:56.16. so only packets from the workstation can be seen.127.0.521737 IP <nop.3) Page 15 | AlliedWare™ OS How To Note: NLB Clustering .0.http > 10.127 to 172.nop. two switches and VRRP Traffic Flows Regardless of whether NLB is in unicast or multicast mode.127.http: .0.127.40194 > 148937179 0> 10.nop.127. HTTP balancing works correctly.16.http: SWE 593168138:593168138(0) win 5840 <mss 1460.0.127 to 172.127 but served by clumpy2 (172.nop.41) as seen by clumpy 1: 15:53:00.timestamp 229906 148937549> Packet Capture 2 An identical HTTP request from 10.16.40194 > 148937179 230102> 10.http: F 273:273(0) ack 583 win 6972 1.nop.timestamp 15:52:56. ack 1481239454 win 5840 172.127.http > 10.127.0.16.0.16.0.wscale 0.127. The second capture is also taken from the perspective of clumpy1 but this time the HTTP request is serviced by clumpy2.127. HTTP load balancing at the packet level follows1.127.0.220206 IP 10. ack 583 win 6972 172.219723 IP 10.0.http > 10.nop.40195: FP 1:393(392) ack 274 win 65262 <nop.0.sackOK> 15:53:00.16.127.0.0.timestamp 148937549 229906> 15:53:00. Packet Capture 1 HTTP request from 10.0.16.40195 > 172.0.http: .16.0.3 (libpcap version 0.nop.http: P 1:274(273) ack 1 win 5840 <nop.0.0. ack 1 win 5840 <nop.16.0.nop.0.127.127.0.0.nop.0.nop.127.0.timestamp 229906 148937549> 15:53:00.0.127.nop.219650 IP 172.16.timestamp 10.16.nop.127.timestamp 0 0.http: .8.

there is no solution on offer in this document for such a scenario. Probes can be used to determine link failures however. If there is a failure further downstream but the link at the NIC end does not go down then the NIC will not change state because it is not aware of any topology change. there appears to be no advantage to enabling probes. While the combined bandwidth of the NICs is never utilised. Summary This setup is an excellent redundancy option. two switches and VRRP NIC Failover Should the node's link go down on the NIC driver will rapidly change the state of the standby NIC to active. in our lab setup with Intel ANS Probes. When operating correctly the node should experience only a few seconds of lost connectivity. Page 16 | AlliedWare™ OS How To Note: NLB Clustering . however.Switch Fault Tolerance. So in this setup. in the configuration here they will not help. In fact. This is not a solution for adapting to failures that occur further down the network. the NIC driver disabled the wrong NIC when a downstream link was broken. failover is fast if the link to the active NIC goes down.

Switch Fault Tolerance with a SwitchBlade Switch Fault Tolerance with a SwitchBlade Switch Fault Tolerance (SFT) provides a failover relationship between two adaptors when each adapter is connected to a separate switch.2.VLAN100 clumpy1-port4. At any given time.port3.16. one adaptor is alive while the other is in standby mode.1 .0.172.16.42 virtual .127 Clumpy2 Page 17 | AlliedWare™ OS How To Note: NLB Clustering .172. 5. Using SFT and one SwitchBlade will permit you to hot swap line cards without network downtime if each server is attached to two different line cards.41 clumpy2-port4.VLAN200 vlan200 .0.172.0.1. Equipment This example uses: z z Intel Pro/1000 MT Dual Port NIC SwitchBlade chassis (AT-SB4108-00) • Switch Control Card (AT-SB4211) • 2 x 8 port UTP trispeed line card (AT-SB44118-100 (RJ45)) z 2 x 32-bit PCs running Windows 2003 Enterprise Edition Network Diagram Clumpy1 PC .0.16. 5.0.3-10.127 CLUSTER .2 .

255.1 Description As in all the other cases where NLB multicast mode is used.254 mask=255.Switch Fault Tolerance with a SwitchBlade Configuration Set the Windows 2003 NLB cluster servers to use multicast mode. set system name="Switchblade" create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 add vlan="100" port=4.3 enable add ip add ip enable add ip ip int=vlan100 ip=172.0.0 ip mscluster arp=172. appear in the IP ARP table.16. and to use the command enable ip mscluster. This setup will give you full redundancy (both at the cluster and node level) when you hot swap a SwitchBlade line card. there should never be an entry in the forwarding database for the multicast MAC. Summary This setup is an excellent redundancy option.0.0 int=vlan200 ip=10.1-4. Remember to add a static ARP for the multicast MAC. there is no solution on offer in this document for such a scenario. however.2 add vlan="200" port=4. of course. This is not a solution for adapting to failures that occur further down the network. While the combined bandwidth of the NICs is never utilised.255.5. Page 18 | AlliedWare™ OS How To Note: NLB Clustering .127 int=vlan100 eth=03-bf-ac-10-00-7f port=4.254 mask=255. The static ARP entry will.0. failover is fast if the link to the active NIC goes down.16.255.1-5.2.0.255.

5-3. ALB is a less efficient but easy-to-implement solution. If you want to provide load balancing with dual NICs in your cluster.16.127 CLUSTER .6 .0.172. teamed NICs and a SwitchBlade Adaptive Load Balancing.127 Clumpy2 Page 19 | AlliedWare™ OS How To Note: NLB Clustering .port3.172.Adaptive Load Balancing.0.3-10. ALB provides load balancing and adapter fault tolerance.172. which is discussed in "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29.1-3.VLAN100 clumpy1-port3. teamed NICs and a SwitchBlade This section describes Adaptive Load Balancing (ALB).2 . Equipment This example uses: z z z z z Intel Pro/1000 MT Dual Port NIC SwitchBlade chassis (AT-SB4108-00) Switch Control Card (AT-SB4211) 8 port UTP trispeed line card (AT-SB4411 8-1000T(RJ45)) 2 x 32-bit PCs running Windows 2003 Enterprise Edition Network diagram Clumpy1 PC .41 clumpy2-port3.16. The NIC driver should analyse the send and receive load on each adapter and balance the rate across them based on the NIC vendor's algorithm/rule set (for example Intel base it on the destination address).42 virtual . then you may instead use LACP. which load balances at the server level and is discussed in "Static Link Aggregation with a SwitchBlade" on page 34.0. Another alternative is Static Link Aggregation.VLAN200 vlan200 .16.0.0.

127 to all ports in the VLAN that belongs to the 172.wscale 0> 15:59:00.127.42721 > <nop. teamed NICs and a SwitchBlade Configuration Set the Windows 2003 NLB cluster servers to use multicast mode.255.127.sackOK.16. flags [DF].42721: S [tcp sum ok] 1420104149:1420104149(0) ack 3360891802 win 16384 <mss 1460.42721 > <nop.wscale 0> 15:59:00. length: 52) .timestamp 244021652 0.0.nop.Adaptive Load Balancing.16.16.0 ip mscluster arp=172.0. flags [DF]. id 23053.0.127 int=vlan100 eth=03-bf-ac-10-00-7f port=3.42721 > 172. flags [none].0. offset 0.0.16.3.16.timestamp 15:59:00.0. 172.127.0. This setup is therefore less efficient than LACP or Static Link Aggregation.wscale 0. which would only have the SwitchBlade send one packet to one port in each team/trunk.255.0.http: SWE [tcp sum ok] 3360891801:3360891801(0) win 5840 <mss 1460. ttl 63.nop.0.3 enable add ip add ip enable add ip ip int=vlan100 ip=172.16.http: 244021652 0> id 23054.1-3.16.6 add vlan="200" port=3. sackOK> 15:59:00.255. offset 0.nop. Notice that only traffic from 10.1 15:59:00. ttl 128.timestamp 244021652 0.0. length:64) 172. length: 60) 10.0.0.timestamp 0 0.16.0/24 subnet.16.16. set system name="Switchblade" create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 add vlan="100" port=3.timestamp (tos 0x0. Description The SwitchBlade will send any traffic destined for 172.sackOK.0.127.127.0.127.42721 > 172.254 mask=255.127.609752 IP (tos 0x0. offset 0.610002 IP 10. id 17555.0.0. length: 60) 10.127 is duplicated (marked in red).2. [tcp sum ok] 1:1(0) ack 1 win 5840 <continued on next page> Page 20 | AlliedWare™ OS How To Note: NLB Clustering .0.0.http: 244021652 0> (tos 0x0.0.127.0. ttl 63.nop. ttl 63.609757 IP (tos 0x0. and to use the command enable ip mscluster.0. turn off probes and Receive Load Balancing (RLB) on your NIC team. offset 0. flags [DF]. length: 52) . id 23053.0.610006 IP 10.1 For better performance and a less chatty network. flags [DF].5-3.nop. Remember to add a static ARP for the multicast MAC. including both teamed NICs on both servers.0.0. ttl 63.nop. [tcp sum ok] 1:1(0) ack 1 win 5840 id 23054.127. Traffic from 172.0 int=vlan200 ip=10.254 mask=255. 172.nop.609936 IP (tos 0x0.http > 10. Packet Captures The following packet capture illustrates the duplicate packets that are sent to the node.127 (in this case clumpy1) is seen only once (marked in blue).nop. offset 0.255.127.nop.http: SWE [tcp sum ok] 3360891801:3360891801(0) win 5840 <mss 1460.

172. 1. ttl 128.127.timestamp (tos 0x0.0.0. offset 0. flags [DF].127.42721 > <nop.127.0. offset 0.3). this ruins all the effort that the clustering process had previously put into hiding the cluster MAC from the switch and will stop the switch from flooding packets that are destined for the cluster MAC. length: 52) 172.127. teamed NICs and a SwitchBlade <continued from previous page> 15:59:00. [tcp sum ok] 397:397(0) ack 275 win 65262 <nop.612316 IP 10.0. This is easy to implement but not as efficient as link aggregation.42721 > <nop.0.http: 244021652 843939> (tos 0x0.0. offset 0.16.0. length: 52) F [tcp sum ok] 274:274(0) ack 397 win 6432 15:59:00. Page 21 | AlliedWare™ OS How To Note: NLB Clustering .nop.http: 244021652 843939> id 23056. It will not be tolerant to failures downstream but the cluster should recover quickly if one or all ports fail on a node. Unfortunately.nop.3 (libpcap version 0.42721: .http > 10. ttl 63. id 17557. Summary This solution provides NIC redundancy and transmit-aggregation.nop.127. length: 52) F [tcp sum ok] 274:274(0) ack 397 win 6432 id 23056.Adaptive Load Balancing.8.16.8.0. The switch chip learns the cluster MAC thanks to an MS-Load balancing packet that has the Ethernet source MAC field set to the cluster MAC (this should not happen). 172.0.The two ALB ports that clumpy1 was connected to on the switch were mirrored and the mirror port was then sniffed by a linux server with tcpdump version 3. ttl 63.timestamp 843939 244021652> Unicast Mode Setting your cluster to use unicast with ALB will lead to connection timeouts. flags [DF]. flags [DF].612321 IP 10.timestamp 15:59:00.612535 IP (tos 0x0.127.16.

16.0/24 AT-9816GB-2 Clumpy2 PRIVATE INTERFACE clumpy1 . ALB is a less efficient but easy to implement solution.0/24 AT-9816GB-1 Cluster 172.0/24 10.0.16.254/24 clumpy1 to port 1 on AT-9816GB-1 port 1on AT-9816GB-2 clumpy2 to port 9 on AT-9816GB-1 port 9 on AT-9816GB-2 WORKSTATION 10.0.16.0.0/24 Clumpy1 Cluster 172.168.172.192.0/24 Tagged VLAN100 & VLAN200 10. then you may instead use LACP.0. Equipment z z z z z z Intel Pro/1000 MT Dual Port NIC Linux PC as the workstation 2 x 32 bit PCs with Windows 2003 Enterprise Edition AT-9924SP as L2 switch connected to workstation 2 x AT-9816GB AT-8748SL as L2 switch on private cluster LAN If only two servers are being used in the cluster.0.168.0.127 02-bf-ac-10-00-7f VRRP on AT -9816GBs VLAN100-172. then the AT-8748SL may be left out and the nodes connected directly to each other with a crossover cable.0.0.41 02-01-ac-10-00-7f clumpy2 .16.0/24 Cluster 172.Adaptive Load Balancing. The AT-9924SP also serves as a media adapter so that the UTP NIC can connect to the fibre AT-9816GB. two switches and VRRP Adaptive Load Balancing.0. The NIC driver should analyse the send and receive load on each adapter and balance the rate across them based on the NIC vendor's algorithm/rule set (for example Intel base it on the destination address).0.0.0.16.41 clumpy2 .192.0.42 CLUSTER INTERFACE clumpy1 .16.0.0.168.16.0/24 Cluster 172.42 02-02-ac-10-00-7f virtual-172.0.16. two switches and VRRP This section describes Adaptive Load Balancing (ALB). ALB provides load balancing and adapter fault tolerance. Network diagram VLAN100 VLAN200 Private 192.168.0/24 L2 Switch Workstation L2 Switch Private 192.172.0/24 10. If you want to provide load balancing with dual NICs in your cluster. which is discussed in "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29.0.0.254/24 VLAN200 . Another alternative is Static Link Aggregation.10.127/24 Page 22 | AlliedWare™ OS How To Note: NLB Clustering . which load balances at the server level and is discussed in "Static Link Aggregation with a SwitchBlade" on page 34.0.

but do not need the static ARP entry. AT -9800 #1 # # SYSTEM configuration # set system name="AT-9816GB-1" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.16.0. For more recent software versions. we have added a static ARP entry to both switches.255. we will only use multicast clustering here.0.254 add vrrp=200 monitoredinterface=vlan200 Page 23 | AlliedWare™ OS How To Note: NLB Clustering . you need to also use the command enable ip mscluster.0. see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6.16. In the following configurations.Adaptive Load Balancing.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172. For details. For the most recent software versions.0. two switches and VRRP Configuration Again.255. you can see the specific multicast address to use in the NLB Manager (see "Set up the NLB Cluster" on page 66).0.0 add ip int=vlan200 ip=10.255.1 mask=255.16. you need to use the command enable ip macdisparity.0.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10. If you need a static ARP entry.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged delete vlan="1" port=2 # # IP configuration # enable ip add ip int=vlan100 ip=172.1 mask=255.255.0 add ip arp=172.0.

0.0.Adaptive Load Balancing.16.0.2 mask=255.2 mask=255.0.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.255. turn off probes and Receive Load Balancing (RLB) on your NIC team.255.0 add ip arp=172.255.255.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged # # IP configuration # enable ip add ip int=vlan100 ip=172.0 add ip int=vlan200 ip=10.16.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.16. Page 24 | AlliedWare™ OS How To Note: NLB Clustering .0.0.0.254 add vrrp=200 monitoredinterface=vlan200 AT -9924SP # # SYSTEM configuration # set system name="AT-9924SP" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=1-8 create vlan="vlan200" vid=200 add vlan="200" port=9-12 create vlan="vlan300" vid=300 add vlan="300" port=13-16 create vlan="vlan400" vid=400 add vlan="400" port=17-20 create vlan="vlan500" vid=500 add vlan="500" port=21-24 Tip: For better performance and a less chatty network. two switches and VRRP AT -9800 #2 # # SYSTEM configuration # set system name="AT-9816GB-2" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.

0.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.16.0.255. This behaviour will once again defeat the way clustering should work because the cluster IP ARP entry has a MAC address found in the forwarding database.255.127 00-50-fc-ee-f5-13 Dynamic active vlan200 10.255 ff-ff-ff-ff-ff-ff Other active vlan100(2) 172.0.41 00-04-23-b5-bb-6f Dynamic active vlan100(9) 172. This is avoided by entering a static IP ARP entry.16.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------- Example output continues on the next page.16.16.255 ff-ff-ff-ff-ff-ff Other active vlan200 255. two switches and VRRP Description In ALB mode.127 03-bf-ac-10-00-7f vlan100 172.0. one of the NICs will answer an ARP request for the virtual IP address with its burned-in MAC.42 00-04-23-ab-37-02 Dynamic active vlan100(2) 172.0. This is wrong: Manager AT-9816GB-1> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200(3) 10. Page 25 | AlliedWare™ OS How To Note: NLB Clustering .255 ff-ff-ff-ff-ff-ff Other active vlan100(9) 172.16.0.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------Manager AT-9816GB-1> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 8 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-04-23-b5-bb-6f 2 dynamic 100 00-04-23-ab-37-02 9 dynamic 100 00-04-23-ab-37-03 2 dynamic 100 00-04-23-b5-bb-6e 1 dynamic 200 00-50-fc-ee-f5-13 3 dynamic This is right: Manager AT-9816GB-1> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200 10.16.0.127 00-04-23-b5-bb-6f Dynamic active vlan100 172.255.16.0.0.0.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(1) 172.Adaptive Load Balancing.0.0.255.

42 00-04-23-ab-37-02 Dynamic active Static active vlan100(1) 172.0.0.16.0.0.16.255. Remember that we added the static IP ARP entry to port 1 on both switches.0.127 00-50-fc-ee-f5-13 Dynamic active vlan200(2) 10.255.16.127 03-bf-ac-10-00-7f vlan100(2) 172. One port on each node's NIC is connected to port 1 of a switch.16.255 ff-ff-ff-ff-ff-ff Other active vlan200 255. two switches and VRRP Manager AT-9816GB-1> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 10 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-00-5e-00-01-64 CPU static 200 00-00-5e-00-01-c8 CPU static 100 00-04-23-b5-bb-6f 2 dynamic 100 00-04-23-ab-37-02 9 dynamic 100 00-04-23-ab-37-03 2 dynamic 100 00-04-23-b5-bb-6e 1 dynamic 200 00-50-fc-ee-f5-13 3 dynamic Manager AT-9816GB-2> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200(2) 10.0.0.Adaptive Load Balancing.0.0.254 00-00-5e-00-01-64 Dynamic active vlan100 172.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------Manager AT-9816GB-2> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 11 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-03-00-e8 CPU static 100 00-00-cd-03-00-e8 CPU static 200 00-00-cd-03-00-e8 CPU static 100 00-04-23-b5-bb-6f 1 dynamic 100 00-04-23-ab-37-02 2 dynamic 100 00-04-23-ab-37-03 9 dynamic 100 00-04-23-b5-bb-6e 2 dynamic 100 00-00-cd-09-73-50 2 dynamic 100 00-00-5e-00-01-64 2 dynamic 200 00-00-5e-00-01-c8 2 dynamic 200 00-50-fc-ee-f5-13 2 dynamic At this point we can observe an interesting phenomenon.16.0. The second port on each node's NIC is attached to port 9 on the other Page 26 | AlliedWare™ OS How To Note: NLB Clustering .0.41 00-04-23-b5-bb-6f Dynamic active vlan100(2) 172.255 ff-ff-ff-ff-ff-ff Other active vlan100(1) 172.254 00-00-5e-00-01-c8 Dynamic active vlan200 10.

127 time delay 0 ms Echo reply 5 from 172.127 To understand the reason for this.127 ether=03-bf-ac-10-00-7f int=vlan100 port=9 Info (1005275): ARP successfully added. Manager AT-9816GB-2> add ip arp=172.127 Request 4 timed-out: No reply from 172.127 Echo reply 1 from 172.16.0.0.0.127 time delay 0 ms Echo reply 5 from 172.127 time delay 0 ms Echo reply 2 from 172. one of the switches now cannot ping the virtual server address: Manager AT-9816GB-1> ping 172.127 time delay 0 ms Echo reply 4 from 172.127 Request 5 timed-out: No reply from 172.127 time delay 0 ms Echo reply 3 from 172.127 time delay 0 ms Echo reply 2 from 172.0. examine the following fix: Manager AT-9816GB-2> del ip arp=172.Adaptive Load Balancing.0.0. two switches and VRRP switch.16.16.16.16.0.0. In the following example.0.0.0.16.127 Request 3 timed-out: No reply from 172.0.127 time delay 0 ms Echo reply 4 from 172.0.127 Request 1 timed-out: No reply from 172.0.0.127 time delay 0 ms Echo reply 3 from 172.0.16.16.127 Echo reply 1 from 172.16.16.0.16.16.16.0.16.0.127 time delay 0 ms Manager AT-9816GB-2> ping 172.16.127 Info (1005272): ARP entry successfully deleted.16.16.127 Request 2 timed-out: No reply from 172.0.127 time delay 0 ms Example output continues on the next page: Page 27 | AlliedWare™ OS How To Note: NLB Clustering .16.16.16. Manager AT-9816GB-2> ping 172.

Adaptive Load Balancing, two switches and VRRP

Manager AT-9816GB-2> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200 10.0.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(2) 172.16.0.41 00-04-23-b5-bb-6f Dynamic active vlan100(9) 172.16.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(9) 172.16.0.127 03-bf-ac-10-00-7f vlan100 172.16.0.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.255.255.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------Manager AT-9816GB-2> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 7 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-03-00-e8 CPU static 100 00-00-cd-03-00-e8 CPU static 200 00-00-cd-03-00-e8 CPU static 100 00-04-23-b5-bb-6f 2 dynamic 100 00-04-23-ab-37-02 9 dynamic 100 00-00-5e-00-01-64 2 dynamic 200 00-00-5e-00-01-c8 2 dynamic

The static IP ARP entry has now been set to port 9 instead of port 1. The next thing to understand is that an IP ARP entry will be used by the switch when it does software lookups. The switch consults its IP ARP table when it pings an address and in the first instance sends pings to port 1 only. From this information and our observation in the example above, we conclude that changing the static IP ARP port worked because under ALB one of the NIC ports is not listened to by the NIC driver for some arbitrary rule at the NIC driver level. ALB should be able to assume that traffic is flooded to all ports and hence the behaviour is not necessarily incorrect. Data that is switched through the switch (as opposed to being sent from the switch CPU) will be flooded. The port lookup for packets going through the switch is done by consulting the FDB, which does not have an entry for the cluster MAC address.

Summary
This solution provides NIC redundancy and transmit-aggregation. It will not be tolerant to failures downstream but the cluster should recover quickly if one or all ports fail on a node.

Page 28 | AlliedWare™ OS How To Note: NLB Clustering

Link Access/Aggregation Control Protocol and a SwitchBlade

Link Access/Aggregation Control Protocol and a SwitchBlade
The following configuration demonstrates how to increase your NLB bandwidth by teaming two NICs. We will use Link Aggregation Control Protocol (LACP) to maximise this bandwidth. Equipment
z z z z z

Intel Pro/1000 MT Dual Port NIC SwitchBlade (AT-SB4108-00) Switch Control Card (AT-SB4211) 24 port trispeed line card (AT-SB4412) 2 x 32-bit PCs running Windows 2003 Enterprise Edition LACP requires a “K revision” line card in the SwitchBlade. Any control card should be okay.

Note:

Network diagram

Clumpy1 PC - VLAN200 vlan200 - port3.3-10.0.0.127

CLUSTER - VLAN100 clumpy1-port 3.1-3.2 - 172.16.0.41 clumpy2-port 3.9-3.10 - 172.16.0.42 virtual - 172.16.0.127 Clumpy2

Note:

The card in the SwitchBlade in the diagram above shows 32 ports rather than the 24 that our setup actually uses.

Page 29 | AlliedWare™ OS How To Note: NLB Clustering

Link Access/Aggregation Control Protocol and a SwitchBlade

Configuration
First, set up LACP on your SwitchBlade. At the hardware level you will need a “K revision” line card, such as a 24 port trispeed card. Set the Windows 2003 NLB cluster servers to use multicast mode. Remember to add a static ARP for the multicast MAC, and to use the command enable ip mscluster.
set system name="Switchblade" create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 add vlan="100" port=3.1-3.2,3.9-3.10 add vlan="200" port=3.3 enable add ip add ip enable add ip ip int=vlan100 ip=172.16.0.254 mask=255.255.255.0 int=vlan200 ip=10.0.0.254 mask=255.255.255.0 ip mscluster arp=172.16.0.127 int=vlan100 eth=03-bf-ac-10-00-7f port=3.1

ena lacp delete lacp port=3.3-3.8,3.11-3.24 set lacp port=3.1-3.2 adminkey=100 set lacp port=3.9-3.10 adminkey=200

Your clustered servers' NICs must support Link Aggregation (802.3ad) mode. This will either be a part of the driver set already or you will need to download the latest drivers from whoever your NIC vendor is. Set up link aggregation on the NICs last. Note: It may take a few moments after Windows starts up for NLB to begin.

Description
When your setup is working you should see results similar to the following:

Manager Switchblade> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200(3.3) 10.0.0.127 00-50-fc-ee-f5-13 Dynamic active vlan200 10.0.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(3.2) 172.16.0.41 00-04-23-b5-bb-6e Dynamic active vlan100(3.10) 172.16.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(3.1) 172.16.0.127 03-bf-ac-10-00-7f vlan100 172.16.0.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.255.255.255 ff-ff-ff-ff-ff-ff Other active

Page 30 | AlliedWare™ OS How To Note: NLB Clustering

Link Access/Aggregation Control Protocol and a SwitchBlade

Manager Switchblade> sh swi fdb inst=3.0 Switch Forwarding Database (hardware) Blade Switch Instance: 3.0, (AT-SB4412 24-10/100/1000T(RJ45)) --------------------------------------------------------------------------Total Number of Entries = 8 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-05-01-14 CPU static 100 00-00-cd-05-01-14 CPU static 200 00-00-cd-05-01-14 CPU static 100 00-04-23-ab-37-02 3.10 dynamic 100 00-04-23-b5-bb-6e 3.2 dynamic 100 00-04-23-b5-bb-6f 3.2 dynamic 100 00-04-23-ab-37-03 3.10 dynamic 200 00-50-fc-ee-f5-13 3.3 dynamic Manager Switchblade> sh swi tab=ip inst=3.0 Blade Switch Instance: 3.0, (AT-SB4412 24-10/100/1000T(RJ45)) CXE IP route table summary information: Total number of entries = 18 ---------------------------------------------------------------------------CXE IP route table: Index Seg Network Nexthop ToProc VRP VID Hits PV Prio Perm ---------------------------------------------------------------------------0 101 192.168.255.1 00-00-cd-05-00-00 0 0 4094 0 0 0 1 1 101 172.16.0.127 03-bf-ac-10-00-7f 0 6 100 0 0 0 1 2 101 172.16.0.42 00-04-23-ab-37-02 0 6 100 0 0 0 1 3 101 10.0.0.127 00-50-fc-ee-f5-13 0 5 200 0 0 0 1 4 101 172.16.0.41 00-04-23-b5-bb-6e 0 6 100 0 0 0 1 5 102 192.168.255.12 00-00-cd-05-00-58 0 0 4094 0 0 0 1 6 102 192.168.255.127 00-00-cd-05-00-58 0 0 4094 0 0 0 1 7 102 192.168.255.140 00-00-cd-05-00-58 0 1 4093 0 0 0 1 8 102 192.168.255.255 00-00-cd-05-00-58 0 1 4093 0 0 0 1 9 102 10.0.0.254 00-00-cd-05-01-14 0 5 200 0 0 0 1 10 102 10.0.0.255 00-00-cd-05-01-14 0 5 200 0 0 0 1 11 102 172.16.0.254 00-00-cd-05-01-14 0 6 100 0 0 0 1 12 102 172.16.0.255 00-00-cd-05-01-14 0 6 100 0 0 0 1 13 109 192.168.255.0 00-00-cd-05-00-58 0 0 4094 0 0 0 1 14 109 192.168.255.128 00-00-cd-05-00-58 0 1 4093 0 0 0 1 15 110 10.0.0.0 00-00-cd-05-01-14 0 5 200 0 0 0 1 16 110 172.16.0.0 00-00-cd-05-01-14 0 6 100 0 0 0 1 17 134 0.0.0.0 00-00-cd-05-00-00 0 0 4094 0 0 0 1

Page 31 | AlliedWare™ OS How To Note: NLB Clustering

0. id 6922. offset 0.127: icmp 64: echo 16:45:44.. You may see the same behaviour with HTTP requests (one port egress.. it never appears in the FDB..127: icmp 64: echo reply seq 57088 (tos 0x0.127 > (tos 0x0. id . ttl 128.0..(ffff...127: icmp 64: echo 0. 1000 Mbps Ports in Trunk ... ttl 128.127: icmp 64: echo reply seq 57344 (tos 0x0.0.127 > 172. 1000 Mbps Ports in Trunk .10 LAG ID: [(8000....208993 IP 84) 172.16... length: 84) 10 request seq 57088 0. where all the MAC addresses in a trunk are associated in FDB with one port in that trunk..0011. in this case it was over port 3. id 6872.0.0000).0. with minimal load. offset 0.16.00-04-23-ab-37 02. The NLB server will decide which port to send the reply over (either 3.2)...(ffff.00-04-23-b5-bb-6e. flags [DF]... lacp29: Speed . length: 84) 10 request seq 57600 and only echo replies on port 3..0....1 you should see only echo requests: 16:45:42.. This is a designed feature of trunking. id . flags [DF].208795 IP (tos 0x0.0000)] Trunk group name ....0. ttl 63.2: 16:47:26. Both ports of the node are associated with the same port.0. they are the second ports of each node.0004. 3.127 > 16:47:27...208777 IP (tos 0x0. offset 0.0.. ping the cluster address.. In this setup. flags [DF].0000)] Notice that the MAC address 03-bf-ac-10-00-7f only appears in the IP ARP table.00. length: 10. This does not break trunking because traffic will still be balanced across all the ports in the trunk. lacp30: Speed .. 3....0.. length: 10. one port ingress) but as traffic increases it is more likely that the SwitchBlade and Windows will balance traffic Page 32 | AlliedWare™ OS How To Note: NLB Clustering .. ttl 128.0.208996 IP 84) 172...00.1 or 3. offset 0. flags [DF].. offset 0..00-00-cd-05-01-14. flags [DF]... ttl 63.0011.0.16..0.00..0.1-3.127 > 172.2.16...Link Access/Aggregation Control Protocol and a SwitchBlade Manager Switchblade> sh lacp trunk LACP Dynamic Trunk Group Information ----------------------------------------------------------------------------Trunk group name .. id .. traffic may be seen to only be received on one port in the trunk and reply sent out the other port.2 LAG ID: [(8000..127: icmp 64: echo 16:45:43. flags [DF]. offset 0..127 > 172... To check that the LACP trunk is working.00-00-cd-05-01-14..208992 IP 84) 172.0. length: 84) 10 request seq 57344 0. id 6921....0.0000).127: icmp 64: echo reply seq 57600 The important point is that the ICMP echo request is sent to one port only in the trunk.208807 IP (tos 0x0..0...16..9-3... On port 3.16. length: 10... Another two (00-04-23-b5-bb-6f and 00-04-23-ab-37-03) appear only in the FDB.... Two of the MAC addresses in the FDB appear in the IP ARP table... ttl 63..0.0005..00. Note that both servers may respond to the ICMP packet depending on the NLB rules so your client could report duplicate replies.127 > 16:47:28..

[tcp sum ok] 397:397(0) ack 275 win 65262 <nop.42628: S [tcp sum ok] 1493187854:1493187854(0) ack 2433262544 win 16384 <mss 1460.127.nop. offset 0. ttl 128.nop.sackOK.nop. which does not rely on special hardware and software to run.42235 > 172.timestamp 235508245 23725> Port 3.42235 > 172.timestamp 0 0.0. length: 447) 172. The alternative is Static Link Aggregation. ack 4292649670 win 5840 <nop.0.16.32002.127.http: F 273:273(0) ack 397 win 6432 <nop.16. For more information about LACP on Allied Telesis switches.0.127.510746 IP (tos 0x0. length:64) 172.42628: FP 1:396(395) ack 274 win 65262 <nop.0.1 would appear as such: 16:20:13.nop.0.http > 10.127. offset 0.0.http: .nop.0.0. The benefits of that may only be slight. id 11984.0. flags [DF].0.2 may appear as such: 17:10:27. However.http > 10.42235 > 172.timestamp 22780 235809610> Please see the section "Special Note About LACP and 802.42628: .0.127. Page 33 | AlliedWare™ OS How To Note: NLB Clustering .wscale 0> 16:20:13.127.511880 IP (tos 0x0. flags [DF].0.nop.127.sackOK> 17:10:27.http: SWE 3617082630:3617082630(0) win 5840 <mss 1460.nop.nop.nop.16.0.127. Summary This method will provide efficient fault tolerance and load balancing.127. consult the IEEE standard 802.856129 IP 10. id 11983. This setup will not give an individual node redundancy if a line card is hot swapped.0.16.3ad" on page 36. ttl 128.16.42235 > 172.0.Link Access/Aggregation Control Protocol and a SwitchBlade across both ports.0.127.127. id 11982.timestamp 235508243 0.nop. ttl 128.127.857435 IP 10.860174 IP 10. because hot swapping one line card will leave the other node(s) up.0. length: 52) 172.127.0. Port 3. For more information on just LACP. see your switch’s Software Reference.858012 IP 10. LACP will give you management capabilities over link aggregation.16.http: P 0:273(273) ack 1 win 5840 <nop.timestamp 235508244 0> 16:20:13.0.0. Cluster redundancy is preserved if the two nodes each connect to two 24-port trispeed cards. better use of bandwidth is gained.127.509249 IP (tos 0x0.nop.timestamp 22780 235809610> 17:10:27.16. The key is that you should never see duplicate unicast packets over two ports in the same trunk. flags [none].wscale 0.0. offset 0.timestamp 235508244 0> 16:20:13.http > 10.

0.Static Link Aggregation with a SwitchBlade Static Link Aggregation with a SwitchBlade Static link aggregation was developed to increase throughput between switches.16.168.0.16.VLAN200 vlan200 .41 clumpy2-port 4.0.2 . this is a viable option.127 Private CLUSTER . Equipment z z z z z z Intel Pro/1000 MT Dual Port NIC Linux PC as the workstation 2 x 32 bit PCs with Windows 2003 Enterprise Edition SwitchBlade chassis (AT-SB4108-00) Switch Control Card (AT-SB4211) 2 x 8 port UTP trispeed line card (AT-SB4411 8-1000T(RJ45)) Network diagram Cluster Clumpy1 PC .172.0.16.port3.1-3.2 . Because Allied Telesis switches support 802.3-10.0/24 Cluster Clumpy2 Page 34 | AlliedWare™ OS How To Note: NLB Clustering .127 private .1-4.42 virtual .192.0.VLAN100 clumpy1-port 3.172.0.172.3ad.

1-4. NLB multicast mode must be used in the cluster. Page 35 | AlliedWare™ OS How To Note: NLB Clustering .255.127 int=vlan100 eth=03-bf-ac-10-00-7f port=3.0.254 mask=255.255. Summary This method will provide efficient fault tolerance and load balancing with the added bonus of working on any Allied Telesis SwitchBlade line card. you cannot connect two ports on the same node to two different cards and expect to be able to trunk them.Static Link Aggregation with a SwitchBlade Configuration # # SYSTEM configuration # set system name="Switchblade" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=3. better use of bandwidth is gained.1-3.1-3. By putting each node on its own line card.2 create vlan="vlan200" vid=200 add vlan="200" port=3.0.254 mask=255.1-4.255. Please see the section "Special Note About LACP and 802. Cluster redundancy is preserved because hot swapping one line card will leave the other node(s) up.3 # # IP configuration # enable ip add ip int=vlan100 ip=172.2.4. Therefore.1 # # SWITCH configuration # create switch trunk=clumpy2 port=4. However.0.2 speed=1000m Description 802.2 speed=1000m create switch trunk=clumpy1 port=3.16. only cluster redundancy is gained.0.3ad" on page 36.3ad trunking demands that both ports in a trunk be on the same switch instance. This setup will not give an individual node redundancy if a line card is hot swapped.16.0 add ip int=vlan200 ip=10.255.0 enable ip mscluster add ip arp=172.

depending on the SwitchBlade software release and control cards.Static Link Aggregation with a SwitchBlade Special Note About LACP and 802. which is a decision the NIC driver makes based on its own algorithm. layer 4 hashing will give a randomly balanced traffic shape for packets with the same source and destination IP. An IP packet may be sent by the SwitchBlade to port 1 of the node and the reply sent to the SwitchBlade from port 2 on the node. The port that the SwitchBlade sends traffic to is dependant on the layer(s) that you choose to involve in the hashing calculation that determines the egress port for a given packet. all traffic cannot be seen on one port alone but sniffing both ports should reveal the entire traffic flow with no duplicate packets. Layer 4 hashing may or may not be on by default. The node will continue to send return traffic based on its own algorithm. the SwitchBlade will use TCP port numbers to determine which physical port in a trunk to send traffic over. Layer 2 and layer 3 hashing is normally on. An IP packet may be sent by the SwitchBlade to port 2 of the node and the reply sent to the SwitchBlade again from port 2 on the node.3ad The SwitchBlade will send IP packets destined for the virtual address to one port in each trunk. you can see that the SwitchBlade control card is using layer 2 and layer 3 hashing. Page 36 | AlliedWare™ OS How To Note: NLB Clustering . The use of layer 2 and/or layer 3 hashing without layer 4 will cause the SwitchBlade to send all IP traffic to the same port on the node every time for packets with the same source and destination IP. Therefore. The node's NIC driver will decide to send return traffic via one NIC. Put simply. Under layer 4 hashing. In the following example output from the command show switch.

.... Control Card Blade Position ......... 9...... 106496 Switch Address ........... Alive IP Address . ON Ageing Timer ....... None Pause Flow Control Mode ............. 300 VLAN classification ..... Disabled UpTime . 192.....168............. 9 Blade Switch Instance .................... 106496 Switch Address .......... 0 Number of Uplink Ports ........................ 0 CAM size ( entries ) ...... Basic Basic Flow Control Length ... None Ports mirroring on Both . 0 Mirroring ...0 IPX Address . Control Card Blade Position .............. CXE K1 Switch Type . ON Ageing Timer .......... ON Number of Fixed Ports ..... 00-00-cd-05-01-14 Learning ..... ON Number of Fixed Ports ............................ 04:09:55 Hashingfield ........0 Blade Instance Health ... MODE1 -----------------------------------------------------------------------Switch Configuration -----------------------------------------------------------------------Silicon Revision . 0 CAM size ( entries ) ........255........ Ageingtime .... DISABLED Mirror port ....................0 IPX Address ........ L2 L3 L3 Test mode ..................... 9 Blade Switch Instance .......255.... To be defined STP Forwarding ... Basic Basic Flow Control Length ..... 10 Number of WAN Interfaces ....... 0 Mirroring ........................ DISABLED Mirror port ...168...................1 Blade Instance Health ..... CXE K1 Switch Type ....... 9. 00-00-cd-05-01-14 Learning . 300 VLAN classification ...... 0 Name of Interface(s) ..................... 192.. 0 Name of Interface(s) ..................... L2 L3 L3 Test mode .. None Ports mirroring on Tx ...........Static Link Aggregation with a SwitchBlade Switch Configuration -----------------------------------------------------------------------Silicon Revision ............. MODE1 Page 37 | AlliedWare™ OS How To Note: NLB Clustering ...................... None Ports mirroring on Tx .. To be defined STP Forwarding ...................................... 04:09:55 Hashingfield .............................. None Ports mirroring on Rx ..... 10 Number of WAN Interfaces . Ageingtime ................. Disabled UpTime ................... None Ports mirroring on Rx . None Pause Flow Control Mode . 0 Number of Uplink Ports ...... None Ports mirroring on Both ....... ffffff00 PCI errors ...................... ffffff00 PCI errors ....................... Alive IP Address ..........

2 speed=1000m create switch trunk=clumpy1 port=3.Static Link Aggregation with a SwitchBlade To force the SwitchBlade to mix traffic between ports on the same node.0 inst=9.0 inst=4.2 speed=1000m set switch port=3.0 instance=9.0 inst=9.1-4. to avoid problems with cards that have pre K-revision silicon.1 inst=3.1-3.0 inst=4. Also.1 This results in the example output from the command show switch as shown on the next page. consider turning off layer 3 hashing on all control and line cards: enable switch mirror create switch trunk=clumpy2 port=4.0 inst=3.1 mirror=both disable disable disable disable ena ena ena ena swi swi swi swi switch switch switch switch hash=l4 hash=l4 hash=l4 hash=l4 hash=l3 hash=l3 hash=l3 hash=l3 instance=9. turn on layer 4 hashing. Page 38 | AlliedWare™ OS How To Note: NLB Clustering .

......... 10 Number of WAN Interfaces ............. Alive IP Address . None Pause Flow Control Mode ........... To be defined STP Forwarding .......... ON Number of Fixed Ports ............. None Ports mirroring on Rx ....Static Link Aggregation with a SwitchBlade Switch Configuration -----------------------------------------------------------------------Silicon Revision .... CXE K1 Switch Type ................................ MODE1 -----------------------------------------------------------------------Switch Configuration -----------------------------------------------------------------------Silicon Revision ... 300 VLAN classification ........................... 0 Name of Interface(s) .............. Basic Basic Flow Control Length ......... ffffff00 PCI errors ............. Basic Basic Flow Control Length . 192.................................. ON Number of Fixed Ports .....................255..... 106496 Switch Address ........ 00-00-cd-05-01-14 Learning ............... 0 Mirroring .............. Control Card Blade Position ....... ON Ageing Timer ..... DISABLED Mirror port .......... 9 Blade Switch Instance .............. 0 Mirroring .0 IPX Address ........ 10 Number of WAN Interfaces ..................... None Ports mirroring on Both ..... 0 Number of Uplink Ports ........................... Disabled UpTime ........ 9 Blade Switch Instance ....... MODE1 Page 39 | AlliedWare™ OS How To Note: NLB Clustering ...................... L2 L4 L3 Test mode . Control Card Blade Position ....... None Ports mirroring on Tx .............................1 Blade Instance Health .............. None Pause Flow Control Mode . 04:33:37 Hashingfield ........... ON Ageing Timer ............. 0 Name of Interface(s) . None Ports mirroring on Tx . None Ports mirroring on Both ....... 0 Number of Uplink Ports ....0 Blade Instance Health ...... ffffff00 PCI errors .......... To be defined STP Forwarding ...... 106496 Switch Address ...................... None Ports mirroring on Rx .. 9..255............ 0 CAM size ( entries ) ................ Alive IP Address .................................. 00-00-cd-05-01-14 Learning ......... DISABLED Mirror port . 0 CAM size ( entries ) .... 04:33:37 Hashingfield ....0 IPX Address .... Ageingtime . Disabled UpTime ..... 300 VLAN classification ........ CXE K1 Switch Type ..168..... 9.... 192..168. Ageingtime .... L2 L4 L3 Test mode ....

At layer 3. aggregation or otherwise then the mode you select is entirely your decision. Network diagram client network 192. This example will use the most commonly implemented teaming .Switch Fault Tolerance (SFT). The following method provides the best level of redundancy and failover at layer 2 and layer 3.100. but other routers will also work.eps Page 40 | AlliedWare™ OS How To Note: NLB Clustering .0/24 Clumpy1 server Clumpy2 server nlb-rec.0/24 eth0 VRRP eth0 AR750S router port1 vlan100 AR750S router port1 vlan100 AT-9924T switch AT-9924T switch 172.0. as long as each layer 3 switch only has one port connected to one layer 2 switch. it uses AR750S routers. The layer 2 switches can be any Allied Telesis switch.How Allied Telesis recommends NLB be implemented at layer 3 How Allied Telesis recommends NLB be implemented at layer 3 If you are interested in teamed NICs for redundancy. Layer 3 switches will also work.16.168.

x mask=255. You should also add a VRRP interface for the 172.0 arp=172. For details. For more recent software versions. see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6.254 add vrrp=2 monitoredinterface=eth0 The switches can be configured with or without an IP address but they are not configured with a static IP ARP. This network can be easily scaled up with more switches and servers.16.254 add vrrp=1 monitoredinterface=vlan100 create vrrp=2 over=vlan100 ipaddress=172. Then create a VRRP interface to be the next hop for the 192.168.168. enable add ip add ip add ip ip interface=eth0 ip=192. or scaled down to one switch and two servers each with a single or dual NIC.0/24 network to reach the NLB server network. For the most recent software versions. The following configuration is applied to both AR routers and includes a monitored interface so that the VRRP master will change if the status of vlan100 or eth0 changes. you can see the specific multicast address to use in the NLB Manager (see "Set up the NLB Cluster" on page 66).100. At the most basic level.255.0 interface=vlan100 ip=172. If you need a static ARP entry.0.16. It is important to note that only one switch connects to the AR750S. ideally on both for redundancy: enable ip igmp ena ip igmp interface=vlan100 Page 41 | AlliedWare™ OS How To Note: NLB Clustering .0. the switches do not need any configuration at all.255. but do not need the static ARP entry. enable vrrp create vrrp=1 over=eth0 ipaddress=192. An IGMP querier configuration is required on at least one switch.16. A switch that supports IGMP snooping is all that is actually required. we have added a static ARP entry to the VLAN interface to which the server's layer 2 network connects.100.x mask=255.16.0. The switches should not be configured with multiple VLANs. you need to also use the command enable ip mscluster.255.100. IGMP snooping is enabled by default on Allied Telesis switches.127 int=vlan100 eth=01-00-5e-7f-00-7f port=1 In the above configuration.255.0.How Allied Telesis recommends NLB be implemented at layer 3 Configuration The AR750S is in charge of routing traffic between the rest of the network and the L2 network that the servers are on.168. This will not work properly if a switch is connected to more than one router.0/24 subnet. Add an IP interface and route information to both of the AR750S routers. you need to use the command enable ip macdisparity.

as shown in the following figure. but select “IGMP multicast”.How Allied Telesis recommends NLB be implemented at layer 3 Configure the NLB servers as normal. Page 42 | AlliedWare™ OS How To Note: NLB Clustering .

Copy the i386 directory to your hard drive if you have space. When the PC reboots it will analyse the hard drive and then convert the file system to NTFS without destroying data (hopefully).0/24 network for the private LAN. but with the FAT file system.clumpy1 X Check that your drive partition uses an NTFS file system If your Windows 2003 installation partition file system is not NTFS. If you already have Windows 2003 installed.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Install Windows 2003 Defaults are okay during the install but you must ensure you use NTFS. Page 43 | AlliedWare™ OS How To Note: NLB Clustering . Set up the private node-to-node ethernet connection and assign each node an IP. Before you begin. This example uses the 192. You cannot proceed without NTFS.168.0. then open up a command prompt (otherwise known as a MSDOS window) and type “convert c: /FS:NTFS /X” and then reboot. then convert the drive now. Installation of first NLB Server .

It may be possible to avoid setting up a DC (you could manually set up DNS perhaps) but it is easier to have a domain controller in a Windows environment. Run dcpromo.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Install a domain controller (DC) if you do not have one Windows 2003 clustering requires DNS lookups to work and in the case of Clustering requires an account within the domain to be set up. follow the prompts as per the screenshots and then reboot afterwards. X Set up a Domain Controller Page 44 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 45 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 46 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 47 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 48 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 49 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 50 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 51 | AlliedWare™ OS How To Note: NLB Clustering .

Page 52 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Configure interfaces Note: In the picture above the Network Load Balancing protocol appears in the list. Do not configure this here but make sure it is present and checked.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 53 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 54 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 55 | AlliedWare™ OS How To Note: NLB Clustering .

the gateway and the virtual IP entry for the cluster. Page 56 | AlliedWare™ OS How To Note: NLB Clustering . We need to add clumpy2. Clumpy1 is already there because that is the server we are using now.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Create extra entries for the Cluster in DNS Right click in the right hand window and select “New Host (A)”.

0/24 (reverse) zone hasn't been created yet.16. You can ignore this and manually add an entry later to the reverse zone. It is probably better to create the reverse lookup zone before adding PTR records.0.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster You get the following warning if the 172. This example will cover the creation of a reverse zone next Page 57 | AlliedWare™ OS How To Note: NLB Clustering .

co.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Create a reverse zone for the test.nz domain Page 58 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 59 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 60 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Manually add the reverse entry for each name (clumpy1.127): Page 61 | AlliedWare™ OS How To Note: NLB Clustering .0.16. cluster and gateway) 1. Right click in the right hand pane of the windows above and select “New Pointer (PTR)”. clumpy2. Here is the example for the name “cluster” (for the cluster IP 172.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 2. Page 62 | AlliedWare™ OS How To Note: NLB Clustering . Click “Browse” in the window above and find the name that you want the reverse entry to relate to.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 63 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 3. Do the same for the gateway address (have you added this to the forward zone yet?): Page 64 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 65 | AlliedWare™ OS How To Note: NLB Clustering .

Start the NLB Manager from: Start > Administrative Tools > Network Load Balancing Manager 2.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster . Start the wizard to create a new cluster: Cluster > New Page 66 | AlliedWare™ OS How To Note: NLB Clustering . X Set up the NLB Cluster 1.

Change the filtering mode to affinity “None”. Page 67 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 3.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 4. On unconfigured interfaces your clumpy2 will have IP addresses of the form 169.0. For the second node you enter the second NIC's private network IP address in the “Host” box. Page 68 | AlliedWare™ OS How To Note: NLB Clustering .0 or perhaps 0. All you need is an IP on the private NIC of clumpy2 and a physical connection. Enter the local IP in the “Host” box and click on the “Connect” button. If you have not configured interfaces other than the private NIC of clumpy2 that is okay.0.0.0. which you should do after you have added the first node. Be patient. For the second node it may take some time after pressing the “Connect” button for the interfaces to appear in the available interfaces box. The box below illustrates that for the second node you must change the priority to 2. The box above illustrates adding the second node. to ensure basic IP connectivity.254. Notice that there are already IP addresses assigned to the other interfaces of clumpy2. Wait for as long as it takes and then select the cluster side IP and click on Next.

Be patient. Page 69 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster It takes quite a long time sometimes for the nodes to converge.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Add IIS to enable node as a Web server 1. Page 70 | AlliedWare™ OS How To Note: NLB Clustering . From the Control Panel select “Add or Remove Programmes” and then “Add/Remove Windows Components”.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 71 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Installation of second NLB server .clumpy 2 X Convert your file system to NTFS if it is not already Page 72 | AlliedWare™ OS How To Note: NLB Clustering .

Page 73 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Install the Domain Controller on clumpy2 Don't set the second server as a backup DNS server—that's too much extra bother at this stage. Use the private network for DNS in the adapter settings.

go back and set up a DNS server through the Network Connections settings in the Control Panel Page 74 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster If you get the message above.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 75 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Use the Administrator account. Page 76 | AlliedWare™ OS How To Note: NLB Clustering . in this example. you can see that manager was used and then subsequently rejected a few steps down the line.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 77 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 78 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 79 | AlliedWare™ OS How To Note: NLB Clustering .

Page 80 | AlliedWare™ OS How To Note: NLB Clustering . Use the NLB Manager to do that. Also. You do not need to do anything to clumpy2 for it to join the NLB cluster. This document should at least leave you with a basic. but working cluster. You will need to make many tweaks to the NLB policy to suit your environment. you may need to spend time convincing Windows 2003 to use the private NIC for cluster heartbeat messages.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Install Windows IIS if you haven't already (refer to the clumpy1 setup). The earlier use of the NLB Manager on clumpy1 should have configured clumpy2 accurately.

Additionally. as described in "Multicast mode with the IGMP option selected" on page 5. Instructions on how to interpret the output is after this example. use the following command to match ports to multicast groups (represented by the multicast MAC address): show switch memory=l2broadcast This produces two tables. Manager > sh swi tab=l2broadcast CXE L2 multicast and VLAN broadcast table: Index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 MAC 55-55-55-55-55-55 01-00-5e-7f-00-7f 01-00-5e-00-01-18 01-00-5e-7f-ff-fe 01-00-5e-00-01-16 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 33-33-00-00-00-00 33-33-00-00-00-00 33-33-00-00-00-00 01-00-00-00-00-00 01-00-00-00-00-00 01-00-00-00-00-00 VID 1 1 1 1 1 1 2 28 1 2 28 1 2 28 1 2 28 L2MC-Group MC Hits 4096 5 6 7 9 4097 4098 4124 1 1 1 2 3 4 4097 4098 4124 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 5 6 7 9 1 2 26 1 1 1 2 3 4 1 2 26 PV Prio Perm DIS DM Expire 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Segment L2MC-Group TagMask 77 77 77 77 77 83 85 85 85 87 87 87 87 87 0 5 6 7 9 1 2 3 4 8190 8189 4097 4098 4124 00000000 00000000 00000000 00000000 00000000 80000000 80000000 80000000 80000000 00000000 00000000 00000000 00000000 00000000 OutportMask Hits 00000000 00000021 00000021 00000021 00000008 80000000 8000003b 80000000 80000800 00000000 00000000 0000003b 00000000 00000800 0 5 6 7 9 1 2 3 4 0 0 0 0 0 BCSC-Id BCSC-Enable Perm 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Page 81 | AlliedWare™ OS How To Note: NLB Clustering . This section describes how to interpret the table output.Appendix 2: IGMP snooping port membership information in switch memory tables Appendix 2: IGMP snooping port membership information in switch memory tables The simplest way to find out which ports belong to a group is to use the command show igmpsnooping. SwitchBlade and AT -9800 On a Switchblade or AT-9800 series switch. as the following example shows for a 12-port AT-9800 switch. you can find specific information about which ports belong to a group in the layer 2 multicast tables.

Convert each hexadecimal digit to a 4-digit binary number (0010 and 0001 in the example). Counting from the right. 3. note the OutportMask field. In the first table. AT-8800. 5. Therefore.. Each 1 corresponds to a port that belongs to the target group. In that row. In that row. Counting from the right. In the example in bold above. Therefore. Manager > sh swi memory=marl Multicast Table-0.Appendix 2: IGMP snooping port membership information in switch memory tables To work out which ports are members of a group: 1. because they are the two right-most digits). use the following command to match ports to multicast groups (represented by the multicast MAC address): show switch memory=marl This produces a single line of output per group. Mac=01-00-5e-7f-ff-fe VLAN=1 ports=00000011 utports=03fffff7 cosDst=0 To work out which ports are members of a group: 1. Mac=01-00-5e-7f-00-7f VLAN=1 ports=00000011 utports=03fffff7 cosDst=0 3. Mac=01-00-5e-00-01-18 VLAN=1 ports=00000011 utports=03fffff7 cosDst=0 2. Select the row that has the MAC address of the L2 multicast group that you are interested in. note the positions of the digit 1 in the binary number. in this example. AT-8600 On these switches. ports 1 and 5 are members of the group 01-00-5e-7f-00-7f in VLAN1. 4. the L2MC Group field value is 5 and the VID is 1. 0 0000 - ports 12-9 0 0000 - ports 8-5 2 0010 6 ports 4-1 1 0001 1 hexadecimal binary port number Rapier. ports 1 and 6 are members of the group 01-00-5e-7f-00-7f in VLAN1. This is a series of hexadecimal digits (0x21 in the example). Each 1 corresponds to a port that belongs to the target group. select the row that has the MAC address of the L2 multicast group that you are interested in.. Convert that hexadecimal number to binary (10001 in the example). 2. note the positions of the digit 1 in each binary number. note the “ports=” value. in this example. 3. Note the number in its “L2MC Group” field and the VID. . as the following example shows. 2.0 3/(1-255) ----------------------------------------------------------------------------1. 4. so this is VLAN1. Page 82 | AlliedWare™ OS How To Note: NLB Clustering . The right-hand most digits each represent 4 concurrent ports (in the range 1-4 and 5-8 in the example. as the following table shows. Find the row in the second table that has the same value in its L2MC Group field. This is a hexadecimal number that indicates the member ports (0x11 in the example in bold above).

C613-16073-00 REV B . in the United States and other countries.6 3 01-00-5e-00-01-18 1 4098 0 2 Ports = 2.alliedtelesis. AT-9900.x900. Inc. logos.11 Asia-Pacific Headquarters | 11 Tai Seng Link | Singapore | 534182 | T: +65 6383 3832 | F: +65 6383 3830 www.com © 2007 Allied Telesis. Information in this document is subject to change without notice. Allied Telesis is a trademark or registered trademark of Allied Telesis. and product designs that are trademarks or registered trademarks are the property of their respective owners.00 | F: +41 91 69769.6 2 01-00-5e-7f-00-7f 1 4102 0 2 Ports = 2.6 4 01-00-5e-00-01-16 1 4096 0 1 Ports = 4 5 03-bf-ac-10-00-7f 1 1 1 1 Ports = 12 USA Headquarters | 19800 North Creek Parkway | Suite 200 | Bothell | WA 98011 | USA | T: +1 800 424 4284 | F: +1 425 481 3895 European Headquarters | Via Motta 24 | 6830 Chiasso | Switzerland | T: +41 91 69769. All company names. Manager > sh switch table=l2multicast L2 Multicast Group and VLAN Broadcast table: -----------------------------------------------------------------------Total number of entries = 6 -----------------------------------------------------------------------Index MAC VID MCGroup CPU_MEM NumPorts PORT_LIST -----------------------------------------------------------------------0 01-00-5e-7f-ff-fa 1 4097 0 1 Ports = 4 1 01-00-5e-7f-ff-fe 1 4099 0 2 Ports = 2. as the following example shows. use the following command to match ports to multicast groups (represented by the multicast MAC address): show switch memory=l2multicast This lists the ports for each group. It is much easier to tell which groups are in each multicast group than it is with the previous switch series. AT-8948 On these switches. All rights reserved. Inc.