You are on page 1of 83

AlliedWareTM OS How To | Configure Microsoft Windows 2003 Network Load Balancing Clustering with Allied Telesis Switches

Introduction
This is a guide to Network Load Balancing (NLB) clustering options with Allied Telesis managed layer 3 devices. NLB is one of the clustering technologies available from Microsoft. It provides high availability for services such as HTTP and FTP, by grouping identical servers into a cluster and sharing the network load between all currently-available servers in the cluster. This How To Note uses scenarios that have been tested in the lab and verified to work. It also deals exclusively with nodes that have dual network cards on the cluster side for bandwidth redundancy and/or aggregation. This document describes best-practice implementations of NLB clustering on Allied Telesis switches. You should read through this document and decide which setup best suits your requirements. Please pay attention to the details, because some traps in setting these scenarios up can cause unexpected results.

C613-16073-00 REV B

www.alliedtelesis.com

Introduction

What information will you find in this document?
This document is divided into the following sections:
z z

"Network Load Balancing clustering modes" on page 3 "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6 "Note about capturing traffic on the clustered servers" on page 7 "How to quickly set up Windows 2003 NLB clustering" on page 7 "Switch Fault Tolerance, two switches and VRRP" on page 8 "Switch Fault Tolerance with a SwitchBlade" on page 17 "Adaptive Load Balancing, teamed NICs and a SwitchBlade" on page 19 "Adaptive Load Balancing, two switches and VRRP" on page 22 "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29 "Static Link Aggregation with a SwitchBlade" on page 34 "How Allied Telesis recommends NLB be implemented at layer 3" on page 40 "Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster" on page 43 "Appendix 2: IGMP snooping port membership information in switch memory tables" on page 81

z z z z z z z z z z z

Which product and software version does this information apply to?
The specified solutions only apply to AT-9800 series and SwitchBlade 4000 series, except for the solution in "How Allied Telesis recommends NLB be implemented at layer 3" on page 40. In that solution, the layer 2 switches can be any Allied Telesis switch and the layer 3 device can be any AR400 or AR700 router. It applies to AlliedWare versions since 2.7.5. The switch or router implementation varies slightly with the software version—see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6.

Page 2 | AlliedWare™ OS How To Note: NLB Clustering

Network Load Balancing clustering modes

Network Load Balancing clustering modes
There are two modes for Network Load Balancing with Windows 2003: unicast and multicast. Multicast mode has a further option of IGMP Multicasting. This section discusses how these modes work. Note that clustering only works, in multicast or unicast mode, if all packets sent to the cluster's IP address are sent to all nodes in the cluster. This means that the cluster-side switch must flood traffic to all ports. To reduce traffic congestion, the cluster should ideally be put in its own VLAN. This recommendation applies regardless of which NLB cluster mode you use. This document focuses on non-IGMP multicast network load balancing in a layer 3 environment. However, IGMP multicasting appears to be an excellent option for layer 2 networks, because traffic to the servers is not seen on other ports unless they send an IGMP report for that group. Traffic to the servers can be protected from prying eyes by using IGMP filtering and static IGMP entries (see How To Configure IGMP for Multicasting on Routers and Managed Layer 3 Switches from www.alliedtelesis.com/resources/literature/howto.aspx). In a layer 3 environment you can be less concerned about the security of traffic because your servers should be in their own VLAN, i.e. not in the same segment as clients. In the section "How Allied Telesis recommends NLB be implemented at layer 3" on page 40, we use multicast mode with IGMP selected.

Unicast mode
In unicast mode, all hosts in the cluster share a single unicast “cluster” MAC address, to go with the cluster IP address. This mode has the disadvantage that it stops cluster hosts from using their own “burned-in” MAC addresses, so hosts cannot contact each other unless you install a second NIC card on each host and use that for intra-cluster communication. The cluster operation forces the switch to flood all packets that are destined for the cluster, by stopping the switch from learning the cluster MAC address. The switch cannot learn the cluster MAC address because it never appears in the source field of the Ethernet header. Instead, each NIC uses a special unicast MAC address in the source field of the Ethernet header. The cluster MAC address must never be found in the source field of the Ethernet header, because otherwise the switch will learn the cluster MAC address, stop flooding packets to all ports, and only one node in the cluster will receive traffic for the cluster's IP. In the two-node cluster that this document deals with, both nodes will use the same cluster MAC address when answering ARP requests for the cluster's address, which is 172.16.0.127.

Page 3 | AlliedWare™ OS How To Note: NLB Clustering

Network Load Balancing clustering modes

Multicast mode
In multicast mode, hosts in the cluster use their real “burned-in” MAC address in the source field of the Ethernet header. However, they answer ARP requests for the cluster IP address with a multicast MAC in the ARP packet's payload, while the Ethernet header on the ARP reply still has the real MAC address. This allows cluster hosts to contact each other. The following tcpdump output shows the multicast MAC:
14:02:57.814560 arp who-has 172.16.0.127 (Broadcast) tell 172.16.0.1 14:02:57.814910 arp reply 172.16.0.127 is-at 03:bf:ac:10:00:7f

Because the ARP reply is a multicast MAC address, Allied Telesis switches will by default ignore it, so that MAC address is never found in the software IP ARP table. The solution to this depends on the software version—see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6. Because this document deals with dual NICs on each node, multicast is often the only NLB mode that these scenarios will work with. The reason is due to a limitation in the network card drivers, which in many of the setups will simply not recognise the unicast MAC address in the destination field of the Ethernet header. In other words the NICs will not process packets destined to the virtual MAC address despite it being the same source MAC they are sending packets from. In both multicast and unicast mode the mechanism to cause flooding is as such: ARP requests for the cluster IP are answered with the virtual cluster MAC address in the ARP packet's payload. The node must send all packets, including ARP requests and replies, with a different source MAC in the Ethernet header. This prevents the switch from entering the virtual cluster MAC into the forwarding database (FDB). The end result is that traffic destined for the virtual MAC will be flooded to all ports in the VLAN associated with the cluster IP. Note: According to Microsoft if clustering is working properly, one node cannot ping another node in the same cluster. This explains why cluster traffic (like MS NLB heartbeats) generated by each node and teaming packets (like Intel ANS Probes) generated by the NIC driver are broadcasted.

Page 4 | AlliedWare™ OS How To Note: NLB Clustering

..6 Vlan Name (vlan id) .. Entry timeout 234 secs Entry timeout 235 secs You can also use the memory tables for information about port membership—see "Appendix 2: IGMP snooping port membership information in switch memory tables" on page 81.. 239......0. the servers send IGMP reports for a group that corresponds to the MAC address they are putting in their ARP responses...127 is-at 01:00:5e:7f:00:7f The Windows client will happily add this IP / MAC combination to its internal ARP table (on a Windows or Linux PC.0..254 Ports 1... which still identifies it as an IP-multicast MAC address: 16:07:03. Group..16..16.......643160 arp who-has 172... cluster hosts can also contact each other.. use “arp -a” to see all ARP entries).. This mode also causes the cluster to use its real “burned in” MAC address in the source field of the ethernet header.....0...127 is then sent with an ethernet destination of 01:00:5e:7f:00:7f...66 16:07:03....... Reports are sent frequently by the NLB servers so there is no concern about IGMP entries timing out on the switch......... Page 5 | AlliedWare™ OS How To Note: NLB Clustering . Off Group List ....... To ensure that the switches do correctly forward the multicast data. The switch's IGMP snooping table (the show igmpsnooping command) reveals this information: IGMP Snooping -----------------------------------------------------------------------Status ..16.. No group memberships....0...127 tell 172.. Enabled Disabled All-groups ports .... This mode uses IGMP to prevent the switch from flooding all ports.. default (1) Fast Leave ...255. The cluster node responds with its burned in MAC in the Ethernet source field.6 Group.....16. Microsoft's IGMP multicasting mode cleverly turns the concept of IGMP clients and hosts on its head..0. Off Group List .. the clustered servers are seen as IGMP clients and the IGMP multicast data (stream) is sent by the workstations trying to access the cluster.255.643330 arp reply 172.... instead traffic only goes to NLB ports. All IP traffic destined for 172... The cluster will answer ARP requests for the cluster IP address with a multicast MAC in the ARP packet's payload. Note though. 239....127 Ports 1...Network Load Balancing clustering modes Multicast mode with the IGMP option selected In multicast mode with the IGMP option selected.. while the Ethernet header on the ARP reply still has the real MAC address... vlan2 (2) Fast Leave . None Vlan Name (vlan id) ..255.... As far as an Allied Telesis switch is concerned.. that the MAC address is slightly different to the previously discussed multicast mode—the MAC address starts with the bytes 01:00:5e..

8.7. There are three sets of behaviour: z with the earliest versions. You can specify any port in the static ARP entry—the number is irrelevant. using the command add ip arp with later 2. you need to enter at least one extra command to get multicast mode to work. This means that you cannot use other switches with configurations that have multiple ports connected to the server cluster. In this How To Note. SwitchBlade and AT-9800 switches ignore the port number and flood the entry to the VLAN. If the Allied Telesis device is layer 3 switching between the clients and the servers. This means that you can use SwitchBlade and AT-9800 switches in all configurations in this How To Note.6 versions and with version 2.6 and 276-01.8.Accepting ARP entries that have a multicast MAC address and a unicast IP address Accepting ARP entries that have a multicast MAC address and a unicast IP address Allied Telesis switches ignore multicast MAC addresses by default. using the command add ip arp. you need to enter the command enable ip mscluster and then enter the multicast MAC as a static entry.6 276-01 ≥ 276-02 ≥ 2. Other switches do not flood the entry to the VLAN.1 and later. depending on the software version. Commands required: add ip arp AT -9800 series: sb-275 to sb275-04 sb-276 sb276-01 sb276-02 ≥ sb-281 SwitchBlade: ≥ sb-275a Other layer 3 devices: 2. with all sb275a versions and with versions 2.1 Yes Yes No No Yes Yes No No No No Yes Yes Yes Yes No Yes Yes Yes No No No Yes Yes No No No No No Yes Yes enable ip mscluster enable ip macdisparity Page 6 | AlliedWare™ OS How To Note: NLB Clustering . Summary The following table shows these differences. you have to also specify a port number. you need to enter the multicast MAC as a static entry. z z Ports in ARP entries When you add a static ARP entry for a VLAN.7. You do not need to enter the multicast MAC as a static entry.7. you need to enter the command enable ip macdisparity. the layer 3 switches have multiple ports connected to the cluster in all configurations except the solution in "How Allied Telesis recommends NLB be implemented at layer 3" on page 40.

The following sections include the switch configurations for a number of NLB options: z z z z z z z "Switch Fault Tolerance. In our setup. The appendix will get you started and after you have had some success you must do more research into cluster policies and design to make your cluster operate efficiently. we conclude that packet sniffers conflict with the network interface driver. However. For that reason. Page 7 | AlliedWare™ OS How To Note: NLB Clustering . The ideas are not overly complex but the mechanisms behind it can take time to understand. The results cannot be trusted. For example. So. or Windows cluster software. and produce erroneous results. two switches and VRRP" on page 8 "Switch Fault Tolerance with a SwitchBlade" on page 17 "Adaptive Load Balancing. but also teamed network cards.Note about capturing traffic on the clustered servers Note about capturing traffic on the clustered servers The setups in this How To Note not only use network load balancing. you should not do packet sniffing on the servers themselves. be advised that this implementation is not guaranteed to offer the best security or design for your organisation. when debugging a cluster. inter-cluster node (server-to-server) pings should not work. ping would succeed when Ethereal was capturing traffic on a node. teamed NICs and a SwitchBlade" on page 19 "Adaptive Load Balancing. two switches and VRRP" on page 22 "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29 "Static Link Aggregation with a SwitchBlade" on page 34 "How Allied Telesis recommends NLB be implemented at layer 3" on page 40 This is followed by "Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster" on page 43. How to quickly set up Windows 2003 NLB clustering Setting up a Windows 2003 Network Load Balancing cluster requires considerable time for research.

0. then the two switches will maintain node and cluster connectivity without network downtime if each server is attached to both switches. At any given time. The AT-9924SP also serves as a media adapter so that the UTP NIC can connect to the fibre AT-9816GB.16.0.0.0/24 Tagged VLAN100 & VLAN200 10.0.254/24 VLAN200 .0/24 AT-9816GB-2 Clumpy2 PRIVATE INTERFACE clumpy1 .0. two switches and VRRP Switch Fault Tolerance.10. Using SFT.42 02-02-ac-10-00-7f virtual-172. two switches and VRRP Switch Fault Tolerance (SFT) provides a failover relationship between two adapters when each adapter is connected to a separate switch.16.0.168.0/24 Cluster 172.168.168.0.0. Equipment z z z z z z Intel Pro/1000 MT Dual Port NIC Linux PC as the workstation 2 x 32 bit PCs with Windows 2003 Enterprise Edition AT-9924SP as L2 switch connected to workstation 2 x AT-9816GB AT-8748SL as L2 switch on private cluster LAN If only two servers are being used in the cluster then the AT-8748SL may be left out and the nodes connected directly to each other with a crossover cable.172.0.42 CLUSTER INTERFACE clumpy1 .16.41 02-01-ac-10-00-7f clumpy2 .0.0/24 10.0.172.0.16.192.0/24 Cluster 172.0/24 10.0.0.0/24 L2 Switch Workstation L2 Switch Private 192.0/24 AT-9816GB-1 Cluster 172.0.192.168.41 clumpy2 .0.Switch Fault Tolerance.0.0.254/24 clumpy1 to port 1 on AT-9816GB-1 port 1on AT-9816GB-2 clumpy2 to port 9 on AT-9816GB-1 port 9 on AT-9816GB-2 WORKSTATION 10. one adapter is alive while the other is in standby mode.16. if one switch fails.127 02-bf-ac-10-00-7f VRRP on AT -9816GBs VLAN100-172.0/24 Clumpy1 Cluster 172.16.16. Network Diagram VLAN100 VLAN200 Private 192.16.0.127/24 Page 8 | AlliedWare™ OS How To Note: NLB Clustering .

0.255. This configuration uses 2 server clusters with 1 private and 2 teamed NICs.254 add vrrp=200 monitoredinterface=vlan200 Page 9 | AlliedWare™ OS How To Note: NLB Clustering .9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged delete vlan="1" port=2 # # IP configuration # enable ip add ip int=vlan100 ip=172.255.1 mask=255.0.0.0. we will give a configuration with NLB in unicast mode.0. two switches and VRRP Unicast Mode First.255.0 add ip int=vlan200 ip=10.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.0 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.255. because of Windows 2003 and NIC driver variance. We cannot guarantee that unicast will work at all.0.16.Switch Fault Tolerance. Unicast VRRP switch configurations AT -9800 #1 # # SYSTEM configuration # set system name="AT-9816GB-1" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.16. Note that multicast mode is favoured over unicast mode.1 mask=255.

9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged # # IP configuration # enable ip add ip int=vlan100 ip=172.0.255.0.255.254 add vrrp=200 monitoredinterface=vlan200 AT -9924SP # # SYSTEM configuration # set system name="AT-9924SP" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=1-8 create vlan="vlan200" vid=200 add vlan="200" port=9-12 create vlan="vlan300" vid=300 add vlan="300" port=13-16 create vlan="vlan400" vid=400 add vlan="400" port=17-20 create vlan="vlan500" vid=500 add vlan="500" port=21-24 Page 10 | AlliedWare™ OS How To Note: NLB Clustering .0.255.16.0 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.2 mask=255.2 mask=255.255. two switches and VRRP AT -9800 #2 # # SYSTEM configuration # set system name="AT-9816GB-2" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.Switch Fault Tolerance.0.16.0.0.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.0 add ip int=vlan200 ip=10.

0.0.127 00-50-fc-ee-f5-13 0 1 200 0 0 0 1 2 101 172.1 00-00-cd-09-73-50 0 0 100 0 0 0 1 5 102 172. 02-02 tells us that it is a clustering MAC address for the node with priority 2. however.0.41 02-bf-ac-10-00-7f 0 4 102 172.255 00-00-cd-09-73-50 0 1 200 0 0 0 1 8 110 172.0. Page 11 | AlliedWare™ OS How To Note: NLB Clustering .42 02-bf-ac-10-00-7f 0 1 101 10.0.0. Notice that the two highlighted MAC entries are very similar but not the same as the cluster's MAC address.255 00-00-cd-09-73-50 0 0 100 0 0 0 1 6 102 10.16.0.16.0 00-00-cd-09-73-50 0 0 100 0 0 0 1 9 110 10.0. if we look at the switch's FDB: Manager top> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 8 --------------------------------------------------------------------------VLAN MAC Address Port Status -------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-00-5e-00-01-64 CPU static 200 00-00-5e-00-01-c8 CPU static 1 dynamic 100 02-01-ac-10-00-7f 9 dynamic 100 02-02-ac-10-00-7f 200 00-50-fc-ee-f5-13 3 dynamic The table above illustrates that the MAC address 02-bf-ac-10-00-7f does not appear in the switch chip's table.16.16.0.Switch Fault Tolerance.0.127 02-bf-ac-10-00-7f 0 0 100 0 0 0 1 0 100 0 0 0 1 3 101 172. Manager top> sh swi tab=ip CXE IP route table: Index Seg Network Nexthop ToProc VRP VID Hits PV Prio Perm ----------------------------------------------------------------------------0 100 0 0 0 1 0 101 172.0. 02-01 tells us that it is a clustering MAC address for the node with priority 1. so a packet with a destination MAC address of 02-bf-ac-10-00-7f must be flooded out all ports in the VLAN.0.16.0.0 00-00-cd-09-73-50 0 1 200 0 0 0 1 The clustered IP's MAC has been learnt and the AT-9816GB will make routing decisions to 172.1 00-00-cd-09-73-50 0 1 200 0 0 0 1 7 102 10.16.41/42/127 based on the MAC address 02-bf-ac-10-00-7f.0.0.16. two switches and VRRP Table Entries Here is an example of the entries that should appear in the switch FDB and ARP tables when unicast mode is used with NLB clustering.

Switch Fault Tolerance.255. Page 12 | AlliedWare™ OS How To Note: NLB Clustering . see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6.255.1 mask=255.0.16. For details.0 add ip int=vlan200 ip=10. you need to use the command enable ip macdisparity. you need to also use the command enable ip mscluster. If you need a static ARP entry. Multicast VRRP switch configurations AT -9800 #1 # # SYSTEM configuration # set system name="AT-9816GB-1" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.0.1 mask=255.0. For the most recent software versions.0.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.0 add ip arp=172. In the following configurations. as shown by the underlined text above.254 add vrrp=200 monitoredinterface=vlan200 Note: The MAC address is now slightly different.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172. we have added a static ARP entry.255.0. you can see the specific multicast address to use in the NLB Manager (see "Set up the NLB Cluster" on page 66).0.0.16. but do not need the static ARP entry.16.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged delete vlan="1" port=2 # # IP configuration # enable ip add ip int=vlan100 ip=172.255. For more recent software versions. two switches and VRRP Multicast Mode This is the same setup but this time with NLB clustering in multicast mode.

0.0 add ip int=vlan200 ip=10.255.16.255.0.16.0.0.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged # IP configuration # enable ip add ip int=vlan100 ip=172.Switch Fault Tolerance.0 add ip arp=172.255.255.16.0.0.254 add vrrp=200 monitoredinterface=vlan200 AT -9924SP # # SYSTEM configuration # set system name="AT-9924SP" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=1-8 create vlan="vlan200" vid=200 add vlan="200" port=9-12 create vlan="vlan300" vid=300 add vlan="300" port=13-16 create vlan="vlan400" vid=400 add vlan="400" port=17-20 create vlan="vlan500" vid=500 add vlan="500" port=21-24 Page 13 | AlliedWare™ OS How To Note: NLB Clustering .0.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1# # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172. two switches and VRRP AT -9800 #2 # SYSTEM configuration # set system name="AT-9816GB-2" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.2 mask=255.2 mask=255.

16.16.255. This has been statically added.0.16.255 ff-ff-ff-ff-ff-ff Other active vlan100(1) 172. There is no entry for the MAC 03-bf-ac-10-00-7f in the forwarding database so traffic will be flooded to all ports in the VLAN that is attached to 172.0.41 00-04-23-b5-bb-6f Dynamic active vlan100(2) 172.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(1) 172.0.0.0.255. there is an entry in the software table for the cluster: IP 172.127 00-50-fc-ee-f5-13 Dynamic active vlan200 10.0.16.Switch Fault Tolerance.16.255 ff-ff-ff-ff-ff-ff Other active Page 14 | AlliedWare™ OS How To Note: NLB Clustering .16.0. two switches and VRRP Table Entries The following tables are the contents of the switch tables for this scenario.0.0.255 ff-ff-ff-ff-ff-ff Other active vlan100 255.127.0/24.0. Note that in this case the individual server IPs (172.127 03-bf-ac-10-00-7f vlan100 172. Manager AT-9816GB-1> sh swi fdb Switch Forwarding Database (hardware) ----------------------------------------------------------------------------Total Number of Entries = 7 -------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-00-5e-00-01-64 CPU static 200 00-00-5e-00-01-c8 CPU static 100 00-04-23-b5-bb-6f 1 dynamic 100 00-04-23-ab-37-02 2 dynamic Manager AT-9816GB-1> sh ip arp Interface IP Address Physical Address ARP Type Status ----------------------------------------------------------------------------vlan200(3) 10.41-42) appear in the ARP table with different MACs and these MACs also appear in the FDB.16.

http: F 273:273(0) ack 583 win 6972 1.127.219723 IP 10. ack 1 win 5840 <nop.0.timestamp 148937549 0> 15:53:00.219650 IP 172.nop.The active port that clumpy1 was connected to on the switch was mirrored and the mirror port was then sniffed by a linux server with tcpdump version 3.nop.127.519534 IP <nop.http: .16. The first capture shows the packet flow when clumpy1 answers the HTTP request (in accordance with the defined clustering rules).127 to 172.40194 > 148937179 230102> 172.nop.0.219277 IP 10.127.127.0.127.0. HTTP load balancing at the packet level follows1. HTTP balancing works correctly.0.0.0. ack 275 win 65262 <nop.16.timestamp 15:52:56.16.0.timestamp 229906 148937549> Packet Capture 2 An identical HTTP request from 10.timestamp 148937549 229906> 15:53:00.0.wscale 0> 15:53:00.0.nop.16.wscale 0.http: F 274:274(0) ack 394 win 6432 <nop.http: .http > 10.127 to 172.0.0.0.sackOK.http > 10.nop.http: .0.nop.127.16.127.http: P 1:274(273) ack 1 win 5840 <nop.16.0.16.3) Page 15 | AlliedWare™ OS How To Note: NLB Clustering .127.127 but served by clumpy2 (172.16.0.42) as seen by clumpy1 (172.0.0.0.0.0. so only packets from the workstation can be seen.16.0. Packet Capture 1 HTTP request from 10.221777 IP 172.nop.0.nop.127.nop.16.127.520006 IP <nop.0.127.0.127.nop.0.0.0.0. The second capture is also taken from the perspective of clumpy1 but this time the HTTP request is serviced by clumpy2. the following traffic flows should apply.nop.220717 IP 172.0.221659 IP 10.127.40195: S 3107871426:3107871426(0) ack 593168139 win 65535 <mss 1460.16.0.0.0.220206 IP 10. ack 583 win 6972 172.127.0.nop.40194 > 148937179 230102> 10.40195 > 172.127.127. two switches and VRRP Traffic Flows Regardless of whether NLB is in unicast or multicast mode.520693 IP <nop.40194 > 148937179 0> 10.127 and served by clumpy1 (172. ack 1481239454 win 5840 172.0.timestamp 148937549 0.0.timestamp 148937549 0> 15:53:00.127.127.0.0.http > 10.16.16.http: P 0:273(273) ack 1 win 5840 172.127.127.nop.521737 IP <nop.0.http: SWE 593168138:593168138(0) win 5840 <mss 1460.timestamp 15:52:56. This is not the fault of the switch.nop. both servers will respond to pings to the cluster address 172.40195 > 172. Interestingly.127.timestamp 10.sackOK> 15:53:00.40195 > 172.timestamp 15:52:56.40195: FP 1:393(392) ack 274 win 65262 <nop.nop.timestamp 229906 148937549> 15:53:00.16.41) as seen by clumpy 1: 15:53:00.8.Switch Fault Tolerance.0.0.8.0.timestamp 0 0.127.16.40195 > 172. Both of these captures are taken from the perspective of clumpy1.16.0.40194 > 148937179 0> 10.3 (libpcap version 0.16.40195:.127.41): 15:52:56.

This is not a solution for adapting to failures that occur further down the network. however. When operating correctly the node should experience only a few seconds of lost connectivity. there is no solution on offer in this document for such a scenario. Page 16 | AlliedWare™ OS How To Note: NLB Clustering . the NIC driver disabled the wrong NIC when a downstream link was broken. in our lab setup with Intel ANS Probes. in the configuration here they will not help. If there is a failure further downstream but the link at the NIC end does not go down then the NIC will not change state because it is not aware of any topology change. Summary This setup is an excellent redundancy option.Switch Fault Tolerance. Probes can be used to determine link failures however. there appears to be no advantage to enabling probes. So in this setup. two switches and VRRP NIC Failover Should the node's link go down on the NIC driver will rapidly change the state of the standby NIC to active. failover is fast if the link to the active NIC goes down. In fact. While the combined bandwidth of the NICs is never utilised.

2. At any given time. one adaptor is alive while the other is in standby mode.16.16.0.3-10.172.172.port3.0.Switch Fault Tolerance with a SwitchBlade Switch Fault Tolerance with a SwitchBlade Switch Fault Tolerance (SFT) provides a failover relationship between two adaptors when each adapter is connected to a separate switch.0.16.41 clumpy2-port4.VLAN100 clumpy1-port4. Using SFT and one SwitchBlade will permit you to hot swap line cards without network downtime if each server is attached to two different line cards.127 CLUSTER .0.1 .2 . Equipment This example uses: z z Intel Pro/1000 MT Dual Port NIC SwitchBlade chassis (AT-SB4108-00) • Switch Control Card (AT-SB4211) • 2 x 8 port UTP trispeed line card (AT-SB44118-100 (RJ45)) z 2 x 32-bit PCs running Windows 2003 Enterprise Edition Network Diagram Clumpy1 PC . 5.1.172.VLAN200 vlan200 .42 virtual .0.127 Clumpy2 Page 17 | AlliedWare™ OS How To Note: NLB Clustering . 5.

255. failover is fast if the link to the active NIC goes down.255.16.127 int=vlan100 eth=03-bf-ac-10-00-7f port=4. This setup will give you full redundancy (both at the cluster and node level) when you hot swap a SwitchBlade line card.0. While the combined bandwidth of the NICs is never utilised.2. Page 18 | AlliedWare™ OS How To Note: NLB Clustering .255. set system name="Switchblade" create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 add vlan="100" port=4. there should never be an entry in the forwarding database for the multicast MAC. there is no solution on offer in this document for such a scenario.0 ip mscluster arp=172.0 int=vlan200 ip=10. and to use the command enable ip mscluster. however. The static ARP entry will. Remember to add a static ARP for the multicast MAC.0. appear in the IP ARP table.254 mask=255.1-4.1-5.2 add vlan="200" port=4.16.0.Switch Fault Tolerance with a SwitchBlade Configuration Set the Windows 2003 NLB cluster servers to use multicast mode.5.255. This is not a solution for adapting to failures that occur further down the network. of course.0. Summary This setup is an excellent redundancy option.1 Description As in all the other cases where NLB multicast mode is used.3 enable add ip add ip enable add ip ip int=vlan100 ip=172.254 mask=255.

127 Clumpy2 Page 19 | AlliedWare™ OS How To Note: NLB Clustering . teamed NICs and a SwitchBlade Adaptive Load Balancing.41 clumpy2-port3.1-3. ALB provides load balancing and adapter fault tolerance.3-10.0.127 CLUSTER .port3. The NIC driver should analyse the send and receive load on each adapter and balance the rate across them based on the NIC vendor's algorithm/rule set (for example Intel base it on the destination address).16.VLAN200 vlan200 . Another alternative is Static Link Aggregation.42 virtual . If you want to provide load balancing with dual NICs in your cluster. then you may instead use LACP. ALB is a less efficient but easy-to-implement solution.VLAN100 clumpy1-port3.172.0.0.172.16.0. which load balances at the server level and is discussed in "Static Link Aggregation with a SwitchBlade" on page 34.6 . teamed NICs and a SwitchBlade This section describes Adaptive Load Balancing (ALB).172. Equipment This example uses: z z z z z Intel Pro/1000 MT Dual Port NIC SwitchBlade chassis (AT-SB4108-00) Switch Control Card (AT-SB4211) 8 port UTP trispeed line card (AT-SB4411 8-1000T(RJ45)) 2 x 32-bit PCs running Windows 2003 Enterprise Edition Network diagram Clumpy1 PC .16.2 . which is discussed in "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29.Adaptive Load Balancing.0.5-3.

0.0. flags [DF]. 172.127.42721 > <nop.0.127.0. which would only have the SwitchBlade send one packet to one port in each team/trunk.3. ttl 128.6 add vlan="200" port=3.nop.0.609752 IP (tos 0x0. set system name="Switchblade" create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 add vlan="100" port=3.610002 IP 10.0.255. ttl 63. Packet Captures The following packet capture illustrates the duplicate packets that are sent to the node. ttl 63.16.sackOK.16.timestamp (tos 0x0.0.255.0 int=vlan200 ip=10.nop.16. length: 52) .0.timestamp 244021652 0. 172.timestamp 244021652 0. turn off probes and Receive Load Balancing (RLB) on your NIC team. length:64) 172. Notice that only traffic from 10.0.127.0.http: 244021652 0> id 23054. Traffic from 172. offset 0.0.nop.127.16.16.42721 > 172.16.wscale 0> 15:59:00. This setup is therefore less efficient than LACP or Static Link Aggregation. id 23053.0.127.sackOK.609936 IP (tos 0x0.nop.0. offset 0. ttl 63.wscale 0> 15:59:00.timestamp 15:59:00.Adaptive Load Balancing.0. sackOK> 15:59:00.0.254 mask=255.nop.0.0 ip mscluster arp=172.0/24 subnet.2.16. offset 0.5-3.nop. flags [DF].16. flags [DF]. id 17555. length: 60) 10. Remember to add a static ARP for the multicast MAC.127. teamed NICs and a SwitchBlade Configuration Set the Windows 2003 NLB cluster servers to use multicast mode.nop.255.127.http: SWE [tcp sum ok] 3360891801:3360891801(0) win 5840 <mss 1460.127 (in this case clumpy1) is seen only once (marked in blue).1-3.127 int=vlan100 eth=03-bf-ac-10-00-7f port=3.254 mask=255.0. flags [none].127 is duplicated (marked in red).255.0. length: 60) 10.127 to all ports in the VLAN that belongs to the 172.wscale 0. [tcp sum ok] 1:1(0) ack 1 win 5840 id 23054. Description The SwitchBlade will send any traffic destined for 172.42721 > <nop.0. flags [DF]. offset 0.127.timestamp 0 0.127. ttl 63. [tcp sum ok] 1:1(0) ack 1 win 5840 <continued on next page> Page 20 | AlliedWare™ OS How To Note: NLB Clustering .nop.0. and to use the command enable ip mscluster.http: SWE [tcp sum ok] 3360891801:3360891801(0) win 5840 <mss 1460.http: 244021652 0> (tos 0x0.0.0.42721 > 172.1 For better performance and a less chatty network. offset 0.609757 IP (tos 0x0.nop.http > 10.610006 IP 10. id 23053.42721: S [tcp sum ok] 1420104149:1420104149(0) ack 3360891802 win 16384 <mss 1460.1 15:59:00.0.16.0.3 enable add ip add ip enable add ip ip int=vlan100 ip=172. length: 52) . including both teamed NICs on both servers.127.16.

127.0. ttl 63.timestamp 843939 244021652> Unicast Mode Setting your cluster to use unicast with ALB will lead to connection timeouts.16. offset 0.0.16.0. length: 52) 172.42721 > <nop.http > 10.timestamp (tos 0x0. offset 0.0.The two ALB ports that clumpy1 was connected to on the switch were mirrored and the mirror port was then sniffed by a linux server with tcpdump version 3. teamed NICs and a SwitchBlade <continued from previous page> 15:59:00.0.nop.3).0. ttl 63. Summary This solution provides NIC redundancy and transmit-aggregation. length: 52) F [tcp sum ok] 274:274(0) ack 397 win 6432 15:59:00. id 17557.http: 244021652 843939> id 23056.127.127. flags [DF].612321 IP 10.0.8.42721 > <nop. flags [DF].127. length: 52) F [tcp sum ok] 274:274(0) ack 397 win 6432 id 23056. The switch chip learns the cluster MAC thanks to an MS-Load balancing packet that has the Ethernet source MAC field set to the cluster MAC (this should not happen).42721: . this ruins all the effort that the clustering process had previously put into hiding the cluster MAC from the switch and will stop the switch from flooding packets that are destined for the cluster MAC.127. ttl 128.127.0. 1. 172.http: 244021652 843939> (tos 0x0. offset 0.8. flags [DF]. Unfortunately. Page 21 | AlliedWare™ OS How To Note: NLB Clustering .3 (libpcap version 0. [tcp sum ok] 397:397(0) ack 275 win 65262 <nop.Adaptive Load Balancing.nop.16.nop.612316 IP 10.0. 172.612535 IP (tos 0x0. It will not be tolerant to failures downstream but the cluster should recover quickly if one or all ports fail on a node.timestamp 15:59:00. This is easy to implement but not as efficient as link aggregation.

0/24 L2 Switch Workstation L2 Switch Private 192.127 02-bf-ac-10-00-7f VRRP on AT -9816GBs VLAN100-172.16.192.0.0/24 AT-9816GB-2 Clumpy2 PRIVATE INTERFACE clumpy1 . then you may instead use LACP.16.0. The NIC driver should analyse the send and receive load on each adapter and balance the rate across them based on the NIC vendor's algorithm/rule set (for example Intel base it on the destination address).0.16.168.127/24 Page 22 | AlliedWare™ OS How To Note: NLB Clustering .0/24 Tagged VLAN100 & VLAN200 10.192. The AT-9924SP also serves as a media adapter so that the UTP NIC can connect to the fibre AT-9816GB.0.172.0.168. two switches and VRRP This section describes Adaptive Load Balancing (ALB). Equipment z z z z z z Intel Pro/1000 MT Dual Port NIC Linux PC as the workstation 2 x 32 bit PCs with Windows 2003 Enterprise Edition AT-9924SP as L2 switch connected to workstation 2 x AT-9816GB AT-8748SL as L2 switch on private cluster LAN If only two servers are being used in the cluster. two switches and VRRP Adaptive Load Balancing.0. which load balances at the server level and is discussed in "Static Link Aggregation with a SwitchBlade" on page 34.Adaptive Load Balancing.0.0.0.0/24 AT-9816GB-1 Cluster 172.168.0.0. ALB is a less efficient but easy to implement solution.254/24 VLAN200 .41 clumpy2 .42 02-02-ac-10-00-7f virtual-172.16.0/24 Clumpy1 Cluster 172.168.172. If you want to provide load balancing with dual NICs in your cluster. Another alternative is Static Link Aggregation.0/24 10.0. which is discussed in "Link Access/Aggregation Control Protocol and a SwitchBlade" on page 29.0.0/24 10. ALB provides load balancing and adapter fault tolerance. Network diagram VLAN100 VLAN200 Private 192.16.16.0.0.16.254/24 clumpy1 to port 1 on AT-9816GB-1 port 1on AT-9816GB-2 clumpy2 to port 9 on AT-9816GB-1 port 9 on AT-9816GB-2 WORKSTATION 10.0.10.0.16.42 CLUSTER INTERFACE clumpy1 .0/24 Cluster 172.0.41 02-01-ac-10-00-7f clumpy2 . then the AT-8748SL may be left out and the nodes connected directly to each other with a crossover cable.0/24 Cluster 172.0.

0.255.255. For details. For more recent software versions.1 mask=255.255.0 add ip arp=172. you need to use the command enable ip macdisparity. we have added a static ARP entry to both switches.0. For the most recent software versions. you can see the specific multicast address to use in the NLB Manager (see "Set up the NLB Cluster" on page 66). AT -9800 #1 # # SYSTEM configuration # set system name="AT-9816GB-1" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.255.16.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged delete vlan="1" port=2 # # IP configuration # enable ip add ip int=vlan100 ip=172.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10.0. If you need a static ARP entry.254 add vrrp=200 monitoredinterface=vlan200 Page 23 | AlliedWare™ OS How To Note: NLB Clustering . In the following configurations.0.0.Adaptive Load Balancing.16.1 mask=255. see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6.0. we will only use multicast clustering here. you need to also use the command enable ip mscluster. but do not need the static ARP entry.0. two switches and VRRP Configuration Again.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.0 add ip int=vlan200 ip=10.16.

0 add ip arp=172.16. Page 24 | AlliedWare™ OS How To Note: NLB Clustering .16.255.0.0.0. two switches and VRRP AT -9800 #2 # # SYSTEM configuration # set system name="AT-9816GB-2" # # VLAN general configuration # create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 # # VLAN port configuration # add vlan="100" port=1.0.255.0 add ip int=vlan200 ip=10.255.254 add vrrp=100 monitoredinterface=vlan100 create vrrp=200 over=vlan200 ipaddress=10. turn off probes and Receive Load Balancing (RLB) on your NIC team.9 add vlan="200" port=3 add vlan="100" port=2 frame=tagged add vlan="200" port=2 frame=tagged # # IP configuration # enable ip add ip int=vlan100 ip=172.0.16.127 int=vlan100 eth=03-bf-ac-10-00-7f port=1 # # VRRP configuration # enable vrrp create vrrp=100 over=vlan100 ipaddress=172.Adaptive Load Balancing.254 add vrrp=200 monitoredinterface=vlan200 AT -9924SP # # SYSTEM configuration # set system name="AT-9924SP" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=1-8 create vlan="vlan200" vid=200 add vlan="200" port=9-12 create vlan="vlan300" vid=300 add vlan="300" port=13-16 create vlan="vlan400" vid=400 add vlan="400" port=17-20 create vlan="vlan500" vid=500 add vlan="500" port=21-24 Tip: For better performance and a less chatty network.255.0.2 mask=255.0.2 mask=255.

0.42 00-04-23-ab-37-02 Dynamic active vlan100(2) 172.16.0.16.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(2) 172.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(1) 172. Page 25 | AlliedWare™ OS How To Note: NLB Clustering . one of the NICs will answer an ARP request for the virtual IP address with its burned-in MAC.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.0.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(9) 172.0.16.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------Manager AT-9816GB-1> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 8 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-04-23-b5-bb-6f 2 dynamic 100 00-04-23-ab-37-02 9 dynamic 100 00-04-23-ab-37-03 2 dynamic 100 00-04-23-b5-bb-6e 1 dynamic 200 00-50-fc-ee-f5-13 3 dynamic This is right: Manager AT-9816GB-1> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200 10. two switches and VRRP Description In ALB mode.16.16.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.0.127 03-bf-ac-10-00-7f vlan100 172.0. This is wrong: Manager AT-9816GB-1> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200(3) 10.16.0.127 00-04-23-b5-bb-6f Dynamic active vlan100 172.255.255.0.41 00-04-23-b5-bb-6f Dynamic active vlan100(9) 172.Adaptive Load Balancing.127 00-50-fc-ee-f5-13 Dynamic active vlan200 10.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------- Example output continues on the next page. This is avoided by entering a static IP ARP entry.0.255.16.0. This behaviour will once again defeat the way clustering should work because the cluster IP ARP entry has a MAC address found in the forwarding database.255.

255 ff-ff-ff-ff-ff-ff Other active vlan200 255.16.254 00-00-5e-00-01-64 Dynamic active vlan100 172.254 00-00-5e-00-01-c8 Dynamic active vlan200 10.255.255. One port on each node's NIC is connected to port 1 of a switch. two switches and VRRP Manager AT-9816GB-1> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 10 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-09-73-50 CPU static 100 00-00-cd-09-73-50 CPU static 200 00-00-cd-09-73-50 CPU static 100 00-00-5e-00-01-64 CPU static 200 00-00-5e-00-01-c8 CPU static 100 00-04-23-b5-bb-6f 2 dynamic 100 00-04-23-ab-37-02 9 dynamic 100 00-04-23-ab-37-03 2 dynamic 100 00-04-23-b5-bb-6e 1 dynamic 200 00-50-fc-ee-f5-13 3 dynamic Manager AT-9816GB-2> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200(2) 10.255 ff-ff-ff-ff-ff-ff Other active vlan100(1) 172.41 00-04-23-b5-bb-6f Dynamic active vlan100(2) 172.16.0.0.0.127 03-bf-ac-10-00-7f vlan100(2) 172.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(1) 172.Adaptive Load Balancing.0.0.16.16. Remember that we added the static IP ARP entry to port 1 on both switches.0.127 00-50-fc-ee-f5-13 Dynamic active vlan200(2) 10. The second port on each node's NIC is attached to port 9 on the other Page 26 | AlliedWare™ OS How To Note: NLB Clustering .0.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------Manager AT-9816GB-2> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 11 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-03-00-e8 CPU static 100 00-00-cd-03-00-e8 CPU static 200 00-00-cd-03-00-e8 CPU static 100 00-04-23-b5-bb-6f 1 dynamic 100 00-04-23-ab-37-02 2 dynamic 100 00-04-23-ab-37-03 9 dynamic 100 00-04-23-b5-bb-6e 2 dynamic 100 00-00-cd-09-73-50 2 dynamic 100 00-00-5e-00-01-64 2 dynamic 200 00-00-5e-00-01-c8 2 dynamic 200 00-50-fc-ee-f5-13 2 dynamic At this point we can observe an interesting phenomenon.16.0.0.0.

0.127 time delay 0 ms Echo reply 3 from 172.0.127 ether=03-bf-ac-10-00-7f int=vlan100 port=9 Info (1005275): ARP successfully added.16.16.127 time delay 0 ms Echo reply 4 from 172.16.127 Echo reply 1 from 172. examine the following fix: Manager AT-9816GB-2> del ip arp=172.16.16.16.16.127 time delay 0 ms Echo reply 5 from 172.0.16.16.0.127 Request 5 timed-out: No reply from 172.16.127 Echo reply 1 from 172.127 To understand the reason for this.127 time delay 0 ms Echo reply 3 from 172.127 Request 2 timed-out: No reply from 172.0.Adaptive Load Balancing.127 Request 1 timed-out: No reply from 172.0.0.127 Request 4 timed-out: No reply from 172.16.127 time delay 0 ms Echo reply 4 from 172.127 time delay 0 ms Example output continues on the next page: Page 27 | AlliedWare™ OS How To Note: NLB Clustering .0. Manager AT-9816GB-2> add ip arp=172.16.127 time delay 0 ms Echo reply 2 from 172.0.0. Manager AT-9816GB-2> ping 172.0.127 time delay 0 ms Echo reply 2 from 172.0.0.16.127 time delay 0 ms Manager AT-9816GB-2> ping 172.0.0.16.16.0.16.16. one of the switches now cannot ping the virtual server address: Manager AT-9816GB-1> ping 172. two switches and VRRP switch.16.16.127 time delay 0 ms Echo reply 5 from 172.0.0.0.16.127 Request 3 timed-out: No reply from 172. In the following example.0.127 Info (1005272): ARP entry successfully deleted.

Adaptive Load Balancing, two switches and VRRP

Manager AT-9816GB-2> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200 10.0.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(2) 172.16.0.41 00-04-23-b5-bb-6f Dynamic active vlan100(9) 172.16.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(9) 172.16.0.127 03-bf-ac-10-00-7f vlan100 172.16.0.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.255.255.255 ff-ff-ff-ff-ff-ff Other active ---------------------------------------------------------------------------Manager AT-9816GB-2> sh swi fdb Switch Forwarding Database (hardware) --------------------------------------------------------------------------Total Number of Entries = 7 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-03-00-e8 CPU static 100 00-00-cd-03-00-e8 CPU static 200 00-00-cd-03-00-e8 CPU static 100 00-04-23-b5-bb-6f 2 dynamic 100 00-04-23-ab-37-02 9 dynamic 100 00-00-5e-00-01-64 2 dynamic 200 00-00-5e-00-01-c8 2 dynamic

The static IP ARP entry has now been set to port 9 instead of port 1. The next thing to understand is that an IP ARP entry will be used by the switch when it does software lookups. The switch consults its IP ARP table when it pings an address and in the first instance sends pings to port 1 only. From this information and our observation in the example above, we conclude that changing the static IP ARP port worked because under ALB one of the NIC ports is not listened to by the NIC driver for some arbitrary rule at the NIC driver level. ALB should be able to assume that traffic is flooded to all ports and hence the behaviour is not necessarily incorrect. Data that is switched through the switch (as opposed to being sent from the switch CPU) will be flooded. The port lookup for packets going through the switch is done by consulting the FDB, which does not have an entry for the cluster MAC address.

Summary
This solution provides NIC redundancy and transmit-aggregation. It will not be tolerant to failures downstream but the cluster should recover quickly if one or all ports fail on a node.

Page 28 | AlliedWare™ OS How To Note: NLB Clustering

Link Access/Aggregation Control Protocol and a SwitchBlade

Link Access/Aggregation Control Protocol and a SwitchBlade
The following configuration demonstrates how to increase your NLB bandwidth by teaming two NICs. We will use Link Aggregation Control Protocol (LACP) to maximise this bandwidth. Equipment
z z z z z

Intel Pro/1000 MT Dual Port NIC SwitchBlade (AT-SB4108-00) Switch Control Card (AT-SB4211) 24 port trispeed line card (AT-SB4412) 2 x 32-bit PCs running Windows 2003 Enterprise Edition LACP requires a “K revision” line card in the SwitchBlade. Any control card should be okay.

Note:

Network diagram

Clumpy1 PC - VLAN200 vlan200 - port3.3-10.0.0.127

CLUSTER - VLAN100 clumpy1-port 3.1-3.2 - 172.16.0.41 clumpy2-port 3.9-3.10 - 172.16.0.42 virtual - 172.16.0.127 Clumpy2

Note:

The card in the SwitchBlade in the diagram above shows 32 ports rather than the 24 that our setup actually uses.

Page 29 | AlliedWare™ OS How To Note: NLB Clustering

Link Access/Aggregation Control Protocol and a SwitchBlade

Configuration
First, set up LACP on your SwitchBlade. At the hardware level you will need a “K revision” line card, such as a 24 port trispeed card. Set the Windows 2003 NLB cluster servers to use multicast mode. Remember to add a static ARP for the multicast MAC, and to use the command enable ip mscluster.
set system name="Switchblade" create vlan="vlan100" vid=100 create vlan="vlan200" vid=200 add vlan="100" port=3.1-3.2,3.9-3.10 add vlan="200" port=3.3 enable add ip add ip enable add ip ip int=vlan100 ip=172.16.0.254 mask=255.255.255.0 int=vlan200 ip=10.0.0.254 mask=255.255.255.0 ip mscluster arp=172.16.0.127 int=vlan100 eth=03-bf-ac-10-00-7f port=3.1

ena lacp delete lacp port=3.3-3.8,3.11-3.24 set lacp port=3.1-3.2 adminkey=100 set lacp port=3.9-3.10 adminkey=200

Your clustered servers' NICs must support Link Aggregation (802.3ad) mode. This will either be a part of the driver set already or you will need to download the latest drivers from whoever your NIC vendor is. Set up link aggregation on the NICs last. Note: It may take a few moments after Windows starts up for NLB to begin.

Description
When your setup is working you should see results similar to the following:

Manager Switchblade> sh ip arp Interface IP Address Physical Address ARP Type Status ---------------------------------------------------------------------------vlan200(3.3) 10.0.0.127 00-50-fc-ee-f5-13 Dynamic active vlan200 10.0.0.255 ff-ff-ff-ff-ff-ff Other active vlan100(3.2) 172.16.0.41 00-04-23-b5-bb-6e Dynamic active vlan100(3.10) 172.16.0.42 00-04-23-ab-37-02 Dynamic active Static active vlan100(3.1) 172.16.0.127 03-bf-ac-10-00-7f vlan100 172.16.0.255 ff-ff-ff-ff-ff-ff Other active vlan200 255.255.255.255 ff-ff-ff-ff-ff-ff Other active

Page 30 | AlliedWare™ OS How To Note: NLB Clustering

Link Access/Aggregation Control Protocol and a SwitchBlade

Manager Switchblade> sh swi fdb inst=3.0 Switch Forwarding Database (hardware) Blade Switch Instance: 3.0, (AT-SB4412 24-10/100/1000T(RJ45)) --------------------------------------------------------------------------Total Number of Entries = 8 --------------------------------------------------------------------------VLAN MAC Address Port Status --------------------------------------------------------------------------1 00-00-cd-05-01-14 CPU static 100 00-00-cd-05-01-14 CPU static 200 00-00-cd-05-01-14 CPU static 100 00-04-23-ab-37-02 3.10 dynamic 100 00-04-23-b5-bb-6e 3.2 dynamic 100 00-04-23-b5-bb-6f 3.2 dynamic 100 00-04-23-ab-37-03 3.10 dynamic 200 00-50-fc-ee-f5-13 3.3 dynamic Manager Switchblade> sh swi tab=ip inst=3.0 Blade Switch Instance: 3.0, (AT-SB4412 24-10/100/1000T(RJ45)) CXE IP route table summary information: Total number of entries = 18 ---------------------------------------------------------------------------CXE IP route table: Index Seg Network Nexthop ToProc VRP VID Hits PV Prio Perm ---------------------------------------------------------------------------0 101 192.168.255.1 00-00-cd-05-00-00 0 0 4094 0 0 0 1 1 101 172.16.0.127 03-bf-ac-10-00-7f 0 6 100 0 0 0 1 2 101 172.16.0.42 00-04-23-ab-37-02 0 6 100 0 0 0 1 3 101 10.0.0.127 00-50-fc-ee-f5-13 0 5 200 0 0 0 1 4 101 172.16.0.41 00-04-23-b5-bb-6e 0 6 100 0 0 0 1 5 102 192.168.255.12 00-00-cd-05-00-58 0 0 4094 0 0 0 1 6 102 192.168.255.127 00-00-cd-05-00-58 0 0 4094 0 0 0 1 7 102 192.168.255.140 00-00-cd-05-00-58 0 1 4093 0 0 0 1 8 102 192.168.255.255 00-00-cd-05-00-58 0 1 4093 0 0 0 1 9 102 10.0.0.254 00-00-cd-05-01-14 0 5 200 0 0 0 1 10 102 10.0.0.255 00-00-cd-05-01-14 0 5 200 0 0 0 1 11 102 172.16.0.254 00-00-cd-05-01-14 0 6 100 0 0 0 1 12 102 172.16.0.255 00-00-cd-05-01-14 0 6 100 0 0 0 1 13 109 192.168.255.0 00-00-cd-05-00-58 0 0 4094 0 0 0 1 14 109 192.168.255.128 00-00-cd-05-00-58 0 1 4093 0 0 0 1 15 110 10.0.0.0 00-00-cd-05-01-14 0 5 200 0 0 0 1 16 110 172.16.0.0 00-00-cd-05-01-14 0 6 100 0 0 0 1 17 134 0.0.0.0 00-00-cd-05-00-00 0 0 4094 0 0 0 1

Page 31 | AlliedWare™ OS How To Note: NLB Clustering

...9-3....16.. offset 0. offset 0.16...0.208807 IP (tos 0x0..0011....16. 1000 Mbps Ports in Trunk ... ttl 128..... ttl 128.127 > (tos 0x0. Both ports of the node are associated with the same port..208993 IP 84) 172.00-04-23-b5-bb-6e.....0.127: icmp 64: echo 16:45:43....127: icmp 64: echo 16:45:44.127: icmp 64: echo reply seq 57088 (tos 0x0.(ffff.0011.... This does not break trunking because traffic will still be balanced across all the ports in the trunk.16..00.10 LAG ID: [(8000.. lacp30: Speed .(ffff.0.127: icmp 64: echo 0..0.. id 6872.. id . This is a designed feature of trunking. In this setup..2 LAG ID: [(8000. length: 10. ttl 63... flags [DF]. one port ingress) but as traffic increases it is more likely that the SwitchBlade and Windows will balance traffic Page 32 | AlliedWare™ OS How To Note: NLB Clustering ..1 or 3.. offset 0.. length: 84) 10 request seq 57600 and only echo replies on port 3. id 6921.2..0004... On port 3.0005. id ......0. length: 84) 10 request seq 57344 0.0.. id 6922. length: 84) 10 request seq 57088 0.0...00-04-23-ab-37 02.127 > 16:47:28. Another two (00-04-23-b5-bb-6f and 00-04-23-ab-37-03) appear only in the FDB. they are the second ports of each node.0. Two of the MAC addresses in the FDB appear in the IP ARP table..00. lacp29: Speed ..16.208795 IP (tos 0x0...0.0... in this case it was over port 3.208996 IP 84) 172.0000). flags [DF]. flags [DF]..1 you should see only echo requests: 16:45:42.2).. length: 10. flags [DF].127: icmp 64: echo reply seq 57600 The important point is that the ICMP echo request is sent to one port only in the trunk. offset 0.0. ping the cluster address.0.Link Access/Aggregation Control Protocol and a SwitchBlade Manager Switchblade> sh lacp trunk LACP Dynamic Trunk Group Information ----------------------------------------------------------------------------Trunk group name .16. To check that the LACP trunk is working.0..127 > 172.127: icmp 64: echo reply seq 57344 (tos 0x0..0.. flags [DF]. The NLB server will decide which port to send the reply over (either 3.127 > 172. it never appears in the FDB.2: 16:47:26.. ttl 63.0000)] Trunk group name .127 > 172. where all the MAC addresses in a trunk are associated in FDB with one port in that trunk.. 1000 Mbps Ports in Trunk .0000).0.00-00-cd-05-01-14. 3. offset 0. You may see the same behaviour with HTTP requests (one port egress. length: 10..208777 IP (tos 0x0. Note that both servers may respond to the ICMP packet depending on the NLB rules so your client could report duplicate replies.00-00-cd-05-01-14.00...127 > 16:47:27.00. 3.1-3. ttl 128..0.0. id . flags [DF]. with minimal load.208992 IP 84) 172.. ttl 63... offset 0.0. traffic may be seen to only be received on one port in the trunk and reply sent out the other port.0000)] Notice that the MAC address 03-bf-ac-10-00-7f only appears in the IP ARP table.

1 would appear as such: 16:20:13.nop.0.509249 IP (tos 0x0.0.timestamp 235508243 0. offset 0.0. The key is that you should never see duplicate unicast packets over two ports in the same trunk.timestamp 235508244 0> 16:20:13.Link Access/Aggregation Control Protocol and a SwitchBlade across both ports. ttl 128.nop.0.nop.timestamp 235508245 23725> Port 3. Page 33 | AlliedWare™ OS How To Note: NLB Clustering .http: SWE 3617082630:3617082630(0) win 5840 <mss 1460.http: F 273:273(0) ack 397 win 6432 <nop. The benefits of that may only be slight.sackOK> 17:10:27.timestamp 22780 235809610> 17:10:27.127.http > 10.0.http: P 0:273(273) ack 1 win 5840 <nop. flags [DF].511880 IP (tos 0x0. length: 52) 172.16.127.127.16.127.0.0. offset 0.127.16.127.0.16. offset 0.856129 IP 10. Summary This method will provide efficient fault tolerance and load balancing.wscale 0.127. flags [none]. [tcp sum ok] 397:397(0) ack 275 win 65262 <nop. However. id 11983.0. LACP will give you management capabilities over link aggregation. flags [DF].0.http > 10.sackOK.510746 IP (tos 0x0.127.858012 IP 10.16.0.timestamp 0 0.42628: S [tcp sum ok] 1493187854:1493187854(0) ack 2433262544 win 16384 <mss 1460.wscale 0> 16:20:13. For more information about LACP on Allied Telesis switches.127.16.42235 > 172. id 11982.0. length: 447) 172. The alternative is Static Link Aggregation.0.857435 IP 10. see your switch’s Software Reference.42235 > 172. Cluster redundancy is preserved if the two nodes each connect to two 24-port trispeed cards.32002.0. ttl 128. ack 4292649670 win 5840 <nop.timestamp 235508244 0> 16:20:13.0.3ad" on page 36. which does not rely on special hardware and software to run.16.nop.nop.0. id 11984.42235 > 172.127.860174 IP 10. better use of bandwidth is gained.nop.nop.nop. because hot swapping one line card will leave the other node(s) up.0.127. Port 3. ttl 128.nop. consult the IEEE standard 802. This setup will not give an individual node redundancy if a line card is hot swapped.2 may appear as such: 17:10:27.127.timestamp 22780 235809610> Please see the section "Special Note About LACP and 802.0.127.0.nop.127.0.42628: FP 1:396(395) ack 274 win 65262 <nop.http: . length:64) 172.42628: .42235 > 172.0. For more information on just LACP.nop.http > 10.

41 clumpy2-port 4.1-3.VLAN200 vlan200 .172.0.port3.VLAN100 clumpy1-port 3.Static Link Aggregation with a SwitchBlade Static Link Aggregation with a SwitchBlade Static link aggregation was developed to increase throughput between switches.127 Private CLUSTER . this is a viable option.0.192.0/24 Cluster Clumpy2 Page 34 | AlliedWare™ OS How To Note: NLB Clustering .0.2 . Equipment z z z z z z Intel Pro/1000 MT Dual Port NIC Linux PC as the workstation 2 x 32 bit PCs with Windows 2003 Enterprise Edition SwitchBlade chassis (AT-SB4108-00) Switch Control Card (AT-SB4211) 2 x 8 port UTP trispeed line card (AT-SB4411 8-1000T(RJ45)) Network diagram Cluster Clumpy1 PC .1-4.127 private .16.172.0.3-10.0.2 .16.42 virtual .3ad.168.16. Because Allied Telesis switches support 802.172.0.

255. Please see the section "Special Note About LACP and 802.1-3.1-3.255.1 # # SWITCH configuration # create switch trunk=clumpy2 port=4. By putting each node on its own line card. NLB multicast mode must be used in the cluster.16. Page 35 | AlliedWare™ OS How To Note: NLB Clustering . Therefore. Summary This method will provide efficient fault tolerance and load balancing with the added bonus of working on any Allied Telesis SwitchBlade line card.255.16. Cluster redundancy is preserved because hot swapping one line card will leave the other node(s) up.0 add ip int=vlan200 ip=10. you cannot connect two ports on the same node to two different cards and expect to be able to trunk them.254 mask=255.0 enable ip mscluster add ip arp=172. However. This setup will not give an individual node redundancy if a line card is hot swapped.0.2.0.3ad trunking demands that both ports in a trunk be on the same switch instance.3ad" on page 36.254 mask=255.255. only cluster redundancy is gained.Static Link Aggregation with a SwitchBlade Configuration # # SYSTEM configuration # set system name="Switchblade" # # VLAN general configuration # create vlan="vlan100" vid=100 add vlan="100" port=3.0.1-4.4.3 # # IP configuration # enable ip add ip int=vlan100 ip=172.0.2 speed=1000m create switch trunk=clumpy1 port=3.1-4.127 int=vlan100 eth=03-bf-ac-10-00-7f port=3. better use of bandwidth is gained.2 create vlan="vlan200" vid=200 add vlan="200" port=3.2 speed=1000m Description 802.

Page 36 | AlliedWare™ OS How To Note: NLB Clustering . An IP packet may be sent by the SwitchBlade to port 1 of the node and the reply sent to the SwitchBlade from port 2 on the node.3ad The SwitchBlade will send IP packets destined for the virtual address to one port in each trunk. which is a decision the NIC driver makes based on its own algorithm.Static Link Aggregation with a SwitchBlade Special Note About LACP and 802. Put simply. depending on the SwitchBlade software release and control cards. Under layer 4 hashing. Layer 4 hashing may or may not be on by default. The use of layer 2 and/or layer 3 hashing without layer 4 will cause the SwitchBlade to send all IP traffic to the same port on the node every time for packets with the same source and destination IP. The node will continue to send return traffic based on its own algorithm. Therefore. the SwitchBlade will use TCP port numbers to determine which physical port in a trunk to send traffic over. you can see that the SwitchBlade control card is using layer 2 and layer 3 hashing. An IP packet may be sent by the SwitchBlade to port 2 of the node and the reply sent to the SwitchBlade again from port 2 on the node. layer 4 hashing will give a randomly balanced traffic shape for packets with the same source and destination IP. all traffic cannot be seen on one port alone but sniffing both ports should reveal the entire traffic flow with no duplicate packets. The port that the SwitchBlade sends traffic to is dependant on the layer(s) that you choose to involve in the hashing calculation that determines the egress port for a given packet. Layer 2 and layer 3 hashing is normally on. The node's NIC driver will decide to send return traffic via one NIC. In the following example output from the command show switch.

..... None Ports mirroring on Rx ...... Ageingtime ....... Disabled UpTime .. DISABLED Mirror port ..... None Ports mirroring on Tx ............................... None Ports mirroring on Tx .. L2 L3 L3 Test mode ..255................... MODE1 -----------------------------------------------------------------------Switch Configuration -----------------------------------------------------------------------Silicon Revision .255..... 106496 Switch Address . Disabled UpTime ........ Basic Basic Flow Control Length . Basic Basic Flow Control Length ..... 0 Name of Interface(s) ............... 9............... 0 CAM size ( entries ) .. 00-00-cd-05-01-14 Learning ...................... Alive IP Address ............................. 192.0 Blade Instance Health .... 300 VLAN classification .. ON Ageing Timer .............. None Ports mirroring on Both ............... 9 Blade Switch Instance . Ageingtime . ON Number of Fixed Ports ...... ON Number of Fixed Ports ...... ffffff00 PCI errors . 9 Blade Switch Instance .............. Alive IP Address ...............168.......... None Pause Flow Control Mode ............ 0 Mirroring ......... 300 VLAN classification .. To be defined STP Forwarding ....................0 IPX Address .............. None Ports mirroring on Both .................... CXE K1 Switch Type ............ ON Ageing Timer .. 00-00-cd-05-01-14 Learning ......... Control Card Blade Position .................... 0 Mirroring ............ CXE K1 Switch Type ....... 0 Name of Interface(s) .............................1 Blade Instance Health .... DISABLED Mirror port ... L2 L3 L3 Test mode ................. To be defined STP Forwarding ..... 04:09:55 Hashingfield ................... 0 Number of Uplink Ports ............ None Pause Flow Control Mode . None Ports mirroring on Rx .............168.........0 IPX Address . ffffff00 PCI errors ... 10 Number of WAN Interfaces .... 192................. Control Card Blade Position .........................Static Link Aggregation with a SwitchBlade Switch Configuration -----------------------------------------------------------------------Silicon Revision ............ 0 CAM size ( entries ) .............. 9...... 04:09:55 Hashingfield ........... MODE1 Page 37 | AlliedWare™ OS How To Note: NLB Clustering .... 0 Number of Uplink Ports . 10 Number of WAN Interfaces .......... 106496 Switch Address .....................

1-4.0 inst=4.0 instance=9.0 inst=3.2 speed=1000m create switch trunk=clumpy1 port=3. to avoid problems with cards that have pre K-revision silicon.1 This results in the example output from the command show switch as shown on the next page. turn on layer 4 hashing. Also.0 inst=4.2 speed=1000m set switch port=3.Static Link Aggregation with a SwitchBlade To force the SwitchBlade to mix traffic between ports on the same node.0 inst=9.1-3. Page 38 | AlliedWare™ OS How To Note: NLB Clustering .1 mirror=both disable disable disable disable ena ena ena ena swi swi swi swi switch switch switch switch hash=l4 hash=l4 hash=l4 hash=l4 hash=l3 hash=l3 hash=l3 hash=l3 instance=9.0 inst=9. consider turning off layer 3 hashing on all control and line cards: enable switch mirror create switch trunk=clumpy2 port=4.1 inst=3.

. 10 Number of WAN Interfaces ... 0 Number of Uplink Ports ...... Disabled UpTime ....... 0 Name of Interface(s) ................ 00-00-cd-05-01-14 Learning .... Alive IP Address ......... 9 Blade Switch Instance ..... CXE K1 Switch Type .... ffffff00 PCI errors ................ 0 Name of Interface(s) ............... 10 Number of WAN Interfaces ............................. Ageingtime .......... 0 CAM size ( entries ) ............. DISABLED Mirror port .......... CXE K1 Switch Type ............ Control Card Blade Position ....... 192................... None Ports mirroring on Both ............................ None Ports mirroring on Rx .......... 106496 Switch Address .......... 0 CAM size ( entries ) ........ None Ports mirroring on Both ... MODE1 -----------------------------------------------------------------------Switch Configuration -----------------------------------------------------------------------Silicon Revision .... None Ports mirroring on Tx ........ None Ports mirroring on Tx ........................ 0 Mirroring ...... Ageingtime .......... ON Number of Fixed Ports .......... None Ports mirroring on Rx .........0 IPX Address ..... 00-00-cd-05-01-14 Learning ................. 9............ DISABLED Mirror port ...255......... 04:33:37 Hashingfield ..............................168.................... To be defined STP Forwarding ..... None Pause Flow Control Mode ............ L2 L4 L3 Test mode . 106496 Switch Address ....... 300 VLAN classification ............... 9 Blade Switch Instance ................. None Pause Flow Control Mode ............ L2 L4 L3 Test mode .................1 Blade Instance Health ....... Basic Basic Flow Control Length .......... 0 Number of Uplink Ports .....0 IPX Address ..... 04:33:37 Hashingfield .. ON Number of Fixed Ports .... ffffff00 PCI errors ....... 0 Mirroring ............ ON Ageing Timer ...........0 Blade Instance Health .... Control Card Blade Position ....... 300 VLAN classification ....... Basic Basic Flow Control Length .. 192..... Disabled UpTime ....168...Static Link Aggregation with a SwitchBlade Switch Configuration -----------------------------------------------------------------------Silicon Revision .................. 9... To be defined STP Forwarding .............. Alive IP Address .. MODE1 Page 39 | AlliedWare™ OS How To Note: NLB Clustering .255..................... ON Ageing Timer .

it uses AR750S routers.Switch Fault Tolerance (SFT).How Allied Telesis recommends NLB be implemented at layer 3 How Allied Telesis recommends NLB be implemented at layer 3 If you are interested in teamed NICs for redundancy. The following method provides the best level of redundancy and failover at layer 2 and layer 3. Network diagram client network 192. aggregation or otherwise then the mode you select is entirely your decision.0/24 eth0 VRRP eth0 AR750S router port1 vlan100 AR750S router port1 vlan100 AT-9924T switch AT-9924T switch 172.eps Page 40 | AlliedWare™ OS How To Note: NLB Clustering .0/24 Clumpy1 server Clumpy2 server nlb-rec.100.16. This example will use the most commonly implemented teaming . The layer 2 switches can be any Allied Telesis switch.0. Layer 3 switches will also work.168. but other routers will also work. At layer 3. as long as each layer 3 switch only has one port connected to one layer 2 switch.

0.254 add vrrp=2 monitoredinterface=eth0 The switches can be configured with or without an IP address but they are not configured with a static IP ARP.127 int=vlan100 eth=01-00-5e-7f-00-7f port=1 In the above configuration.16. Add an IP interface and route information to both of the AR750S routers.0/24 network to reach the NLB server network. You should also add a VRRP interface for the 172.0/24 subnet. This network can be easily scaled up with more switches and servers. An IGMP querier configuration is required on at least one switch. enable add ip add ip add ip ip interface=eth0 ip=192.0.16. ideally on both for redundancy: enable ip igmp ena ip igmp interface=vlan100 Page 41 | AlliedWare™ OS How To Note: NLB Clustering .255. you need to use the command enable ip macdisparity. The following configuration is applied to both AR routers and includes a monitored interface so that the VRRP master will change if the status of vlan100 or eth0 changes. the switches do not need any configuration at all.0 arp=172. For the most recent software versions. we have added a static ARP entry to the VLAN interface to which the server's layer 2 network connects. IGMP snooping is enabled by default on Allied Telesis switches.100.0. but do not need the static ARP entry.168.100.x mask=255. For details. see "Accepting ARP entries that have a multicast MAC address and a unicast IP address" on page 6. A switch that supports IGMP snooping is all that is actually required.254 add vrrp=1 monitoredinterface=vlan100 create vrrp=2 over=vlan100 ipaddress=172. you need to also use the command enable ip mscluster. At the most basic level. or scaled down to one switch and two servers each with a single or dual NIC.0.16. you can see the specific multicast address to use in the NLB Manager (see "Set up the NLB Cluster" on page 66).168. The switches should not be configured with multiple VLANs. Then create a VRRP interface to be the next hop for the 192. If you need a static ARP entry.255. enable vrrp create vrrp=1 over=eth0 ipaddress=192.16.168.100.255. For more recent software versions.0 interface=vlan100 ip=172.How Allied Telesis recommends NLB be implemented at layer 3 Configuration The AR750S is in charge of routing traffic between the rest of the network and the L2 network that the servers are on. This will not work properly if a switch is connected to more than one router.255. It is important to note that only one switch connects to the AR750S.x mask=255.

as shown in the following figure. but select “IGMP multicast”.How Allied Telesis recommends NLB be implemented at layer 3 Configure the NLB servers as normal. Page 42 | AlliedWare™ OS How To Note: NLB Clustering .

but with the FAT file system. Installation of first NLB Server . Set up the private node-to-node ethernet connection and assign each node an IP. Copy the i386 directory to your hard drive if you have space.0.0/24 network for the private LAN.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Install Windows 2003 Defaults are okay during the install but you must ensure you use NTFS. then convert the drive now.clumpy1 X Check that your drive partition uses an NTFS file system If your Windows 2003 installation partition file system is not NTFS. This example uses the 192. You cannot proceed without NTFS. If you already have Windows 2003 installed. When the PC reboots it will analyse the hard drive and then convert the file system to NTFS without destroying data (hopefully). then open up a command prompt (otherwise known as a MSDOS window) and type “convert c: /FS:NTFS /X” and then reboot.168. Before you begin. Page 43 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Install a domain controller (DC) if you do not have one Windows 2003 clustering requires DNS lookups to work and in the case of Clustering requires an account within the domain to be set up. Run dcpromo. follow the prompts as per the screenshots and then reboot afterwards. It may be possible to avoid setting up a DC (you could manually set up DNS perhaps) but it is easier to have a domain controller in a Windows environment. X Set up a Domain Controller Page 44 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 45 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 46 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 47 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 48 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 49 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 50 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 51 | AlliedWare™ OS How To Note: NLB Clustering .

Do not configure this here but make sure it is present and checked. Page 52 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Configure interfaces Note: In the picture above the Network Load Balancing protocol appears in the list.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 53 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 54 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 55 | AlliedWare™ OS How To Note: NLB Clustering .

Clumpy1 is already there because that is the server we are using now.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Create extra entries for the Cluster in DNS Right click in the right hand window and select “New Host (A)”. the gateway and the virtual IP entry for the cluster. Page 56 | AlliedWare™ OS How To Note: NLB Clustering . We need to add clumpy2.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster You get the following warning if the 172. This example will cover the creation of a reverse zone next Page 57 | AlliedWare™ OS How To Note: NLB Clustering . It is probably better to create the reverse lookup zone before adding PTR records.0/24 (reverse) zone hasn't been created yet.16. You can ignore this and manually add an entry later to the reverse zone.0.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Create a reverse zone for the test.nz domain Page 58 | AlliedWare™ OS How To Note: NLB Clustering .co.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 59 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 60 | AlliedWare™ OS How To Note: NLB Clustering .

clumpy2.127): Page 61 | AlliedWare™ OS How To Note: NLB Clustering . Here is the example for the name “cluster” (for the cluster IP 172.16. Right click in the right hand pane of the windows above and select “New Pointer (PTR)”.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Manually add the reverse entry for each name (clumpy1. cluster and gateway) 1.0.

Click “Browse” in the window above and find the name that you want the reverse entry to relate to. Page 62 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 2.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 63 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 3. Do the same for the gateway address (have you added this to the forward zone yet?): Page 64 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 65 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster . Start the wizard to create a new cluster: Cluster > New Page 66 | AlliedWare™ OS How To Note: NLB Clustering . Start the NLB Manager from: Start > Administrative Tools > Network Load Balancing Manager 2. X Set up the NLB Cluster 1.

Page 67 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 3. Change the filtering mode to affinity “None”.

to ensure basic IP connectivity. On unconfigured interfaces your clumpy2 will have IP addresses of the form 169.254. For the second node you enter the second NIC's private network IP address in the “Host” box. For the second node it may take some time after pressing the “Connect” button for the interfaces to appear in the available interfaces box. which you should do after you have added the first node. All you need is an IP on the private NIC of clumpy2 and a physical connection.0.0. The box above illustrates adding the second node. Page 68 | AlliedWare™ OS How To Note: NLB Clustering . Notice that there are already IP addresses assigned to the other interfaces of clumpy2. Wait for as long as it takes and then select the cluster side IP and click on Next. The box below illustrates that for the second node you must change the priority to 2. Enter the local IP in the “Host” box and click on the “Connect” button. Be patient. If you have not configured interfaces other than the private NIC of clumpy2 that is okay.0 or perhaps 0.0.0.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster 4.

Be patient.Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster It takes quite a long time sometimes for the nodes to converge. Page 69 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Add IIS to enable node as a Web server 1. From the Control Panel select “Add or Remove Programmes” and then “Add/Remove Windows Components”. Page 70 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 71 | AlliedWare™ OS How To Note: NLB Clustering .

clumpy 2 X Convert your file system to NTFS if it is not already Page 72 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Installation of second NLB server .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster X Install the Domain Controller on clumpy2 Don't set the second server as a backup DNS server—that's too much extra bother at this stage. Page 73 | AlliedWare™ OS How To Note: NLB Clustering . Use the private network for DNS in the adapter settings.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster If you get the message above. go back and set up a DNS server through the Network Connections settings in the Control Panel Page 74 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 75 | AlliedWare™ OS How To Note: NLB Clustering .

in this example. you can see that manager was used and then subsequently rejected a few steps down the line. Page 76 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Use the Administrator account.

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 77 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 78 | AlliedWare™ OS How To Note: NLB Clustering .

Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Page 79 | AlliedWare™ OS How To Note: NLB Clustering .

You do not need to do anything to clumpy2 for it to join the NLB cluster. Also. The earlier use of the NLB Manager on clumpy1 should have configured clumpy2 accurately. you may need to spend time convincing Windows 2003 to use the private NIC for cluster heartbeat messages. Page 80 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 1 —Quick Start Guide to setting up a Windows 2003 NLB cluster Install Windows IIS if you haven't already (refer to the clumpy1 setup). but working cluster. You will need to make many tweaks to the NLB policy to suit your environment. Use the NLB Manager to do that. This document should at least leave you with a basic.

SwitchBlade and AT -9800 On a Switchblade or AT-9800 series switch. Manager > sh swi tab=l2broadcast CXE L2 multicast and VLAN broadcast table: Index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 MAC 55-55-55-55-55-55 01-00-5e-7f-00-7f 01-00-5e-00-01-18 01-00-5e-7f-ff-fe 01-00-5e-00-01-16 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 01-00-5e-00-00-00 33-33-00-00-00-00 33-33-00-00-00-00 33-33-00-00-00-00 01-00-00-00-00-00 01-00-00-00-00-00 01-00-00-00-00-00 VID 1 1 1 1 1 1 2 28 1 2 28 1 2 28 1 2 28 L2MC-Group MC Hits 4096 5 6 7 9 4097 4098 4124 1 1 1 2 3 4 4097 4098 4124 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 5 6 7 9 1 2 26 1 1 1 2 3 4 1 2 26 PV Prio Perm DIS DM Expire 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Segment L2MC-Group TagMask 77 77 77 77 77 83 85 85 85 87 87 87 87 87 0 5 6 7 9 1 2 3 4 8190 8189 4097 4098 4124 00000000 00000000 00000000 00000000 00000000 80000000 80000000 80000000 80000000 00000000 00000000 00000000 00000000 00000000 OutportMask Hits 00000000 00000021 00000021 00000021 00000008 80000000 8000003b 80000000 80000800 00000000 00000000 0000003b 00000000 00000800 0 5 6 7 9 1 2 3 4 0 0 0 0 0 BCSC-Id BCSC-Enable Perm 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Page 81 | AlliedWare™ OS How To Note: NLB Clustering .Appendix 2: IGMP snooping port membership information in switch memory tables Appendix 2: IGMP snooping port membership information in switch memory tables The simplest way to find out which ports belong to a group is to use the command show igmpsnooping. Instructions on how to interpret the output is after this example. Additionally. This section describes how to interpret the table output. as the following example shows for a 12-port AT-9800 switch. use the following command to match ports to multicast groups (represented by the multicast MAC address): show switch memory=l2broadcast This produces two tables. as described in "Multicast mode with the IGMP option selected" on page 5. you can find specific information about which ports belong to a group in the layer 2 multicast tables.

. Each 1 corresponds to a port that belongs to the target group. Select the row that has the MAC address of the L2 multicast group that you are interested in. 5. In that row. Convert that hexadecimal number to binary (10001 in the example). ports 1 and 5 are members of the group 01-00-5e-7f-00-7f in VLAN1. Therefore. Note the number in its “L2MC Group” field and the VID. This is a series of hexadecimal digits (0x21 in the example). AT-8600 On these switches. In the first table. 2. note the OutportMask field. 2. . in this example. use the following command to match ports to multicast groups (represented by the multicast MAC address): show switch memory=marl This produces a single line of output per group. The right-hand most digits each represent 4 concurrent ports (in the range 1-4 and 5-8 in the example. 0 0000 - ports 12-9 0 0000 - ports 8-5 2 0010 6 ports 4-1 1 0001 1 hexadecimal binary port number Rapier. Mac=01-00-5e-00-01-18 VLAN=1 ports=00000011 utports=03fffff7 cosDst=0 2. Counting from the right.0 3/(1-255) ----------------------------------------------------------------------------1. note the positions of the digit 1 in the binary number. Find the row in the second table that has the same value in its L2MC Group field. in this example. Manager > sh swi memory=marl Multicast Table-0. Page 82 | AlliedWare™ OS How To Note: NLB Clustering . In that row. so this is VLAN1. 3. as the following example shows. select the row that has the MAC address of the L2 multicast group that you are interested in. 4. the L2MC Group field value is 5 and the VID is 1. because they are the two right-most digits).. Mac=01-00-5e-7f-00-7f VLAN=1 ports=00000011 utports=03fffff7 cosDst=0 3. 3. In the example in bold above. 4. Convert each hexadecimal digit to a 4-digit binary number (0010 and 0001 in the example). Counting from the right. ports 1 and 6 are members of the group 01-00-5e-7f-00-7f in VLAN1. note the “ports=” value. Mac=01-00-5e-7f-ff-fe VLAN=1 ports=00000011 utports=03fffff7 cosDst=0 To work out which ports are members of a group: 1. This is a hexadecimal number that indicates the member ports (0x11 in the example in bold above). Each 1 corresponds to a port that belongs to the target group.Appendix 2: IGMP snooping port membership information in switch memory tables To work out which ports are members of a group: 1. as the following table shows. note the positions of the digit 1 in each binary number. Therefore. AT-8800.

Information in this document is subject to change without notice.6 2 01-00-5e-7f-00-7f 1 4102 0 2 Ports = 2. in the United States and other countries.x900.11 Asia-Pacific Headquarters | 11 Tai Seng Link | Singapore | 534182 | T: +65 6383 3832 | F: +65 6383 3830 www. and product designs that are trademarks or registered trademarks are the property of their respective owners. Allied Telesis is a trademark or registered trademark of Allied Telesis. as the following example shows. It is much easier to tell which groups are in each multicast group than it is with the previous switch series. C613-16073-00 REV B . logos. AT-9900. Inc. AT-8948 On these switches.6 4 01-00-5e-00-01-16 1 4096 0 1 Ports = 4 5 03-bf-ac-10-00-7f 1 1 1 1 Ports = 12 USA Headquarters | 19800 North Creek Parkway | Suite 200 | Bothell | WA 98011 | USA | T: +1 800 424 4284 | F: +1 425 481 3895 European Headquarters | Via Motta 24 | 6830 Chiasso | Switzerland | T: +41 91 69769.00 | F: +41 91 69769.6 3 01-00-5e-00-01-18 1 4098 0 2 Ports = 2. Manager > sh switch table=l2multicast L2 Multicast Group and VLAN Broadcast table: -----------------------------------------------------------------------Total number of entries = 6 -----------------------------------------------------------------------Index MAC VID MCGroup CPU_MEM NumPorts PORT_LIST -----------------------------------------------------------------------0 01-00-5e-7f-ff-fa 1 4097 0 1 Ports = 4 1 01-00-5e-7f-ff-fe 1 4099 0 2 Ports = 2. All rights reserved. All company names.alliedtelesis. use the following command to match ports to multicast groups (represented by the multicast MAC address): show switch memory=l2multicast This lists the ports for each group.com © 2007 Allied Telesis. Inc.