You are on page 1of 42

Minimizing the Risks with Enterprise Multi-Site Data Center L2 Connectivity

BRKDCT-2840

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

1

Goals of this Session…
Present alternatives for interconnecting multiple Data Center locations Present tested methods in production for minimizing the risks associated with meeting these connectivity requirements.

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

Session Agenda
Data Center Interconnection – Common Scenarios and Terms Options for Layer 2 Interconnectivity Recommended Designs for Optimizing Traffic Flows EoMPLS and VPLS Stability Testing Q&A

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

4

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

2

Layer 2 / 3 Clusters Use Cases Risks Solution Types

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

Layer 2 / 3 Clusters
Intra-Cluster node communications
Flow Types Traditionally Layer2 Communications on Private and/or Public interfaces IPv4 and/or IPv6 possible depending on clustering package used Ability to prioritize interfaces

Client Access to Cluster
DNS/Active Directory resolution by clients Shared Virtual IP for service discovery Caching issues can inhibit Layer3 clustering Client application can have logic to re-establish connections

Quorum considerations to avoid split-brain
Additional cluster nodes at alternate sites to achieve a majority node set (MNS) Possible extensions such as ping-groups (Linux-HA) to have a quorum mechanism without a member node Shoot The Other Node In The Head topologies to resolve conflicts (STONITH)

Mechanisms to facilitate service restoration in another location
VMware Site Recovery Manager (SRM) is one example Microsoft Server 2008 Layer 3 Clustering is another Remapping of service to new IP/DNS entry
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

BRKDCT-2840 14688_05_2008_c1

6

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

3

Some Layer 2 Use Cases
Extending Operating System / File System clusters Extending Database clusters Virtual machine mobility Physical machine mobility Legacy devices/apps with embedded IP addressing Time to deployment and operational reasons Extend DC to solve power/heat/space limitations

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

7

Layer 2 Risks
Flooding of packets between data center’s Rapid Spanning Tree (RSTP) is not easily scalable and risk grows as diameter grows RSTP has no domain isolation – issue in single DC can propagate First hop resolution and inbound service selection can cause verbose inter-data center traffic In general Cisco recommends L3 routing for geographically diverse locations This session focuses on making limited L2 connectivity as stable as possible
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

8

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

4

Layer 2 Solution Types
Light customer owned fiber to build a separate L2 network
No STP isolation between sites

Purchase multiple wavelengths from SP
Cost rises, still nothing to offer STP isolation

Redesign data center RSTP domain using Multiple Spanning Tree (MST) regions
STP domain concept Fundamental change requiring large time investment Operational differences and MST database management

Implement a L2 solution to virtualize transport over L3
Virtual Switching System L2TPv3 for point to point (possible STP isolation issues) EoMPLS for point to point (possible STP isolation issues) Multipoint bridging using Virtual Private LAN Services (VPLS)
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

9

Session Agenda
Data Center Interconnection – Common Scenarios and Terms Options for Layer 2 Interconnectivity Recommended Designs for Optimizing Traffic Flows EoMPLS and VPLS Stability Testing Q&A

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

5

L2TPv3 Virtual Switching System EoMPLS VPLS

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

Layer 2 Prerequisites For All Options
This session assumes a fairly detailed knowledge of Spanning Tree Protocol Items we leverage in this solution:
802.1w 802.1s Port Fast BPDU Filter BPDU Guard Root Guard Loop Guard

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

12

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

6

L2TP Version 3
Encapsulates Ethernet frames inside IP packets to pass layer 3 network Layer 2 Tunneling Protocol (L2TPv3) provides routing separation from metro core devices providing connectivity – Customer Edge (CE) flapping routes wont propagate inside IP network Point to point links between locations Wide range of hardware support including ISR,72xx,73xx,ASR100x,76xx IPSec securing of tunnel straightforward Data plane rate limiting in L2 still needs protection BPDU’s still pass between locations and STP root is same

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

L2TPv3 View
3 point to point links shown in 3 site solution In layer2 environment, BPDUs must cross links and are used for loop detection In layer3 environment, point to point nature ideal for /30 subnets Data plane rate limiting in L2 still needs protection STP domains are shared between sites L2TPv3 point to point tunnels
CE PE IP PE CE

CE
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

7

L2TPv3 on 7600 Example

2.2.2.2

3.3.3.3

CE1
gi9/3/1 VLAN50

PE1
gi8/1 192.168.3.2

IP Core IP Core

PE2
gi8/1 gi9/3/1

CE2

gi8/1

gi8/2

VLAN50

PE3
gi9/3/1 4.4.4.4

Edge Facing: 7600-SIP-400

CE3
Cookie Control Data

L2

IP

SessionID

Maximum number of L2TPv3 Tunnels per SIP400 : 512 Maximum number of L2TPv3 Sessions : 8K
BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

15

L2TPv3 on 7600 PE1 Example Config
hostname 7609S-A ! vlan 50 ! pseudowire-class inter-ether-vlan encapsulation l2tpv3 protocol l2tpv3 ip local interface Loopback0 ! interface Loopback0 ip address 2.2.2.2 255.255.255.255 mls l2tpv3 reserve slot 9 ! ! router ospf 1 log-adjacency-changes network 2.2.2.2 0.0.0.0 area 0 network 192.168.3.0 0.0.0.3 area 0 ! interface GigabitEthernet9/3/1.50 encapsulation dot1Q 50 xconnect 3.3.3.3 75 pw-class inter-ether-vlan ! interface GigabitEthernet9/3/1 no ip address no negotiation auto interface GigabitEthernet8/1 ip address 192.168.3.2 255.255.255.252 mtu 1532

* Example with SIP400 with SPA 2x1GE supported (12.2.33SRC IOS software)
BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

8

Calculating L2TPv3 MTU Requirements
With and without IPSec shown Examples (all in Bytes):

Edge L2TPv3 L2TPv3 with IPSec

Transport

L2TPv3

IPSec Header

Total

1500 1500

20 20

12 12

0 16+

1532 1548+

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

17

Layer 2 Extension Without Tunnels/Tags (VSS)
6500 with Virtual Switching System cluster Supported distances at 40km All traffic flows to a VSS member node Hub-and-spoke topology from a layer 2 perspective Dedicated links to VSS members from each datacenter aggregation switch Can consume lamda’s or fiber strands quickly Data plane rate limiting in L2 still needs protection STP domains are not isolated unless we BPDU filter at all VSS aggregation switches

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

9

Virtual Switching System Design
VSS

Data Center #3
12 Lambda/24 Strand Example 4 Additional Lambda/8 Strands per new DC L2 Service Only from Provider

L2 LH Fiber/DWDM L3 LH Fiber/DWDM L2 Local Fiber L3 Local Fiber

Data Center #1

Data Center #2

VSS

VSS

VSS

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

19

Virtual Switching System L2 View
VSS

Data Center #3
All links are port channels to Central VSS

L2 LH Fiber/DWDM

L2 Local Fiber

BPDU Filtering

Data Center #1

Data Center #2

BPDU Filtering

BPDU Filtering

VSS VSS

VSS

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

10

EoMPLS (Ethernet over MPLS)
Encapsulates Ethernet frames inside MPLS packets to pass layer 3 network EoMPLS has routing separation from metro core devices providing connectivity – CE flapping routes wont propogate inside MPLS Point to point links between locations Data plane rate limiting in L2 still needs protection

EoMPLS is a pseudo-wire
CE PE MPLS PE CE

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

Virtual Private LAN Service (VPLS)
VPLS defines an architecture allows MPLS networks offer Layer 2 multipoint Ethernet Services Metro Core emulates an IEEE Ethernet bridge (virtual) Virtual Bridges linked with EoMPLS Pseudo Wires Data plane rate limiting in L2 still needs protection
VPLS is an Architecture
CE PE MPLS PE CE

CE
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

22

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

11

VPLS Components
Pseudo Wires within LSP Attachment circuits Port or VLAN mode Virtual Switch Interface (VSI) terminates PW and provides Ethernet bridge function Mesh of LSP between N-PEs

CE router

N-PE

N-PE

CE router

CE router

CE router

CE switch

MPLS Core

CE switch

Targeted LDP between PEs to exchange VC labels for Pseudo Wires

CE router CE switch

Red VFI Yellow VFI Blue VFI
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Attachment CE can be a switch or router

N-PE
23

Virtual Forwarding Instance (VFI)
IOS Representation of Virtual Switch Interface Flooding / Forwarding
MAC table instances per customer (port/vlan) for each PE VFI will participate in learning and forwarding process Associate ports to MAC, flood unknowns to all other ports

Address Learning / Aging
LDP enhanced with additional MAC List TLV (label withdrawal) MAC timers refreshed with incoming frames

Loop Prevention
Create full-mesh of Pseudo Wire VCs (EoMPLS) Unidirectional LSP carries VCs between pair of N-PE Per VPLS Uses “split horizon” concepts to prevent loops
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

24

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

12

VPLS Details
This session shows use cases for VPLS For more technical details, please attend BRKAGG2000 (Dmitry Bokotey), or BRKAGG-3001 session (Dennis Cai) This Data Center Interconnect solution uses some facets of MPLS, but not a full MP-BGP with multi VRF type implementation

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

Direct Attachment Configuration

1.1.1.1

2.2.2.2

CE1
gi3/0 VLAN100

PE1
pos4/1

PE2 Self-Managed Self-Managed MPLS Core MPLS Core
pos3/0 pos3/1 pos4/3 gi4/4

CE2

VLAN100

PE3
gi4/2 3.3.3.3 VLAN100

CE2

CEs are all part of same VPLS instance (VCID = 56)
CE router connects using VLAN 100 over sub-interface
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

26

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

13

Direct Attachment CE Router Configuration
interface GigabitEthernet 2/1.100 encapsulation dot1q 100 ip address 192.168.20.1 interface GigabitEthernet 1/3.100 encapsulation dot1q 100 ip address 192.168.20.2

CE1
Subnet 192.168.20.0/24

CE2

VLAN100

VLAN100

interface GigabitEthernet 2/0.100 encapsulation dot1q 100 ip address 192.168.20.3

CE2
VLAN100

CE routers sub-interface on same VLAN
Can also be just port based (NO VLAN)
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

27

Direct Attachment VFI Configuration
l2 vfi VPLS-A manual vpn id 56 neighbor 2.2.2.2 encapsulation mpls neighbor 3.3.3.3 encapsulation mpls
1.1.1.1

l2 vfi VPLS-A manual vpn id 56 neighbor 1.1.1.1 encapsulation mpls neighbor 3.3.3.3 encapsulation mpls
2.2.2.2

CE1
gi3/0 VLAN100

PE1
pos4/1

MPLS Core MPLS Core

PE2
pos4/3 gi4/4

CE2

pos3/0

pos3/1

VLAN100

PE3
gi4/2 3.3.3.3 VLAN100

CE2
l2 vfi VPLS-A manual vpn id 56 neighbor 2.2.2.2 encapsulation mpls neighbor 1.1.1.1 encapsulation mpls

Create the Pseudo Wires between N-PE routers
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

28

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

14

Direct Attachment CE Router (VLAN Based)
Same set of commands on each PE Configured on the CE facing interface
1.1.1.1 2.2.2.2

CE1
gi3/0 VLAN100

PE1
pos4/1

MPLS Core MPLS Core

PE2
pos4/3 gi4/4

CE2

Interface GigabitEthernet3/0VLAN100 pos3/0 pos3/1 switchport switchport mode trunk switchport trunk encapsulation dot1q PE3 switchport trunk allowed vlan 100 gi4/2 ! 3.3.3.3 CE2 Interface vlan 100 VLAN100 no ip address xconnect vfi VPLS-A This command associates the ! VLAN with the VPLS instance vlan 100 VLAN100 = VCID 56 state active

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

Calculating Core MTU Requirements
Core MTU ≥ Edge MTU + Transport Header + (MPLS Label Stack * MPLS Header Size) Edge MTU is the MTU configured in the CE-facing PE interface Examples (all in Bytes):

Edge EoMPLS Port Mode EoMPLS VLAN Mode

Transport

MPLS Stack

MPLS Header

Total

1500 1500

14 18

2 2

4 4

1522 1526

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

15

End to End VPLS and EoMPLS Design
Layer 3 Core Intranet
WCore1 WCore2 ECore1 ECore2

WMC1 DC Core Po1 WAgg1 Po4 Po3 Agg Po3 Po4 WAgg2 Po1 WMC2 Ten3/0/0 Ten3/0/0 Ten4/0/0 Ten3/0/0 Ten3/0/0

EMC1 DC Core Po1

Po2 Po2

Ten4/0/0

Po2 Po2 Po4 EAgg1 Po1

Po4 Po3 Po3

EAgg2

VPLS / EoMPLS Domain
Ten4/0/0 Ten4/0/0

Agg

EMC2

Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Loss of Link/Node Server Farm
BRKDCT-2840 14688_05_2008_c1

Access

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

Access to Aggregation Connections
Rapid PVST is existing protocol, and no desire to force a change Aggregation switches are root for all intra-DC VLANs The peer aggregation switch is secondary root DO NOT customize the bridge priorities to lower values HSRP tested for first hop redundancy from server (more later)
Server Farm Agg

Access

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

32

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

16

Layer 3 Aggregation and Core Connections
IF dual supervisor modules, need non-stop forwarding (NSF) under routing process

Layer 3 connections from DC Core to Enterprise Core Aggregation switch L3 connected to DC Core Hanging L3 links in diagram, are to Metro Core switches which are Ethernet over MPLS links Hanging L3 links are for peering the DC Cores in each location in a point-topoint scenario

Layer 3 Enterprise Core

DC Core

Agg

Bidirectional forwarding detection (bfd) interval 100 min_rx 100 multiplier 3
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

33

EoMPLS / VPLS Infrastructure
Loopbacks chosen as peering points for EoMPLS and VPLS xconnects Horizontal links represent 10GE on DWDM service between data centers (alternate paths) Vertical links represent intra-DC 10GE connections Mpls LDP enabled globally (not a full P / PE MPLS implmentation) Links to/from aggregation switches for Layer 2, are storm-control limited for broadcasts and multicasts to 1% (protect data plane) MTU increased to 1522 bytes on the L3 MPLS links for the MPLS tagging
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

VPLS / EoMPLS Domain

Metro Core

Metro Core

BRKDCT-2840 14688_05_2008_c1

34

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

17

Metro Switch Interconnectivity

Ten3/0/0

Ten3/0/0

Ten4/0/0 Ten4/0/0

IGP Routing Process connecting MPLS PE’s

Ten4/0/0

Ten4/0/0

Ten3/0/0 Metro Core

Ten3/0/0 Metro Core

L3 Links (10GE)

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

35

Metro Switch Interconnectivity (EIGRP)
Layer3 interface between the Metro Switches configured for MPLS
mpls label protocol ldp ! interface TenGigabitEthernet3/0/0 description MPLS Interface to peer N-PE in DC #2 mtu 1522 ip address 192.168.1.1 255.255.255.252 ip hello-interval eigrp 5 1 ip hold-time eigrp 5 3 ip authentication mode eigrp 5 md5 ip authentication key-chain eigrp 5 password logging event link-status load-interval 30 udld port disable mls qos trust dscp mpls ip !
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

36

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

18

Metro Switch Interconnectivity (EIGRP) (Cont.)
! interface TenGigabitEthernet4/0/0 description MPLS Interface to local peer N-PE mtu 1522 ip address 192.168.1.9 255.255.255.252 ip hello-interval eigrp 5 1 ip hold-time eigrp 5 3 ip authentication mode eigrp 5 md5 ip authentication key-chain eigrp 5 password logging event link-status load-interval 30 udld port disable mls qos trust dscp mpls ip

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

37

Metro Switch interconnectivity (OSPF)
! interface TenGigabitEthernet3/0/0 description MPLS Interface to peer N-PE in DC #2 mtu 1522 ip address 192.168.1.1 255.255.255.252 ip ospf network point-to-point logging event link-status load-interval 30 udld port disable mls qos trust dscp mpls ip router ospf 1 log-adjacency-changes nsf cisco timers throttle spf 50 100 5000 timers lsa arrival 0 timers pacing flood 15 network 192.168.0.0 0.0.255.255 area 0
BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

38

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

19

Metro Switch interconnectivity (OSPF) (Cont.)
! interface TenGigabitEthernet4/0/0 description MPLS Interface to local peer N-PE mtu 1522 ip address 192.168.1.9 255.255.255.252 ip ospf network point-to-point logging event link-status load-interval 30 udld port disable mls qos trust dscp mpls ip router ospf 1 log-adjacency-changes nsf cisco timers throttle spf 50 100 5000 timers lsa arrival 0 timers pacing flood 15 network 192.168.0.0 0.0.255.255 area 0
BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

39

Metro Switch Example IP Addressing
Loopbacks and WAN links use 192.168.0.0 addressing WestMetroCore1
Loopback0 IP Address = 192.168.255.250/32 Interface Te3/0/0 IP Address = 192.168.1.1/30 Interface Te4/0/0 IP Address = 192.168.1.9/30

WestMetroCore2
Loopback0 IP Address = 192.168.255.251/32 Interface Te3/0/0 IP Address = 192.168.1.5/30 Interface Te4/0/0 IP Address = 192.168.1.10/30
BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

40

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

20

Metro Switch Example IP Addressing (Cont.)
EastMetroCore1
Loopback0 IP Address = 192.168.255.252/32 Interface Te3/0/0 IP Address = 192.168.1.2/30 Interface Te4/0/0 IP Address = 192.168.1.13/30

EastMetroCore2
Loopback0 IP Address = 192.168.255.253/32 Interface Te3/0/0 IP Address = 192.168.1.6/30 Interface Te4/0/0 IP Address = 192.168.1.14/30

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Metro Switch Routing
No Dynamic routing between the Metro Core switches and other Data Center switches router eigrp 5 (the routing instance for the MPLS domain, LDP passes on this)
passive-interface default no passive-interface TenGigabitEthernet3/0/0 no passive-interface TenGigabitEthernet4/0/0 network 192.168.0.0 0.0.255.255 no auto-summary Nsf

Enable NSF for LDP, mpls ldp graceful-restart global configuration

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

21

Metro Switch Interconnectivity
EastMetroCore1#sh ip route Gateway of last resort is 10.98.128.1 to network 0.0.0.0 C D C D D D C C D S* 10.0.0.0/24 is subnetted, 1 subnets 10.98.128.0 is directly connected, GigabitEthernet5/2 192.168.255.0/32 is subnetted, 4 subnets 192.168.255.253 [90/128512] via 192.168.1.14, 3w0d, TenGigabitEthernet4/0/0 192.168.255.252 is directly connected, Loopback0 192.168.255.251 [90/128768] via 192.168.1.14, 3w0d, TenGigabitEthernet4/0/0 [90/128768] via 192.168.1.1, 3w0d, TenGigabitEthernet3/0/0 192.168.255.250 [90/128512] via 192.168.1.1, 3w0d, TenGigabitEthernet3/0/0 192.168.1.0/30 is subnetted, 4 subnets 192.168.1.8 [90/768] via 192.168.1.1, 3w0d, TenGigabitEthernet3/0/0 192.168.1.12 is directly connected, TenGigabitEthernet4/0/0 192.168.1.0 is directly connected, TenGigabitEthernet3/0/0 192.168.1.4 [90/768] via 192.168.1.14, 3w0d, TenGigabitEthernet4/0/0 0.0.0.0/0 [1/0] via 10.98.128.1

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

EoMPLS for Layer3
Layer 3 Core Intranet

DC Core

METRO CORE
PW – Pseudo Wires

DC Core

Agg

Agg

EoMPLS

Metro Core

Metro Core

Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm
BRKDCT-2840 14688_05_2008_c1

Access

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

22

EoMPLS for Layer3 Configuration
EastMetroCore1 ! interface Loopback0 description Loopback interface for PW peering ip address 192.168.255.252 255.255.255.255 ! interface GigabitEthernet1/1 xconnect 192.168.255.250 250252 encapsulation mpls ! interface GigabitEthernet2/1
description EASTCORE:8/1 - WESTCORE:1/2 description EASTCORE:9/1 - WESTCORE:1/1

xconnect 192.168.255.250 252250 encapsulation mpls

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

EoMPLS for Layer3 Configuration (Cont.)
WestMetroCore1 ! interface Loopback0 description Loopback interface for PW peering ip address 192.168.255.250 255.255.255.255 ! interface GigabitEthernet1/1 xconnect 192.168.255.252 250252 encapsulation mpls ! interface GigabitEthernet2/1
description WESTCORE:1/2 - EASTCORE:8/1 description WESTCORE:1/1 - EASTCORE:9/1

xconnect 192.168.255.252 252250 encapsulation mpls

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

46

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

23

VPLS for Layer2
Layer 3 Core Intranet

DC Core

METRO CORE

DC Core

VFI
Agg Agg

PW – Pseudo Wires
Metro Core Metro Core

Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm
BRKDCT-2840 14688_05_2008_c1

Access

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

VPLS for Layer2
l2 vfi vlan3700 manual vpn id 3700 Layer 3 Core neighbor 192.168.255.251 encapsulation mplsIntranet neighbor 192.168.255.252 encapsulation mpls neighbor 192.168.255.253 encapsulation mpls
DC Core

l2 vfi vlan3700 manual vpn id 3700 neighbor 192.168.255.250 encapsulation mpls neighbor 192.168.255.251 encapsulation mpls neighbor 192.168.255.253 encapsulation mpls
DC Core

METRO CORE

Agg

Agg

PW – Pseudo Wires
Metro Core Metro Core

l2 vfi vlan3700 manual Accessid 3700 vpn neighbor 192.168.255.250 encapsulation mpls neighbor 192.168.255.252 encapsulation mpls neighbor 192.168.255.253 encapsulation mpls
Server Farm
BRKDCT-2840 14688_05_2008_c1

l2 vfi vlan3700 manual Access vpn id 3700 neighbor 192.168.255.250 encapsulation mpls L2 Links (GE or 10GE) neighbor 192.168.255.251 encapsulation mpls L3 Links (GE or 10GE) neighbor 192.168.255.252 encapsulation mpls
Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

48

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

24

VPLS for Layer2
interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700
Layer 3 Core Intranet

interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700
DC Core

DC Core

METRO CORE

VLAN 3700
Agg Agg

PW – Pseudo Wires
Metro Core Metro Core

Access

interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700

interface Vlan3700 no ip address load-interval 30 L2 Links (GE or 10GE) xconnect vfi vlan3700
L3 Links (GE or 10GE)

Access

Server Farm
BRKDCT-2840 14688_05_2008_c1

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

VPLS for Layer2 Configuration
WestMetroCore1 ! interface Loopback0 description Loopback interface for PW peering ip address 192.168.255.250 255.255.255.255 ! l2 vfi vlan3700 manual vpn id 3700 neighbor 192.168.255.252 encapsulation mpls neighbor 192.168.255.251 encapsulation mpls neighbor 192.168.255.253 encapsulation mpls ! interface Vlan3700 no ip address load-interval 30 xconnect vfi vlan3700 ! interface Port-channel1 description WESTAGG1 switchport trunk allowed vlan 3700-3760

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

25

Spanning Tree
Spanning-Tree BPDU(s) will NOT traverse between the Data Centers – It isn’t needed (and blocked) with VPLS We still need to control data plane layer 2 events (i.e. limit the traffic) Since enterprises want dual N-PE devices, and VPLS blocks BPDU’s, we require method to block within a local DC

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

Spanning Tree – Without N-PE MST Region or EEM
Broadcast, Multicast, Unknown Unicast

Layer 3 Core Intranet

DC Core

DC Core

VPLS / EoMPLS Domain
Agg

RSTP
X X
Metro Core Metro Core

RSTP
X X

Agg

Access

Access

Without layer 2 link between Metro Switches there is a loop. Each side has a “U” shape with Metro and Agg switches. Broadcast storms.
Server Farm
BRKDCT-2840 14688_05_2008_c1

L2 Links (GE or 10GE) L3 Links (GE or 10GE)

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

52

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

26

Spanning Tree - MST
Root Bridge in West DC for all VLANs that Go Between Data Centers
Layer 3 Core Intranet

Root Bridge in East DC for all VLANs that Go Between Data Centers

DC Core Single L2 MST Bridge Single L2 MST Bridge

DC Core

MST
Agg

VPLS / EoMPLS Domain

MST
Agg

RSTP
Metro Core Metro Core

RSTP

Access L2 Links (GE or 10GE) L3 Links (GE or 10GE) Server Farm
BRKDCT-2840 14688_05_2008_c1

Access

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Spanning-Tree
MST (802.1s) chosen to present Metro Cores as single bridge Red Layer 2 link is access port channel with a VLAN that represents the MST0 instance to make the MST group MST bridge priority set to 0 (Metro Core will be root of Inter-DC VLANs) Spanning tree root-guard enabled on Metro Cores toward aggregation switches (protects in case the red MST link fails) Only inter-DC VLANs allowed on trunks to/from aggregation switches Set spanning-tree vlan cost to set the priorities on the agg switches links to metro core – will allow us to put some VLANs on upper Metro Core, some on Lower by default

Single L2 MST Bridge

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

54

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

27

Spanning Tree - MST
interface Port-channel4 description Port Channel to WestMetroCore1 Layer Core spanning-tree vlan 3702,3706,3710,3714,37183cost 8
Intranet

DC Core Single L2 MST Bridge Single L2 MST Bridge

DC Core

X X
Agg

MST

VPLS / EoMPLS Domain

MST

X X
Agg

X

RSTP
X X

X
Metro Core Metro Core

X

RSTP
X X

X

Access

Access

interface Port-channel4 description Port Channel to WestMetroCore2 spanning-tree vlan 3700,3704,3712,3716 cost 8
Server Farm
BRKDCT-2840 14688_05_2008_c1

Server Farm

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

55

MST Configuration
Configuration on Primary N-PE (root of Inter-DC VLANs):
spanning-tree mode mst spanning-tree extend system-id ! spanning-tree mst configuration name WESTDC revision 50 ! spanning-tree mst 0 priority 0 ! interface Port-channel5 description port-channel to westagg1 switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan 3700 switchport mode trunk storm-control broadcast level 1.00 storm-control multicast level 1.00 spanning-tree guard root ! interface Port-channel6 description port-channel to peer N-PE switchport switchport mode access BRKDCT-2840 ! 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

56

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

28

Alternative VPLS Solution
If you have different requirements such as:
QinQ Not Moving STP Root for Inter DC vLANs

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Storm Control
Traffic storms when packets flood the LAN Traffic storm control feature prevents LAN ports from being disrupted by broadcast or multicast flooding Rate limiting for unknown unicast (UU) must be handled at Data Center aggregation; unknown unicast flood rate-limiting (UUFRL):
mls rate-limit layer2 unknown rate-in-pps [burst-size]

Storm Control is configured as a percentage of the link that storm traffic is allowed to use.
storm-control broadcast level 1.00 (% of b/w may vary – need to baseline) storm-control multicast level 1.00 (% of b/w may vary – need to baseline)

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

29

3 or More Data Center Locations
EoMPLS will allow multiple point to point links between any 2 sites Can build a full mesh of links to interconnect layer 3 devices VPLS scales by adding peer xconnects under the VFI in the IOS configuration Split horizon with MST local to data center will make for simple growth Limits dependant on amounts of L2 traffic especially multicast, as these are replicated on each PW

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

3 Site Drawing with EoMPLS PW’s for L3
Server Farm

Server Farm
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Server Farm 60

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

30

3 Site Drawing with VPLS PW’s for L2
Server Farm

Server Farm
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Server Farm 61

Summary of Tagging Section
EoMPLS well suited for Router-Router links VPLS well suited for Switch-Switch links Straightforward to scale to multiple Data Center locations MST and EEM semiphores both work well
One tradeoff is QinQ support against number of VLANs to pass Another is the root of the spanning tree for inter-DC VLANs

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

31

Session Agenda
Data Center Interconnection – Common Scenarios and Terms Options for Layer 2 Interconnectivity Recommended Designs for Optimizing Traffic Flows EoMPLS and VPLS Stability Testing Q&A

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

Flow Optimization and Symmetry Site Selection and Inbound Flows First Hop Outbound Intra/Inter Site Failover VMware DRS Case

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

64

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

32

Optimizing Traffic Patterns and HA Design
Many tradeoffs in understanding flows in multi-DC design Slides that follow are a specific recommendation that meets the following requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.) Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS, etc.) Site failure will allow failover, with IP mobility to resolve caching issues Single points of failure in gear wont cause site failover Indicate a location preference for a service to the Layer 3 network If broadcast storm in DC, limit impacts to other DC’s Ability to connect to services in both DC locations (active/active per application) DNS to round-robin clients to DC Allow backup server farms with same service VIP (for backup connections on site fail)

This is a solution in production at some customers

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

65

Sample Cluster – Service Normally in Left DC
Default Gateway Shared Between Sites
10.1.1.0/25 & 10.1.1.128/25 advertised into L3 -EEM or RHI can be used to get very granular
Layer3 Core

10.1.1.0/24 advertised into L3 Backup should main site go down

Active/Standby Pairs: FW IPS NLB SSL WAN Accel

Active/Standby Pairs: FW IPS NLB SSL WAN Accel

VLAN A

VLAN A

10.1.1.1 HSRP Group 1 Priority 140 and 130

10.1.1.1 HSRP Group 1 Priority 120 and 110

Cluster Node A Cluster VLAN C (L2 Only) Cluster VLAN D (L2 Only)

Cluster Node B

-Cluster VIP = 10.1.1.100 Preempt -Default GW = 10.1.1.1
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

-Cluster VIP = 10.1.1.100 -Default GW = 10.1.1.1
66

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

33

Optimizing Traffic Patterns and HA Design
Many tradeoffs in understanding flows in multi-DC design Slides that follow are a specific recommendation that meets the following requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.) Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS, etc.) Site failure will allow failover, with IP mobility to resolve caching issues Single points of failure in gear wont cause site failover Indicate a location preference for a service to the Layer 3 network If broadcast storm in DC, limit impacts to other DC’s Ability to connect to services in both DC locations (active/active per application) DNS to round-robin clients to DC Allow backup server farms with same service VIP (for backup connections on site fail)

This is a solution in production at some customers

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

67

Sample Cluster – Broadcast Storm in Left DC
Broadcast, Multicast, Unknown Unicast
10.1.1.0/25 & 10.1.1.128/25 advertised into L3 -EEM or RHI can be used to get very granular
Layer3 Core

10.1.1.0/24 advertised into L3 Backup should main site go down

VLAN A

VLAN A

10.1.1.1 HSRP Group 1 Priority 140 and 130

10.1.1.1 HSRP Group 1 Priority 120 and 110

Cluster Node A Cluster VLAN C (L2 Only) Cluster VLAN D (L2 Only)

Cluster Node B

-Cluster VIP = 10.1.1.100 Preempt -Default GW = 10.1.1.1
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

-Cluster VIP = 10.1.1.100 -Default GW = 10.1.1.1
68

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

34

Optimizing Traffic Patterns and HA Design
Many tradeoffs in understanding flows in multi-DC design Slides that follow are a specific recommendation that meets the following requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.) Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS, etc.) Site failure will allow failover, with IP mobility to resolve caching issues Single points of failure in gear wont cause site failover Indicate a location preference for a service to the Layer 3 network If broadcast storm in DC, limit impacts to other DC’s Ability to connect to services in both DC locations (active/active per application) DNS to round-robin clients to DC Allow backup server farms with same service VIP (for backup connections on site fail)

This is a solution in production at some customers

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

Active/Active Per Application (VIP at Either)
10.1.1.0/25 & 10.1.1.128/25 advertised into L3 -EEM or RHI can be used to get very granular
Layer3 Core

10.1.1.0/24 advertised into L3 Backup should main site go down

10.1.1.0/24 advertised into L3 Backup should main site go down

10.1.2.0/25 & 10.1.2.128/25 advertised into L3 -EEM or RHI can be used to get very granular
DNS: www-hr.acme.com -> 10.1.1.100 www-news.acme.com -> 10.1.2.100

10.1.2.1 HSRP Group 2 Priority 140 and 130
VLAN A VLAN A

10.1.1.1 HSRP Group 1 Priority 140 and 130
Cluster Node A

10.1.2.1 HSRP Group 2 Priority 120 and 110

10.1.1.1 HSRP Group 1 Priority 120 and 110
Cluster Node B

Cluster VLAN C (L2 Only)

-Cluster VIP = 10.1.2.100 -Default GW = 10.1.2.1

Cluster VLAN D (L2 Only)

-Cluster VIP = 10.1.1.100 Preempt -Default GW = 10.1.1.1
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

-Cluster VIP = 10.1.1.100 -Cluster VIP = 10.1.2.100 Preempt -Default GW = 10.1.1.1 -Default GW = 10.1.2.1
70

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

35

Optimizing Traffic Patterns and HA Design
Many tradeoffs in understanding flows in multi-DC design Slides that follow are a specific recommendation that meets the following requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.) Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS, etc.) Site failure will allow failover, with IP mobility to resolve caching issues Single points of failure in gear wont cause site failover Indicate a location preference for a service to the Layer 3 network If broadcast storm in DC, limit impacts to other DC’s Ability to connect to services in both DC locations (active/active per application) DNS to round-robin clients to DC Allow backup server farms with same service VIP (for backup connections on site fail)

This is a solution in production at some customers

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

71

Active/Active Per Application (VIP at Both)
10.1.1.0/25 & 10.1.1.128/25 advertised into L3 -EEM or RHI can be used to get very granular
Layer3 Core

10.1.1.0/24 advertised into L3 Backup should main site go down

10.1.1.0/24 advertised into L3 Backup should main site go down

10.1.2.0/25 & 10.1.2.128/25 advertised into L3 -EEM or RHI can be used to get very granular
DNS: www-hr.acme.com -> 10.1.1.100 10.1.2.100

10.1.2.1 HSRP Group 2 Priority 140 and 130
VLAN A VLAN A

10.1.1.1 HSRP Group 1 Priority 140 and 130
Cluster Node A

10.1.2.1 HSRP Group 2 Priority 120 and 110

10.1.1.1 HSRP Group 1 Priority 120 and 110
Cluster Node B

Cluster VLAN C (L2 Only)

-Cluster VIP = 10.1.2.100 -Default GW = 10.1.2.1

Cluster VLAN D (L2 Only)

-Cluster VIP = 10.1.1.100 Preempt -Default GW = 10.1.1.1
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

-Cluster VIP = 10.1.1.100 -Cluster VIP = 10.1.2.100 Preempt -Default GW = 10.1.1.1 -Default GW = 10.1.2.1
72

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

36

Session Agenda
Data Center Interconnection – Common Scenarios and Terms Options for Layer 2 Interconnectivity Recommended Designs for Optimizing Traffic Flows EoMPLS and VPLS Stability Testing Q&A

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

73

CPOC Tested Failover Numbers

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

74

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

37

EoMPLS and VPLS Stability Testing
Testing of link outage scenarios
Pulling fiber connections Administratively shutting down interfaces Pulling active cards and supervisors

Testing of failure and fail-back timing Tests grouped by location in the network
Metro Core failures Aggregation failures Layer 3 Core failures

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

75

Metro Core Failover/Failback Tests

Link Down
Top Rail Pull Top Rail Admin Vertical Rail Pull 2x10GE Card Fail 2x10GE Card AS Node Power Off MST Link Pull Primary Sup Pull 105mSec 133mSec 0 1.2Sec 718mSec 379mSec 0 516mSec

Link Up
1mSec 1mSec 0 5.4Sec 5.7Sec 6.4Sec 0 0

x x xx

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

38

Embedded Event Manager
Scripting based on events Script initiator is a tracking of node reachability Bring up interfaces in a known order Allow traffic flows based on a time delay

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

77

EEM Policy to Handle VPLS Down
In case VPLS redundancy is not possible an EEM policy can be used to prevent black-hole when VPLS path goes down Since, the LAN modules come-up before the WAN Modules used EEM and EOT to control
event manager applet TRACK_ES20_DOWN event track 40 state down action 1.0 cli command "config t" action 2.0 cli command "interface TenGigabitEthernet4/4" action 3.0 cli command "shutdown" action 4.0 syslog msg "EEM has shutdown the SVI's" event manager applet TRACK_ES20_UP event track 40 state up action 1.0 cli command "config t" action 2.0 cli command "interface TenGigabitEthernet4/4" action 3.0 cli command "no shutdown" action 4.0 syslog msg "EEM has enabled the Ten4/4" event manager applet UP_TEN4/4 event track 25 state up action 1.0 cli command "config t" action 2.0 cli command "interface TenGigabitEthernet4/4" action 3.0 cli command "no shutdown" action 4.0 syslog msg "EEM has unshut Ten4/4" event manager applet test event syslog pattern "Module 4: Passed Online Diagnostics" action 1.0 cli command "config t" action 2.0 cli command "interface TenGigabitEthernet4/4" action 3.0 cli command "shutdown" action 4.0 syslog msg "EEM has shutdown Ten4/4"

track 1 interface GigabitEthernet3/0/0 line-protocol ! track 2 interface GigabitEthernet3/0/1 line-protocol ! track 20 ip route 10.1.133.226 255.255.255.255 reachability ! track 21 ip route 10.1.133.222 255.255.255.255 reachability ! track 25 list boolean and object 20 object 21 delay up 90 ! track 40 list boolean or object 1 object 2 delay up 90 !

10.1.133.226 and 222 are remote N-PEs Gig3/0/0 and 3/0/1 are VPLS uplinks TenGigE4/4 is the link to local Agg switch
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

78

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

39

Aggregation Failover/Failback Tests

Link Down
Left Agg Pull Left Agg Admin Access Link Pull Agg Port-Ch Fail Agg Port-Ch AS 344mSec 668mSec 63mSec (L2), 123mSec (L3) 0 0

Link Up
5.7Sec 5.7Sec 17mSec (L2), 101mSec (L3) 0 0

x x x

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

79

Core Failover/Failback Tests

x x x

Link Down
Core-Core Shut Core-MC Shut Core-Core with Core-MC Shut Core Sup Pull Core Reload 0 0 0 734mSec 0

Link Up
0 0 0 0 0

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

80

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

40

Q and A

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

81

Recommendations
Recommended Reading:
MPLS and VPN Architectures, Volume II by Jim Guichard Network Virtualization by Victor Moreno

Check the Recommended Reading flyer for suggested books Related technology breakouts:
BRKAGG-2000 Implementation and utilization of Layer 2 VPN technologies TECAGG-2003 Layer 2 Virtual Private Networks – Converged IP/MPLS Network

NSITE is compiling test results for both the MST N-PE and EEM Solution

Available Onsite at the Cisco Company Store
BRKDCT-2840 14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

82

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

41

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

83

BRKDCT-2840 14688_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

84

© 2006, Cisco Systems, Inc. All rights reserved. 14688_05_2008_c1.scr

42