You are on page 1of 42

Minimizing the Risks with

Enterprise Multi-Site Data
Center L2 Connectivity

BRKDCT-2840

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1
14688_05_2008_c1.scr
Goals of this Session…

ƒ Present alternatives for interconnecting multiple Data
Center locations
ƒ Present tested methods in production for minimizing the
risks associated with meeting these connectivity
requirements.

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Session Agenda

ƒ Data Center Interconnection – Common Scenarios
and Terms
ƒ Options for Layer 2 Interconnectivity
ƒ Recommended Designs for Optimizing Traffic Flows
ƒ EoMPLS and VPLS Stability Testing
ƒQ&A

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2
14688_05_2008_c1.scr
Layer 2 / 3 Clusters
Use Cases Risks
Solution Types

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

Layer 2 / 3 Clusters
ƒ Intra-Cluster node communications
Flow Types
Traditionally Layer2
Communications on Private and/or Public interfaces
IPv4 and/or IPv6 possible depending on clustering package used
Ability to prioritize interfaces
ƒ Client Access to Cluster
DNS/Active Directory resolution by clients
Shared Virtual IP for service discovery
Caching issues can inhibit Layer3 clustering
Client application can have logic to re-establish connections
ƒ Quorum considerations to avoid split-brain
Additional cluster nodes at alternate sites to achieve a majority node set (MNS)
Possible extensions such as ping-groups (Linux-HA) to have a quorum mechanism without
a member node
Shoot The Other Node In The Head topologies to resolve conflicts (STONITH)
ƒ Mechanisms to facilitate service restoration in another location
VMware Site Recovery Manager (SRM) is one example
Microsoft Server 2008 Layer 3 Clustering is another
BRKDCT-2840
Remapping of service to new IP/DNS entry
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3
14688_05_2008_c1.scr
Some Layer 2 Use Cases

ƒ Extending Operating System / File System clusters
ƒ Extending Database clusters
ƒ Virtual machine mobility
ƒ Physical machine mobility
ƒ Legacy devices/apps with embedded IP addressing
ƒ Time to deployment and operational reasons
ƒ Extend DC to solve power/heat/space limitations

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Layer 2 Risks

ƒ Flooding of packets between data center’s
ƒ Rapid Spanning Tree (RSTP) is not easily scalable and
risk grows as diameter grows
ƒ RSTP has no domain isolation – issue in single DC can
propagate
ƒ First hop resolution and inbound service selection can
cause verbose inter-data center traffic
ƒ In general Cisco recommends L3 routing for
geographically diverse locations
ƒ This session focuses on making limited L2 connectivity
as stable as possible
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4
14688_05_2008_c1.scr
Layer 2 Solution Types
ƒ Light customer owned fiber to build a separate L2 network
No STP isolation between sites
ƒ Purchase multiple wavelengths from SP
Cost rises, still nothing to offer STP isolation
ƒ Redesign data center RSTP domain using Multiple Spanning Tree (MST)
regions
STP domain concept
Fundamental change requiring large time investment
Operational differences and MST database management
ƒ Implement a L2 solution to virtualize transport over L3
Virtual Switching System
L2TPv3 for point to point (possible STP isolation issues)
EoMPLS for point to point (possible STP isolation issues)
Multipoint bridging using Virtual Private LAN Services (VPLS)
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

Session Agenda

ƒ Data Center Interconnection – Common Scenarios
and Terms
ƒ Options for Layer 2 Interconnectivity
ƒ Recommended Designs for Optimizing Traffic Flows
ƒ EoMPLS and VPLS Stability Testing
ƒQ&A

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5
14688_05_2008_c1.scr
L2TPv3 Virtual
Switching System
EoMPLS VPLS

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

Layer 2 Prerequisites For All Options

ƒ This session assumes a fairly detailed knowledge of
Spanning Tree Protocol
ƒ Items we leverage in this solution:
802.1w
802.1s
Port Fast
BPDU Filter
BPDU Guard
Root Guard
Loop Guard

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6
14688_05_2008_c1.scr
L2TP Version 3
ƒ Encapsulates Ethernet frames inside IP packets to pass layer 3
network
ƒ Layer 2 Tunneling Protocol (L2TPv3) provides routing separation
from metro core devices providing connectivity – Customer Edge
(CE) flapping routes wont propagate inside IP network
ƒ Point to point links between locations
ƒ Wide range of hardware support including
ISR,72xx,73xx,ASR100x,76xx
ƒ IPSec securing of tunnel straightforward
ƒ Data plane rate limiting in L2 still needs protection
ƒ BPDU’s still pass between locations and STP root is same

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

L2TPv3 View
ƒ 3 point to point links shown in 3 site solution
ƒ In layer2 environment, BPDUs must cross links and are used for loop
detection
ƒ In layer3 environment, point to point nature ideal for /30 subnets
ƒ Data plane rate limiting in L2 still needs protection
ƒ STP domains are shared between sites
L2TPv3 point to point tunnels

PE PE
CE CE

IP

CE
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7
14688_05_2008_c1.scr
L2TPv3 on 7600 Example

2.2.2.2 3.3.3.3
PE1 IP PE2
CE1 IPCore
Core CE2
gi8/1 gi8/1
gi9/3/1 192.168.3.2 gi9/3/1
VLAN50 gi8/1 gi8/2 VLAN50

PE3 Edge Facing: 7600-SIP-400
gi9/3/1
4.4.4.4 CE3

L2 IP SessionID Cookie Control Data
Maximum number of L2TPv3 Tunnels per SIP400 : 512
Maximum number of L2TPv3 Sessions : 8K

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

L2TPv3 on 7600 PE1 Example Config
hostname 7609S-A interface GigabitEthernet8/1
! ip address 192.168.3.2 255.255.255.252
vlan 50 mtu 1532
! !
pseudowire-class inter-ether-vlan interface GigabitEthernet9/3/1
encapsulation l2tpv3 no ip address
protocol l2tpv3 no negotiation auto
ip local interface Loopback0 !
! interface GigabitEthernet9/3/1.50
interface Loopback0 encapsulation dot1Q 50
ip address 2.2.2.2 255.255.255.255 xconnect 3.3.3.3 75 pw-class inter-ether-vlan
mls l2tpv3 reserve slot 9 !
! router ospf 1
log-adjacency-changes
network 2.2.2.2 0.0.0.0 area 0
network 192.168.3.0 0.0.0.3 area 0

* Example with SIP400 with SPA 2x1GE supported (12.2.33SRC IOS software)

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8
14688_05_2008_c1.scr
Calculating L2TPv3 MTU Requirements

ƒ With and without IPSec shown
ƒ Examples (all in Bytes):

IPSec
Edge Transport L2TPv3 Total
Header

L2TPv3 1500 20 12 0 1532

L2TPv3 with IPSec 1500 20 12 16+ 1548+

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Layer 2 Extension Without Tunnels/Tags
(VSS)
ƒ 6500 with Virtual Switching System cluster
ƒ Supported distances at 40km
ƒ All traffic flows to a VSS member node
ƒ Hub-and-spoke topology from a layer 2 perspective
ƒ Dedicated links to VSS members from each datacenter
aggregation switch
ƒ Can consume lamda’s or fiber strands quickly
ƒ Data plane rate limiting in L2 still needs protection
ƒ STP domains are not isolated unless we BPDU filter at all VSS
aggregation switches

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9
14688_05_2008_c1.scr
Virtual Switching System Design
VSS
Data Center #3 L2 LH Fiber/DWDM
L3 LH Fiber/DWDM
L2 Local Fiber
12 Lambda/24 Strand Example
4 Additional Lambda/8 Strands per new DC L3 Local Fiber
L2 Service Only from Provider

Data Center #1 Data Center #2

VSS

VSS VSS

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Virtual Switching System L2 View
VSS
Data Center #3 L2 LH Fiber/DWDM

L2 Local Fiber
All links are port channels to Central VSS

BPDU Filtering

Data Center #1 Data Center #2

BPDU Filtering BPDU Filtering

VSS VSS

VSS

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10
14688_05_2008_c1.scr
EoMPLS (Ethernet over MPLS)
ƒ Encapsulates Ethernet frames inside MPLS packets to pass layer
3 network
ƒ EoMPLS has routing separation from metro core devices providing
connectivity – CE flapping routes wont propogate inside MPLS
ƒ Point to point links between locations
ƒ Data plane rate limiting in L2 still needs protection

EoMPLS is a pseudo-wire

PE PE
CE CE

MPLS

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

Virtual Private LAN Service (VPLS)
ƒ VPLS defines an architecture allows MPLS networks offer Layer 2
multipoint Ethernet Services
ƒ Metro Core emulates an IEEE Ethernet bridge (virtual)
ƒ Virtual Bridges linked with EoMPLS Pseudo Wires
ƒ Data plane rate limiting in L2 still needs protection
VPLS is an Architecture

PE PE
CE CE

MPLS

CE
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11
14688_05_2008_c1.scr
VPLS Components
Pseudo Wires within LSP
Virtual Switch Interface (VSI)
terminates PW and provides
Attachment circuits Ethernet bridge function
Port or VLAN mode
Mesh of LSP between N-PEs
N-PE N-PE
CE router CE router

CE router CE router

CE switch CE switch
MPLS
Core

Targeted LDP between PEs to
exchange VC labels for Pseudo CE router
Wires Attachment CE
can be a switch or
CE switch router
Red VFI

Yellow VFI

Blue VFI N-PE
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

Virtual Forwarding Instance (VFI)
ƒ IOS Representation of Virtual Switch Interface
ƒ Flooding / Forwarding
MAC table instances per customer (port/vlan) for each PE
VFI will participate in learning and forwarding process
Associate ports to MAC, flood unknowns to all other ports
ƒ Address Learning / Aging
LDP enhanced with additional MAC List TLV (label withdrawal)
MAC timers refreshed with incoming frames
ƒ Loop Prevention
Create full-mesh of Pseudo Wire VCs (EoMPLS)
Unidirectional LSP carries VCs between pair of N-PE Per
VPLS Uses “split horizon” concepts to prevent loops
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12
14688_05_2008_c1.scr
VPLS Details

ƒ This session shows use cases for VPLS
ƒ For more technical details, please attend BRKAGG-
2000 (Dmitry Bokotey), or BRKAGG-3001 session
(Dennis Cai)
ƒ This Data Center Interconnect solution uses some
facets of MPLS, but not a full MP-BGP with multi VRF
type implementation

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Direct Attachment Configuration

1.1.1.1 2.2.2.2
PE1 PE2
CE1 pos4/1
CE2
Self-Managed pos4/3
Self-Managed
gi3/0 MPLS
MPLSCore
Core gi4/4
VLAN100 pos3/0 pos3/1 VLAN100

PE3
gi4/2
3.3.3.3 CE2
VLAN100

ƒ CEs are all part of same VPLS instance (VCID = 56)
CE router connects using VLAN 100 over sub-interface

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13
14688_05_2008_c1.scr
Direct Attachment CE Router
Configuration
interface GigabitEthernet 2/1.100 interface GigabitEthernet 1/3.100
encapsulation dot1q 100 encapsulation dot1q 100
ip address 192.168.20.1 ip address 192.168.20.2

CE1 CE2

Subnet
VLAN100 192.168.20.0/24 VLAN100

interface GigabitEthernet 2/0.100
encapsulation dot1q 100
ip address 192.168.20.3
CE2
VLAN100

ƒ CE routers sub-interface on same VLAN
Can also be just port based (NO VLAN)
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

Direct Attachment VFI Configuration
l2 vfi VPLS-A manual l2 vfi VPLS-A manual
vpn id 56 vpn id 56
neighbor 2.2.2.2 encapsulation mpls neighbor 1.1.1.1 encapsulation mpls
neighbor 3.3.3.3 encapsulation mpls neighbor 3.3.3.3 encapsulation mpls

1.1.1.1 2.2.2.2
PE1 MPLS PE2
CE1 MPLSCore
Core CE2
pos4/1 pos4/3
gi3/0 gi4/4
VLAN100 pos3/0 pos3/1 VLAN100

PE3
gi4/2
3.3.3.3 CE2
VLAN100
l2 vfi VPLS-A manual
ƒ Create the Pseudo vpn id 56

Wires between N-PE
neighbor 2.2.2.2 encapsulation mpls
neighbor 1.1.1.1 encapsulation mpls
routers
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14
14688_05_2008_c1.scr
Direct Attachment CE Router (VLAN
Based)
ƒ Same set of commands on each PE
ƒ Configured on the CE facing interface
1.1.1.1 2.2.2.2
PE1 MPLS PE2
CE1 MPLSCore
Core CE2
pos4/1 pos4/3
gi3/0 gi4/4
VLAN100 pos3/0 Interface
pos3/1 GigabitEthernet3/0VLAN100
switchport
switchport mode trunk
switchport
PE3 trunk encapsulation dot1q
gi4/2
switchport trunk allowed vlan 100
3.3.3.3
!
CE2
Interface vlan 100
VLAN100 no ip address
This command associates the xconnect vfi VPLS-A
VLAN with the VPLS instance !
vlan 100
VLAN100 = VCID 56
state active

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Calculating Core MTU Requirements
ƒ Core MTU ≥ Edge MTU + Transport Header + (MPLS Label Stack
* MPLS Header Size)
ƒ Edge MTU is the MTU configured in the CE-facing PE interface
ƒ Examples (all in Bytes):

MPLS MPLS
Edge Transport Total
Stack Header

EoMPLS Port Mode 1500 14 2 4 1522

EoMPLS VLAN Mode 1500 18 2 4 1526

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15
14688_05_2008_c1.scr
End to End VPLS and EoMPLS Design

Layer 3 Core
Intranet
WCore1 WCore2 ECore1 ECore2

WMC1 EMC1
DC Core Ten3/0/0 Ten3/0/0 DC Core
Po1
Po1
WAgg1 EAgg2
Po4 Ten4/0/0 Ten4/0/0 Po2 Po4
Po2
VPLS / EoMPLS
Po3 Po3 Po4 Domain Po4 Po3 Po3
Po2 Po2
Ten4/0/0 Ten4/0/0
Agg WAgg2 EAgg1 Agg
Po1 Ten3/0/0 Ten3/0/0
Po1

WMC2 EMC2

Access Access

L2 Links (GE or 10GE)
L3 Links (GE or 10GE)

Loss of Link/Node
Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

Access to Aggregation Connections
ƒ Rapid PVST is existing protocol, and no
desire to force a change
ƒ Aggregation switches are root for all
intra-DC VLANs
Agg
ƒ The peer aggregation switch is
secondary root
ƒ DO NOT customize the bridge priorities
to lower values Access

ƒ HSRP tested for first hop redundancy
from server (more later)
Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16
14688_05_2008_c1.scr
Layer 3 Aggregation and Core
Connections
IF dual supervisor modules, need non-stop forwarding (NSF)
under routing process

Layer 3
ƒ Layer 3 connections from DC Enterprise Core
Core to Enterprise Core
ƒ Aggregation switch L3
connected to DC Core
ƒ Hanging L3 links in diagram,
are to Metro Core switches DC Core
which are Ethernet over
MPLS links
ƒ Hanging L3 links are for
peering the DC Cores in Agg
each location in a point-to-
point scenario
Bidirectional forwarding detection (bfd)
interval 100 min_rx 100 multiplier 3

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

EoMPLS / VPLS Infrastructure
ƒ Loopbacks chosen as peering points for
EoMPLS and VPLS xconnects
ƒ Horizontal links represent 10GE on
DWDM service between data centers
(alternate paths)
ƒ Vertical links represent intra-DC 10GE
connections VPLS / EoMPLS
Domain
ƒ Mpls LDP enabled globally (not a full P /
PE MPLS implmentation)
ƒ Links to/from aggregation switches for Metro Core Metro Core
Layer 2, are storm-control limited for
broadcasts and multicasts to 1% (protect
data plane)
ƒ MTU increased to 1522 bytes on the L3
BRKDCT-2840
MPLS links for the MPLS tagging
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17
14688_05_2008_c1.scr
Metro Switch Interconnectivity

Ten3/0/0 Ten3/0/0

Ten4/0/0
Ten4/0/0 IGP Routing Process
connecting
MPLS PE’s
Ten4/0/0
Ten4/0/0

Ten3/0/0 Ten3/0/0
Metro Core Metro Core

L3 Links (10GE)

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

Metro Switch Interconnectivity (EIGRP)
ƒ Layer3 interface between the Metro Switches
configured for MPLS
mpls label protocol ldp
!
interface TenGigabitEthernet3/0/0
description MPLS Interface to peer N-PE in DC #2
mtu 1522
ip address 192.168.1.1 255.255.255.252
ip hello-interval eigrp 5 1
ip hold-time eigrp 5 3
ip authentication mode eigrp 5 md5
ip authentication key-chain eigrp 5 password
logging event link-status
load-interval 30
udld port disable
mls qos trust dscp
mpls ip
!
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18
14688_05_2008_c1.scr
Metro Switch Interconnectivity (EIGRP)
(Cont.)

!
interface TenGigabitEthernet4/0/0
description MPLS Interface to local peer N-PE
mtu 1522
ip address 192.168.1.9 255.255.255.252
ip hello-interval eigrp 5 1
ip hold-time eigrp 5 3
ip authentication mode eigrp 5 md5
ip authentication key-chain eigrp 5 password
logging event link-status
load-interval 30
udld port disable
mls qos trust dscp
mpls ip

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

Metro Switch interconnectivity (OSPF)

!
interface TenGigabitEthernet3/0/0
description MPLS Interface to peer N-PE in DC #2
mtu 1522
ip address 192.168.1.1 255.255.255.252
ip ospf network point-to-point
logging event link-status
load-interval 30
udld port disable
mls qos trust dscp
mpls ip

router ospf 1
log-adjacency-changes
nsf cisco
timers throttle spf 50 100 5000
timers lsa arrival 0
timers pacing flood 15
network 192.168.0.0 0.0.255.255 area 0

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19
14688_05_2008_c1.scr
Metro Switch interconnectivity (OSPF)
(Cont.)

!
interface TenGigabitEthernet4/0/0
description MPLS Interface to local peer N-PE
mtu 1522
ip address 192.168.1.9 255.255.255.252
ip ospf network point-to-point
logging event link-status
load-interval 30
udld port disable
mls qos trust dscp
mpls ip

router ospf 1
log-adjacency-changes
nsf cisco
timers throttle spf 50 100 5000
timers lsa arrival 0
timers pacing flood 15
network 192.168.0.0 0.0.255.255 area 0

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

Metro Switch Example IP Addressing

ƒ Loopbacks and WAN links use 192.168.0.0 addressing
ƒ WestMetroCore1
Loopback0 IP Address = 192.168.255.250/32
Interface Te3/0/0 IP Address = 192.168.1.1/30
Interface Te4/0/0 IP Address = 192.168.1.9/30

ƒ WestMetroCore2
Loopback0 IP Address = 192.168.255.251/32
Interface Te3/0/0 IP Address = 192.168.1.5/30
Interface Te4/0/0 IP Address = 192.168.1.10/30

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20
14688_05_2008_c1.scr
Metro Switch Example IP Addressing
(Cont.)

ƒ EastMetroCore1
Loopback0 IP Address = 192.168.255.252/32
Interface Te3/0/0 IP Address = 192.168.1.2/30
Interface Te4/0/0 IP Address = 192.168.1.13/30

ƒ EastMetroCore2
Loopback0 IP Address = 192.168.255.253/32
Interface Te3/0/0 IP Address = 192.168.1.6/30
Interface Te4/0/0 IP Address = 192.168.1.14/30

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Metro Switch Routing
ƒ No Dynamic routing between the Metro Core switches and other
Data Center switches
ƒ router eigrp 5 (the routing instance for the MPLS domain, LDP
passes on this)
passive-interface default
no passive-interface TenGigabitEthernet3/0/0
no passive-interface TenGigabitEthernet4/0/0
network 192.168.0.0 0.0.255.255
no auto-summary
Nsf
ƒ Enable NSF for LDP, mpls ldp graceful-restart global configuration

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21
14688_05_2008_c1.scr
Metro Switch Interconnectivity
EastMetroCore1#sh ip route

Gateway of last resort is 10.98.128.1 to network 0.0.0.0

10.0.0.0/24 is subnetted, 1 subnets
C 10.98.128.0 is directly connected, GigabitEthernet5/2
192.168.255.0/32 is subnetted, 4 subnets
D 192.168.255.253
[90/128512] via 192.168.1.14, 3w0d, TenGigabitEthernet4/0/0
C 192.168.255.252 is directly connected, Loopback0
D 192.168.255.251
[90/128768] via 192.168.1.14, 3w0d, TenGigabitEthernet4/0/0
[90/128768] via 192.168.1.1, 3w0d, TenGigabitEthernet3/0/0
D 192.168.255.250
[90/128512] via 192.168.1.1, 3w0d, TenGigabitEthernet3/0/0
192.168.1.0/30 is subnetted, 4 subnets
D 192.168.1.8 [90/768] via 192.168.1.1, 3w0d, TenGigabitEthernet3/0/0
C 192.168.1.12 is directly connected, TenGigabitEthernet4/0/0
C 192.168.1.0 is directly connected, TenGigabitEthernet3/0/0
D 192.168.1.4 [90/768] via 192.168.1.14, 3w0d, TenGigabitEthernet4/0/0
S* 0.0.0.0/0 [1/0] via 10.98.128.1

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

EoMPLS for Layer3

Layer 3 Core
Intranet

DC Core
METRO CORE DC Core

PW – Pseudo Wires

Agg Agg
EoMPLS

Metro Core Metro Core

Access Access

L2 Links (GE or 10GE)
L3 Links (GE or 10GE)

Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22
14688_05_2008_c1.scr
EoMPLS for Layer3 Configuration
EastMetroCore1
!
interface Loopback0
description Loopback interface for PW peering
ip address 192.168.255.252 255.255.255.255
!
interface GigabitEthernet1/1
description EASTCORE:9/1 - WESTCORE:1/1
xconnect 192.168.255.250 250252 encapsulation mpls
!
interface GigabitEthernet2/1
description EASTCORE:8/1 - WESTCORE:1/2
xconnect 192.168.255.250 252250 encapsulation mpls

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

EoMPLS for Layer3 Configuration (Cont.)

WestMetroCore1
!
interface Loopback0
description Loopback interface for PW peering
ip address 192.168.255.250 255.255.255.255
!
interface GigabitEthernet1/1
description WESTCORE:1/1 - EASTCORE:9/1
xconnect 192.168.255.252 250252 encapsulation mpls
!
interface GigabitEthernet2/1
description WESTCORE:1/2 - EASTCORE:8/1
xconnect 192.168.255.252 252250 encapsulation mpls

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23
14688_05_2008_c1.scr
VPLS for Layer2

Layer 3 Core
Intranet

DC Core
METRO CORE DC Core

VFI
Agg Agg

PW – Pseudo Wires
Metro Core Metro Core

Access Access

L2 Links (GE or 10GE)
L3 Links (GE or 10GE)

Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

VPLS for Layer2
l2 vfi vlan3700 manual l2 vfi vlan3700 manual
vpn id 3700 Layer 3 Core vpn id 3700
neighbor 192.168.255.251 encapsulation mplsIntranet neighbor 192.168.255.250 encapsulation mpls
neighbor 192.168.255.252 encapsulation mpls neighbor 192.168.255.251 encapsulation mpls
neighbor 192.168.255.253 encapsulation mpls neighbor 192.168.255.253 encapsulation mpls

DC Core
METRO CORE DC Core

Agg Agg

PW – Pseudo Wires
Metro Core Metro Core

l2 vfi vlan3700 manual l2 vfi vlan3700 manual
vpn id 3700
Access vpn id 3700 Access
neighbor 192.168.255.250 encapsulation mpls neighbor 192.168.255.250 encapsulation mpls
L2 Links (GE or 10GE)
neighbor 192.168.255.252 encapsulation mpls neighbor 192.168.255.251 encapsulation mpls
L3 Links (GE or 10GE)
neighbor 192.168.255.253 encapsulation mpls neighbor 192.168.255.252 encapsulation mpls
Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24
14688_05_2008_c1.scr
VPLS for Layer2
interface Vlan3700
Layer 3 Core interface Vlan3700
no ip address Intranet no ip address
load-interval 30
load-interval 30
xconnect vfi vlan3700
xconnect vfi vlan3700

DC Core
METRO CORE DC Core

VLAN
3700 Agg
Agg

PW – Pseudo Wires
Metro Core Metro Core

interface Vlan3700 interface Vlan3700
Access no ip address no ip address Access
load-interval 30 load-interval 30
xconnect vfi vlan3700 L2 Links (GE or 10GE)
xconnect vfi vlan3700
L3 Links (GE or 10GE)

Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

VPLS for Layer2 Configuration
WestMetroCore1
!
interface Loopback0
description Loopback interface for PW peering
ip address 192.168.255.250 255.255.255.255
!
l2 vfi vlan3700 manual
vpn id 3700
neighbor 192.168.255.252 encapsulation mpls
neighbor 192.168.255.251 encapsulation mpls
neighbor 192.168.255.253 encapsulation mpls
!
interface Vlan3700
no ip address
load-interval 30
xconnect vfi vlan3700
!
interface Port-channel1
description WESTAGG1
switchport trunk allowed vlan 3700-3760

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25
14688_05_2008_c1.scr
Spanning Tree

ƒ Spanning-Tree BPDU(s) will NOT traverse
between the Data Centers – It isn’t needed
(and blocked) with VPLS
ƒ We still need to control data plane layer 2
events (i.e. limit the traffic)
ƒ Since enterprises want dual N-PE devices, and
VPLS blocks BPDU’s, we require method to
block within a local DC

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

Spanning Tree – Without N-PE MST
Region or EEM
Broadcast, Multicast,
Unknown Unicast
Layer 3 Core
Intranet

DC Core DC Core

VPLS / EoMPLS
Domain

Agg Agg
RSTP RSTP
X X X X
Metro Core Metro Core

Access Access
ƒ Without layer 2 link between Metro Switches there is a
loop. Each side has a “U” shape with Metro and Agg
switches. Broadcast storms.
L2 Links (GE or 10GE)
Server Farm Server Farm
L3 Links (GE or 10GE)
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26
14688_05_2008_c1.scr
Spanning Tree - MST
Root Bridge in West DC Root Bridge in East DC
for all VLANs that Go Layer 3 Core for all VLANs that Go
Between Data Centers Intranet Between Data Centers

DC Core DC Core

Single L2
Single L2
MST Bridge
MST Bridge
VPLS / EoMPLS
MST Domain MST
Agg Agg

RSTP RSTP
Metro Core Metro Core

Access Access

L2 Links (GE or 10GE)
L3 Links (GE or 10GE)

Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Spanning-Tree
ƒ MST (802.1s) chosen to present Metro Cores as single
bridge
ƒ Red Layer 2 link is access port channel with a VLAN that
represents the MST0 instance to make the MST group
ƒ MST bridge priority set to 0 (Metro Core will be root of
Inter-DC VLANs)
ƒ Spanning tree root-guard enabled on Metro Cores toward
aggregation switches (protects in case the red MST link
fails)
ƒ Only inter-DC VLANs allowed on trunks to/from
aggregation switches Single L2
MST Bridge
ƒ Set spanning-tree vlan cost to set the priorities on the agg
switches links to metro core – will allow us to put some
VLANs on upper Metro Core, some on Lower by default

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27
14688_05_2008_c1.scr
Spanning Tree - MST
interface Port-channel4
description Port Channel to WestMetroCore1
Layer 3cost
spanning-tree vlan 3702,3706,3710,3714,3718 Core 8
Intranet

DC Core DC Core

Single L2
Single L2
MST Bridge
MST Bridge X
X VPLS / EoMPLS
MST Domain MST
X X
Agg Agg

X RSTP X X
RSTP X
X X X X
Metro Core Metro Core

Access Access

interface Port-channel4
description Port Channel to WestMetroCore2
spanning-tree vlan 3700,3704,3712,3716 cost 8
Server Farm Server Farm

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

MST Configuration
ƒ Configuration on Primary N-PE (root of Inter-DC
VLANs):
spanning-tree mode mst
spanning-tree extend system-id
!
spanning-tree mst configuration
name WESTDC
revision 50
!
spanning-tree mst 0 priority 0
!
interface Port-channel5
description port-channel to westagg1
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 3700
switchport mode trunk
storm-control broadcast level 1.00
storm-control multicast level 1.00
spanning-tree guard root
!
interface Port-channel6
description port-channel to peer N-PE
switchport
switchport mode access
BRKDCT-2840 !
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28
14688_05_2008_c1.scr
Alternative VPLS Solution

ƒ If you have different requirements such as:
QinQ
Not Moving STP Root for Inter DC vLANs

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

Storm Control
ƒ Traffic storms when packets flood the LAN
ƒ Traffic storm control feature prevents LAN ports from being
disrupted by broadcast or multicast flooding
ƒ Rate limiting for unknown unicast (UU) must be handled at Data
Center aggregation; unknown unicast flood rate-limiting (UUFRL):
mls rate-limit layer2 unknown rate-in-pps [burst-size]
ƒ Storm Control is configured as a percentage of the link that storm
traffic is allowed to use.
storm-control broadcast level 1.00 (% of b/w may vary – need to
baseline)
storm-control multicast level 1.00 (% of b/w may vary – need to
baseline)

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29
14688_05_2008_c1.scr
3 or More Data Center Locations

ƒ EoMPLS will allow multiple point to point links between
any 2 sites
ƒ Can build a full mesh of links to interconnect layer 3
devices
ƒ VPLS scales by adding peer xconnects under the VFI
in the IOS configuration
ƒ Split horizon with MST local to data center will make for
simple growth
ƒ Limits dependant on amounts of L2 traffic especially
multicast, as these are replicated on each PW

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

3 Site Drawing with EoMPLS PW’s for L3

Server Farm

Server Farm Server Farm
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30
14688_05_2008_c1.scr
3 Site Drawing with VPLS PW’s for L2

Server Farm

Server Farm Server Farm
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

Summary of Tagging Section

ƒ EoMPLS well suited for Router-Router links
ƒ VPLS well suited for Switch-Switch links
ƒ Straightforward to scale to multiple Data Center
locations
ƒ MST and EEM semiphores both work well
One tradeoff is QinQ support against number of VLANs to pass
Another is the root of the spanning tree for inter-DC VLANs

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31
14688_05_2008_c1.scr
Session Agenda

ƒ Data Center Interconnection – Common Scenarios
and Terms
ƒ Options for Layer 2 Interconnectivity
ƒ Recommended Designs for Optimizing Traffic Flows
ƒ EoMPLS and VPLS Stability Testing
ƒQ&A

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

Flow Optimization and Symmetry
Site Selection and Inbound Flows
First Hop Outbound
Intra/Inter Site Failover
VMware DRS Case

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32
14688_05_2008_c1.scr
Optimizing Traffic Patterns and HA
Design
ƒ Many tradeoffs in understanding flows in multi-DC design
ƒ Slides that follow are a specific recommendation that meets the following
requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s
Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.)
Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS,
etc.)
Site failure will allow failover, with IP mobility to resolve caching issues
Single points of failure in gear wont cause site failover
Indicate a location preference for a service to the Layer 3 network
If broadcast storm in DC, limit impacts to other DC’s
Ability to connect to services in both DC locations (active/active per application)
DNS to round-robin clients to DC
Allow backup server farms with same service VIP (for backup connections on site fail)
ƒ This is a solution in production at some customers

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

Sample Cluster – Service Normally in Left DC
Default Gateway Shared Between Sites
10.1.1.0/25 & 10.1.1.128/25 advertised into L3 10.1.1.0/24 advertised into L3
-EEM or RHI can be used to get very granular Layer3 Core Backup should main site go down

Active/Standby Pairs: Active/Standby Pairs:
FW FW
IPS IPS
NLB NLB
SSL SSL
WAN Accel WAN Accel

VLAN A VLAN A
10.1.1.1 HSRP Group 1 10.1.1.1 HSRP Group 1
Priority 140 and 130 Priority 120 and 110

Cluster Node A Cluster Node B

Cluster VLAN C (L2 Only)

Cluster VLAN D (L2 Only)

-Cluster VIP = 10.1.1.100 Preempt -Cluster VIP = 10.1.1.100
-Default GW = 10.1.1.1 -Default GW = 10.1.1.1
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33
14688_05_2008_c1.scr
Optimizing Traffic Patterns and HA
Design
ƒ Many tradeoffs in understanding flows in multi-DC design
ƒ Slides that follow are a specific recommendation that meets the following
requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s
Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.)
Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS,
etc.)
Site failure will allow failover, with IP mobility to resolve caching issues
Single points of failure in gear wont cause site failover
Indicate a location preference for a service to the Layer 3 network
If broadcast storm in DC, limit impacts to other DC’s
Ability to connect to services in both DC locations (active/active per application)
DNS to round-robin clients to DC
Allow backup server farms with same service VIP (for backup connections on site fail)
ƒ This is a solution in production at some customers

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

Sample Cluster – Broadcast Storm in Left DC
Broadcast, Multicast, Unknown Unicast
10.1.1.0/25 & 10.1.1.128/25 advertised into L3 10.1.1.0/24 advertised into L3
-EEM or RHI can be used to get very granular Layer3 Core Backup should main site go down

VLAN A VLAN A
10.1.1.1 HSRP Group 1 10.1.1.1 HSRP Group 1
Priority 140 and 130 Priority 120 and 110

Cluster Node A Cluster Node B

Cluster VLAN C (L2 Only)

Cluster VLAN D (L2 Only)

-Cluster VIP = 10.1.1.100 Preempt -Cluster VIP = 10.1.1.100
-Default GW = 10.1.1.1 -Default GW = 10.1.1.1
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34
14688_05_2008_c1.scr
Optimizing Traffic Patterns and HA
Design
ƒ Many tradeoffs in understanding flows in multi-DC design
ƒ Slides that follow are a specific recommendation that meets the following
requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s
Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.)
Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS,
etc.)
Site failure will allow failover, with IP mobility to resolve caching issues
Single points of failure in gear wont cause site failover
Indicate a location preference for a service to the Layer 3 network
If broadcast storm in DC, limit impacts to other DC’s
Ability to connect to services in both DC locations (active/active per application)
DNS to round-robin clients to DC
Allow backup server farms with same service VIP (for backup connections on site fail)
ƒ This is a solution in production at some customers

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

Active/Active Per Application (VIP at Either)
10.1.1.0/24 advertised into L3
10.1.1.0/25 & 10.1.1.128/25 advertised into L3
Backup should main site go down
-EEM or RHI can be used to get very granular
Layer3 Core
10.1.1.0/24 advertised into L3
Backup should main site go down 10.1.2.0/25 & 10.1.2.128/25 advertised into L3
-EEM or RHI can be used to get very granular

DNS:
www-hr.acme.com -> 10.1.1.100
www-news.acme.com -> 10.1.2.100

10.1.2.1 HSRP Group 2
Priority 140 and 130

VLAN A VLAN A

10.1.2.1 HSRP Group 2
10.1.1.1 HSRP Group 1 Priority 120 and 110 10.1.1.1 HSRP Group 1
Priority 140 and 130 Priority 120 and 110

Cluster Node A Cluster Node B

Cluster VLAN C (L2 Only)
-Cluster VIP = 10.1.2.100
-Default GW = 10.1.2.1 Cluster VLAN D (L2 Only)

-Cluster VIP = 10.1.1.100 Preempt -Cluster VIP = 10.1.1.100
-Default GW = 10.1.1.1 -Default GW = 10.1.1.1 -Cluster VIP = 10.1.2.100 Preempt
-Default GW = 10.1.2.1
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35
14688_05_2008_c1.scr
Optimizing Traffic Patterns and HA
Design
ƒ Many tradeoffs in understanding flows in multi-DC design
ƒ Slides that follow are a specific recommendation that meets the following
requirements:
Minimize inter-DC traffic to maintenance/failure scenario’s
Ability to extend clusters between locations (OS, FS, DB, VMware DRS, etc.)
Desire to keep flows symmetric in/out of a location for DC services (FW, NLB, IPS, WAAS,
etc.)
Site failure will allow failover, with IP mobility to resolve caching issues
Single points of failure in gear wont cause site failover
Indicate a location preference for a service to the Layer 3 network
If broadcast storm in DC, limit impacts to other DC’s
Ability to connect to services in both DC locations (active/active per application)
DNS to round-robin clients to DC
Allow backup server farms with same service VIP (for backup connections on site fail)
ƒ This is a solution in production at some customers

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

Active/Active Per Application (VIP at Both)
10.1.1.0/24 advertised into L3
10.1.1.0/25 & 10.1.1.128/25 advertised into L3
Backup should main site go down
-EEM or RHI can be used to get very granular
Layer3 Core
10.1.1.0/24 advertised into L3
Backup should main site go down 10.1.2.0/25 & 10.1.2.128/25 advertised into L3
-EEM or RHI can be used to get very granular

DNS:
www-hr.acme.com -> 10.1.1.100
10.1.2.100

10.1.2.1 HSRP Group 2
Priority 140 and 130

VLAN A VLAN A

10.1.2.1 HSRP Group 2
10.1.1.1 HSRP Group 1 Priority 120 and 110 10.1.1.1 HSRP Group 1
Priority 140 and 130 Priority 120 and 110

Cluster Node A Cluster Node B

Cluster VLAN C (L2 Only)
-Cluster VIP = 10.1.2.100
-Default GW = 10.1.2.1 Cluster VLAN D (L2 Only)

-Cluster VIP = 10.1.1.100 Preempt -Cluster VIP = 10.1.1.100
-Default GW = 10.1.1.1 -Default GW = 10.1.1.1 -Cluster VIP = 10.1.2.100 Preempt
-Default GW = 10.1.2.1
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36
14688_05_2008_c1.scr
Session Agenda

ƒ Data Center Interconnection – Common Scenarios
and Terms
ƒ Options for Layer 2 Interconnectivity
ƒ Recommended Designs for Optimizing Traffic Flows
ƒ EoMPLS and VPLS Stability Testing
ƒQ&A

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

CPOC Tested
Failover Numbers

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37
14688_05_2008_c1.scr
EoMPLS and VPLS Stability Testing

ƒ Testing of link outage scenarios
Pulling fiber connections
Administratively shutting down interfaces
Pulling active cards and supervisors

ƒ Testing of failure and fail-back timing
ƒ Tests grouped by location in the network
Metro Core failures
Aggregation failures
Layer 3 Core failures

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

Metro Core Failover/Failback Tests

Link Down Link Up
Top Rail Pull 105mSec 1mSec x x
Top Rail Admin 133mSec 1mSec

xx
Vertical Rail Pull 0 0

2x10GE Card Fail 1.2Sec 5.4Sec

2x10GE Card AS 718mSec 5.7Sec

Node Power Off 379mSec 6.4Sec

MST Link Pull 0 0

Primary Sup Pull 516mSec 0

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38
14688_05_2008_c1.scr
Embedded Event Manager

ƒ Scripting based on events
ƒ Script initiator is a tracking of node reachability
ƒ Bring up interfaces in a known order
ƒ Allow traffic flows based on a time delay

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

EEM Policy to Handle VPLS Down
ƒ In case VPLS redundancy is not possible an EEM policy can be used to prevent black-hole
when VPLS path goes down
ƒ Since, the LAN modules come-up before the WAN Modules used EEM and EOT to control
event manager applet TRACK_ES20_DOWN
event track 40 state down
track 1 interface GigabitEthernet3/0/0 line-protocol action 1.0 cli command "config t"
! action 2.0 cli command "interface TenGigabitEthernet4/4"
track 2 interface GigabitEthernet3/0/1 line-protocol action 3.0 cli command "shutdown"
! action 4.0 syslog msg "EEM has shutdown the SVI's"
track 20 ip route 10.1.133.226 255.255.255.255 reachability event manager applet TRACK_ES20_UP
! event track 40 state up
track 21 ip route 10.1.133.222 255.255.255.255 reachability action 1.0 cli command "config t"
! action 2.0 cli command "interface TenGigabitEthernet4/4"
track 25 list boolean and action 3.0 cli command "no shutdown"
object 20 action 4.0 syslog msg "EEM has enabled the Ten4/4"
object 21 event manager applet UP_TEN4/4
delay up 90 event track 25 state up
! action 1.0 cli command "config t"
track 40 list boolean or action 2.0 cli command "interface TenGigabitEthernet4/4"
object 1 action 3.0 cli command "no shutdown"
object 2 action 4.0 syslog msg "EEM has unshut Ten4/4"
delay up 90 event manager applet test
! event syslog pattern "Module 4: Passed Online Diagnostics"
action 1.0 cli command "config t"
action 2.0 cli command "interface TenGigabitEthernet4/4"
action 3.0 cli command "shutdown"
10.1.133.226 and 222 are remote N-PEs action 4.0 syslog msg "EEM has shutdown Ten4/4"
Gig3/0/0 and 3/0/1 are VPLS uplinks
TenGigE4/4 is the link to local Agg switch
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39
14688_05_2008_c1.scr
Aggregation Failover/Failback Tests

Link Down Link Up
Left Agg Pull 344mSec 5.7Sec

Left Agg Admin 668mSec 5.7Sec

Access Link Pull 63mSec (L2),
123mSec (L3)
17mSec (L2),
101mSec (L3) x
Agg Port-Ch Fail 0 0
x
Agg Port-Ch AS 0 0

x

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

Core Failover/Failback Tests
x
x
x
Link Down Link Up
Core-Core Shut 0 0

Core-MC Shut 0 0

Core-Core with 0 0
Core-MC Shut
Core Sup Pull 734mSec 0

Core Reload 0 0

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved. 40
14688_05_2008_c1.scr
Q and A

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

Recommendations
ƒ Recommended Reading:
MPLS and VPN Architectures, Volume II
by Jim Guichard
Network Virtualization by Victor Moreno
ƒ Check the Recommended Reading flyer
for suggested books
ƒ Related technology breakouts:
BRKAGG-2000 Implementation and
utilization of Layer 2 VPN technologies
TECAGG-2003 Layer 2 Virtual Private
Networks – Converged IP/MPLS Network
ƒ NSITE is compiling test results for both
the MST N-PE and EEM Solution

Available Onsite at the Cisco Company Store
BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved. 41
14688_05_2008_c1.scr
Complete Your Online
Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

BRKDCT-2840
14688_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84

© 2006, Cisco Systems, Inc. All rights reserved. 42
14688_05_2008_c1.scr