SAN Extension Design and Operation

BRKSAN-2704

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

1

Design Section Agenda
Basic SAN Extension Principles
Dual Path, HA, and Transport Options

Enhancing SAN Extension
Optical: Extended B2B Credits, B2B Credit Spoofing, Port Tracking FCIP: Compression, Encryption, Application Acceleration, QOS

Advanced FCIP SAN Extension Designs
Multiple Tunnels, Multiple Links

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

SAN Design Principles

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

4

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

2

Design Criteria

Factors to Consider for SAN Extension Applications using SAN Extension
Synchronous/asynchronous replication, data backup/restore

Application latency requirements
Applications that use synchronous replication may be impacted

Application throughput requirements
Determines bandwidth requirements

Transport options available
What choices are available for SAN Extension

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

Typical SAN Design
Site A Access Fabrics Replication Fabrics

FC

Servers with two Fibre Channel connections to storage arrays for high availability
Use of multipath software is required in dual fabric host design

DC Interconnect Network

SAN extension fabrics typically separate from host access fabrics
Replication fabric requirements vary by replication method deployed

Site B Replication Fabrics

FC

“A” Fabrics
BRKSAN-2704 14571_04_2008_c1

“B” Fabrics
Cisco Public

© 2008 Cisco Systems, Inc. All rights reserved.

6

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

3

Basic HA SAN Extension Network
Site A Access VSANs Replication VSANs

High-Availability Replication Design:
FC

Conventional approach is dual fabrics (e.g., yellow VSAN and blue VSAN) over distance “Client protection”—arrays provide protection against failures in either fabric May be augmented with additional “network protection” via port channels and/or optical protection schemes
“B” VSANs
Cisco Public

DC Interconnect Network

Site B Replication VSANs

FC

“A” VSANs
BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

7

SAN Extension Design: Adding Link HA
Site A Access VSANs Replication VSANs

FC

Port Channels Increase Resilience for High-Availability with FC or FCIP Links Appears as a single logical link (up to sixteen member links)
Port Channels

Protecting the fabric from network failure

Site B Replication VSANs

FC

Route port-channel member links over diverse geographic paths Load balancing on SRCID/ DESTID or SRCID/DESTID/ OXID basis (unidirectional per VSAN)
“B” VSANs
Cisco Public

“A” VSANs
BRKSAN-2704 14571_04_2008_c1

SCSI exchange is smallest atomic unit, so frame order kept intact
8

© 2008 Cisco Systems, Inc. All rights reserved.

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

4

Hybrid SAN Extension Implementations
Hub and Spoke
FC
FC FC FC FC FC FC

FC

FC

IP Network

FC
FC

Central Site

Remote Sites

Multi-Hop
Primary DC
FC FC

Secondary DC
FC FC

Backup Site
IP Network
FC

FC

DWDM

Synchronous Replication
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Asynchronous Replication/Backup
9

SAN Extension Solutions

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

5

SAN Extension Solutions
Internet

SAN Extension Design Options
Fibre Channel over IP (FCIP) • Asynchronous data replication (T1-multiGigabit • Higher latency but longer distances SONET/SDH • Mostly asynchronous data replication • Typically OC-3 to OC-48 CWDM • Synchronous and asynchronous data replication • Low interface density: 1–8 FC/FICON interfaces • High bandwidth: 1G-2G lambdas DWDM Synchronous and asynchronous data replication High interface density High bandwidth: 2.5G and 10G lambdas IP WAN

Stateful Firewalls Content Caching
High Density Multilayer LAN Switch

Server Load Balancing WAAS Intrusion Detection Front-End Application Servers with iSCSI Back-End Application Servers The Integrated Data Center Encompassing Extensive Availability LAN/SAN services

iSCSI
HighDensity Multilayer SAN Director

The Remote Data Center
Backup and/or Hot-Standby Servers EnterpriseClass Tape Arrays Enterprise-Class Storage Arrays
11

Fibre Channel over IP Fibre Channel over SONET Fibre Channel over CWDM Fibre Channel over DWDM ESCON over DWDM

SONET/SDH CWDM Point-Point DWDM Network
Cisco Public

Enterprise-Class Storage Arrays
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved.

SAN Extension Technology Options
Increasing Distance Data Center Campus Metro Regional National Global Dark Fiber Sync Optical
Limited by Optics (Power Budget) Limited by Optics (Power Budget) Limited by BB_Credits

CWDM Sync (1,2 Gbps)

DWDM Sync (1,2,4,10 Gbps per λ) SONET/SDH Sync (1,2 Gbps + Subrate)

Async

IP

MDS9000 FCIP Sync (Metro Eth)

Async (WAN,1 Gbps)

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

12

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

6

Dark Fiber
Single 1/2/4/10 Gbps FC link per fiber pair
FC

Port Channel 2-16 1/2/4/10 Gbps FC over Two Diverse Paths Diverse Paths— Multiple Fiber Pairs Each Path

SW (850 nm) over 62.5/ 125 µm multimode SW (850 nm) over 50/125 µm multimode LW (1310 nm) over 9/125 µm single mode

Distance Based on Fiber Type, Optic Type, Link Speed

Client protection only; Upper Layer Protocol (ULP), either SAN or application, responsible for failover protection

FC

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

Coarse Wavelength Division Multiplexing (CWDM)
1470 nm 1510 nm 1550 nm 1590 nm

OADM Mux/Demux

1490 nm 1530 nm 1570 nm 1610 nm

8-channel WDM at 20-nm spacing (cf DWDM at <1-nm spacing)
1470, 1490, 1510, 1530, 1550, 1570, 1590, 1610 nm

“Colored” CWDM SFPs (or GBICs) used in FC switches (no transponder required) Optical multiplexing done in CWDM OADM (optical add/drop multiplexer)
Passive (unpowered) device; just mirrors and prisms

Up to 30-dB power budget (36-dB typical) on SM fiber
~100-km point-to-point or ~40-km ring

1/2 gigabit Fibre Channel and 1 Gigabit Ethernet currently
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

7

Two-Site CWDM Storage Network
HA resilience against fiber cut—“client” protection
FC

Diverse Paths: One Fiber Pair Each Path Network

4-member port channel— 2 x 2 diverse paths
Port Channel 2-16 1/2 Gbps FC over Two Diverse Paths Network
MUX-4

Port channel appears as single logical link E_Port or TE_Port for carriage of VSANs Load balance by src/dst (or src/dst/oxid) Fiber cut will halve capacity from 16 Gbps to 8 Gbps but not alter fabric topology— no FSPF change

MUX-4

MUX-4

Pass Pass
MUX-4

2 Gbps CWDM SFPs

FC

MUX-8 would double capacity or leave spare wavelengths for GigE channels
Cisco Public

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

15

CWDM Optics Without Multiplexor
CWDM optics do not require MUX
4 Fibre Paths Between Each Switch
FC

If dark fiber available, can be used like typical SFPs

Port Channel 4 x 2 Gbps

Can use different wavelengths or the same wavelengths on all interfaces Use of optical attenuators may be required for shorter distance fiber runs
Optical power meter used to measure signal strength

2 Gbps CWDM SFPs— Different Wavelengths
FC

2 Gbps CWDM SFPs—Same Wavelength

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

8

Dense Wavelength Division Multiplexing (DWDM)
Higher density than CWDM
32 lambdas or channels in narrow band around 1550 nm at 100G-Hz spacing (0.8 nm)

Erbium-Doped Fiber Amplifier (EDFA) amplifiable allows for longer distances than CWDM Carriage of 1, 2, 4, or 10 Gbps FC, FICON, GigE, 10 GigE, ESCON, IBM GDPS Data center to data center Protection options: client, splitter, or linecard

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

17

DWDM Protection Alternatives for Storage
Optical Splitter Protection
Optical Splitter Working Lambda

Single transponder required Protects against fiber breaks Failover causes loss of light (and fabric change if only link)

Protected Lambda

Linecard or Y-Cable Protection
Y-cable Working Lambda

Dual transponders required
More expensive than splitter-based protection

Transmits over both circuits, but only one accepted
Protected Lambda
BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

9

DWDM HA Storage Network Topology
Client protection recommended
FC

Fabric and application responsible for failover recovery
Port Channel 2–16 1/2/4/10 Gbps FC over Two Diverse Paths

Diverse Paths— One-Fiber Pair Each Path DWDM Ring

Port channel provides resilience
Port-channel members follow diverse paths Single fiber cut will not affect fabric (no RSCNs, etc.) Use “Src/Dst” hash for load balancing (rather than “Src/Dst/Oxid” per exchange) for each extended VSAN

FC

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

19

Fibre Channel over SONET/SDH
FC over SONET/SDH (FCoS) follows same distance rules as other optical technologies
Port Channel 2–16 1/2 Gbps FC over Two Diverse Paths SONET / SDH Network Diverse Network Paths

FC

BB_Credits in Fibre Channel switch limits distance

Outage in SONET/SDH network will not cause loss of light
Recovers in <50 ms May cause some loss BB_Credit loss from in flight traffic MDS9000 will recover lost BB_Credits

FC

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

10

Fibre Channel over IP (FCIP) Design
Same port channeling and VSAN trunking rules apply as with FC links Port channel individual FCIP links to separate Ethernet switches/routers
Each WAN link carries two FCIP tunnels Load balancing on SRCID/DESTID or SRCID/DESTID/OXID basis (unidirectionally per VSAN) Certain replication protocols require SRCID/DESTID load balancing
FC

FC

Diverse Network Paths

Port Channel 2–16 FCIP Interfaces Over Diverse Paths

FICON, IBM PPRC, HP CA-EVA
Cisco Public

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

21

FCIP Frame Detail
Ethernet IP Header Header 14 20 TCP TCP FCIP Header Opts Header 20 94 FCIP Overhead for Ethernet Frames: 94 Byte Header + 4 Byte CRC = 98 Bytes 12 28 EISL opt Hdr Hdr 4 8 0-16 FC Frame Ethernet CRC32 4 SOF

Max 2148 (E_Port) + EISL and Opt Headers EISL and Optional Headers If TE_Port, then 8 Bytes Added to FC Frame (After SOF) for VSAN Routing

Max Fiber Channel frame is 2148 bytes plus optional extras FCIP will segment and reassemble FC frames if MTU too small (TCP payload on second or subsequent packets) Jumbo frames may increase performance
IP MTU of 2300 avoids splitting of TCP frames

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

22

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

11

Storage Traffic and TCP
Storage traffic:
Quite bursty Latency sensitive (sync apps) Requires high, instantaneous throughput

Traditional TCP:
Tries to be network sociable Tries to avoid congestion (overrunning downstream routers) Backs off when congestion detected Slow to ramp up over long links (slow start and congestion avoidance)

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

MDS FCIP TCP Behavior
Reduce probability of drops
Bursts controlled through per flow shaping and congestion window control less likely to overrun routers

Increased resilience to drops
Uses SACK, fast retransmit and shaping

Aggressive slow start q
Initial rate controlled by “min-available-bandwidth” Max rate controlled by “max-bandwidth”
Differences with Normal TCP: When congestion occurs with other conventional TCP traffic, FCIP is more aggressive during recovery (“bullying” the other traffic)
Aggression is proportional to the min-available-bandwidth configuration
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

24

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

12

Frame Buffering: FCIP and FC
Traffic Flow
FC

TCP Windowing Flow Control BB_Credit Flow Control GigE FC Receive Buffers Slower WAN Link GigE FCIP Receive Buffers BB_Credit Flow Control

FC

Backlog Here if Queue Can’t Drain Due To: Slow WAN link and long RTT Packet loss and retransmissions Many sources (only one shown) Buffer too big

FCIP presents a lower bandwidth pipe (if WAN link)
Drain rate (send rate) depends upon bandwidth and congestion

Slow ramp up of traditional TCP can cause FC frame expiry in some conditions
Mixture of slow link (e.g., <DS3/E3; retransmissions, many sources, big buffers)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

25

FCIP TCP Packet Shaping: MDS9000
Source
Shaping Avoids Congestion at This Point

Destination

Gigabit Ethernet

45 Mbps

Gigabit Ethernet

Traffic Flow
Source Sends Packets at Rate Consumable by Downstream Path

Interpacket Gap to Accommodate Slow Downstream Link (e.g., 45 mbps)

Shaper sends at a rate consumable by the downstream path
Immediately sends at “minimum-bandwidth” rate (avoids early stages of traditional slow start) Ramps up to “maximum-bandwidth” rate (using usual slow start and congestion avoidance methods)

Requirements for shaper to engage:
Min-available-bandwidth > 1/20 max-bandwidth SACK (Selective Ack) must be enabled
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

26

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

13

MDS9000 FCIP TCP Behavior
For example: a dedicated link
Entire link is always available, so… “min bandwidth” = “max bandwidth” FCIP will always send at 95% to 100% of max rate without ramp up Traffic is shaped at sending rate (max-bw)
Dedicated FCIP Link Minimum Bandwidth = Maximum Bandwidth Rate

After retransmission (congestion), send er resumes at min (= max rate)

Behavior mimics UDP “blast” but with benefits of retransmission capability and shaping

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

27

Enhancing SAN Extension

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

28

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

14

Extending Optical SAN Extension
BB_Credits and Distance
1 Gbps FC ~2 km per Frame

2 Gbps FC

~1 km per Frame

4 Gbps FC

~½ km per Frame

16 Km

BB_Credits are used to ensure enough FC frames in flight A full (2112 byte) FC frame is approx 2-km long @ 1 Gbps, 1-km long @ 2 Gbps and ½-km long at 4 Gbps As distance increases, the number of available BB_Credits need to increase as well Insufficient BB_Credits will throttle performance—no data will be transmitted until R_RDY is returned
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

29

Extending Optical SAN Extension
Fibre Channel Frame Buffering
Traffic Flow
FC FC

BB_Credit Flow Control

BB_Credit Flow Control

BB_Credit Flow Control

2-8 BB_Credit

16-255 BB_Credit FC Receive Buffers FC Receive Buffers

2-8 BB_Credit

Buffer to buffer credits (BB_Credit) are negotiated between each device in a FC fabric; no concept of end to end buffering One buffer used per FC frame, irregardless of frame size; small FC frame uses same buffer as large FC frame FC frames buffered and queued in intermediate switches Hop-by-hop traffic flow paced by return of Receiver Ready (R_RDY) frames; can only transmit up to the number of BB_Credits before traffic is throttled
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

15

Extending Optical SAN Extension
SAN Network Solutions for Increasing Distance
Shared FX-Port (Fixed) 16 Port 1/2 Gbps N/A FX-Port (Default) 16 Dedicated FX-Port / Extended E-Port E-Port Credits (Default) (Min-Max) (Min-Max) 255 2-255 N/A Speed (Gbps) 1 2 1 2 1 2 1 2-250 2564095 7514095 2 4 2-750 10 Max Distance (*) (km) 510 255 24 12 7000 3500 8190 4095 2047 800 No No Disable Ports for Max Credits? No

32 Port 1/2 Gbps 14/2 Port 1/2 Gbps 9216i Fabric Switch 12 Port 1/2/4 Gbps 24 Port 1/2/4 Gbps 48 Port 1/2/4 Gbps 4 Port 10 Gbps

12

12

12

12

N/A 2563500

No Disable 3 in a 4-Port Group, if More than 2400 per Port

N/A N/A 16 16 N/A

16 16 16 16 16

255 250 250 125 750

2-255

(*) Assuming
BRKSAN-2704 14571_04_2008_c1

Max Frame Size
Cisco Public

© 2008 Cisco Systems, Inc. All rights reserved.

31

Extending Optical SAN Extension
Optical Solutions for Increasing Distance
ONS 15454 SL-Series Card
No Spoofing
FC

Spoofing
B2B Negotiation R_Rdy Frame

Negotiates up 255 BB_Credit with FC switch Spoofs R_RDYs to FC switch (release 5.0)

B2B Negotiation

SONET/ SDH Network
B2B Negotiation

Frame Ack

Frame

Has 1200 BB_Credits between SL cards Extends distances to 2300 km @ 1G FC or 1150 km @ 2G FC

R_Rdy

Frame

FC

Configuration Note: When Using ONS 15454 R_RDY Spoofing with MDS 9000, Disable BB State Change Notification on Connected MDS Port: interface fc 1/1 no switchport fcbbscn
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

R_Rdy

Frame

32

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

16

Improving Optical Recovery
1) MDS Detects Link Failure 2) MDS Brings Down Array Port

Port Track for Resilient SAN Extension Solutions

Optical Network

3) Array Retries I/O on Alternate Path

Optical Network

Arrays recover from a link failure via I/O timeouts; however, this can take several seconds or longer MDS port-track addresses this by monitoring the WAN/MAN link and if it detects a failure, it will bring down the corresponding link connected to the array The array after detecting a link failure will redirect the I/O to another link without waiting for the I/O to timeout
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

33

Improving Optical Recovery
3) MDS Detects Link Failure 4) MDS Brings Down Array Port

Port Tracking and ONS FLC or Squelching
1) ONS Detects Link Failure

Optical Network 2) ONS Brings Down Client Transponder

5) Array Retries I/O on Alternate Path

Optical Network

The MDS port-tracking feature can be used with the ONS 15530 Forward Laser Control (FLC) or ONS 15454 squelching feature to further track failures in the network, improving the ability to detect failed paths Forward laser control, squelching and port-tracking offer end-to-end path failure detection
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

34

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

17

FCIP Data Compression
Cisco uses RFC standard compression algorithms implemented in both hardware and software MDS 9000 8-port IP Storage Services Module
Software-based compression for FCIP

MDS 9000 14/2-port Multiprotocol Services Module
Hardware and software-based compression and hardware-based encryption for FCIP

MDS 9000 18/4-port Multiservice Module
Hardware and software-based compression, hardware-based encryption, and intelligent fabric-based application services

Three compression algorithms—modes 1–3 plus auto mode Compressibility is data stream dependent
All nulls or ones → high compression (>30:1) Random data (e.g., encrypted) → low compression (~1:1)

“Typical” rate is around 2:1, but may vary considerably Application throughput is the most important factor
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

35

FCIP Data Compression and TCP Windowing
Compression has the effect of a variable bandwidth path TCP window applies to data stream before compression
If window size not increased, throughput will not increase Need to compensate with larger TCP max window size

MDS9000 incorporates moving average feedback to dynamically adjust TCP window according to compression rate Feedback mechanism is not available when using IP networkbased compression solutions— manual adjustment of TCP window size required
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

36

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

18

Compression Performance Envelopes
MSM (SAN-OS 3.x) Mode - Auto

MPS (SANOS 2.x) Mode 1

IPS (SANOS 2.x) Mode 1

IPS/MPS (SANOS 2.x) Mode 2

IPS = MDS 9000 IP Storage Services Module MPS = MDS 9000 14/2-Port Multiprotocol Services Module MSM = MDS 9000 18/ 4-Port Multiservice Module

IPS/MPS (SANOS 2.x) Mode 3

1

10

100

1000

Network Speed (Mbps)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

37

IPSec Encryption for FCIP
FCIP Link Encryption Provides: Data confidentiality—sender can encrypt packets before transmitting them across a network Data integrity—receiver can authenticate packets sent by the IPSec sender to ensure that the data has not been altered during transmission Data origin authentication—receiver can authenticate the source of the IPSec packets sent; this service is dependent upon the data integrity service Anti-replay protection—receiver can detect and reject replayed packets

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

38

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

19

Hardware-Based IPSec Encryption
Remote Tape Backup

Primary Site

Remote Replication

IP Network

Tape Backup and Remote Replication Secured with IPsec

Hardware-based GigE wire-rate performance with latency ~ 10 µs per packet Standards-based IPSec encryption—implements RFC 2402 to 2410, and 2412
IKE for protocol/algorithm negotiation and key generation Encryption: AES (128 or 256 bit key), DES (56 bit), 3DES (168 bit)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

39

Write Acceleration
Enables extended distance capabilities for remote replication technologies Better performance using FC/FCIP-WA—up to 2X the performance over given distance
Reduces effective I/O latency within SAN extension solutions

Built into services modules (IPS, MPS, MSM, SSM)—transparent to disk arrays Highly resilient solution—no data stored in MDS 9000 switch
FCIP Write Acceleration (WA)
FC or IP Network
WRITE DATA XFER_RDY
2.00 1.90 1.80 1.70

Write Acceleration (at Various Link Speeds and Write Sizes)
1 Gbps 622 Mbps 155 Mbps 45 Mbps

WA XFER_RDY

WA

Ratio

1.60 1.50 1.40 1.30 1.20 1.10 1.00 1 2 3 4

STATUS

32kB 45M 32kB 155M 32kB 622M 32kB 1G
8 9 10 11 12 13

Reduction in I/O Latency Equal to One Round Trip Time (RTT)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

5

6

7

RTT (ms) 40

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

20

Write Acceleration Design Considerations
High Availability and Load Balancing: Can be used for native FC replication (SSM) or FCIP replication (IPS, MPS and MSM) Port channels may be used for HA Equal cost FSPF load balancing for FCIP Write Acceleration not supported Works with:
EMC SRDF, Mirrorview, SANCOPY HDS TrueCopy HP CA-XP, CA-MVA IBM FlashCopy, FastT
Cisco Public

FC

Port channel

FC

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

41

Additional FC-WA Information
Fibre Channel write acceleration requires specific source and destination device configuration
FC-WA requires the initiator and target be connected to the SSM ports Configure scsi-flow feature on specific SSM ports Configure the specific initiator and target PWWNs and VSANs to use FC-WA Configure the number of 2K buffers to be used by FC-WA (Number of concurrent SCSI writes * size of SCSI writes in bytes)/FCP data frame size in bytes)

No restrictions on HA design; FSPF and port channeling may be used
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

21

FCIP Tape Acceleration
Server CMD Xfer_Rdy Data Status
Cmd-1 Cmd-2 Cmd-3 Write Filemarks Cmd

Tape

Write Filemarks Status

Tape acceleration is an enhancement of write acceleration Local MDS proxies as a tape library Remote MDS proxies as a backup server Status proxied Write Filemarks checkpoints process
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

43

FCIP Tape Acceleration Flow Control
Server CMD Xfer_Rdy Data Status
Cmd-1 Cmd-2 Cmd-3 Cmd-4 Xfer_Rdy
Sta -1

Flow Control

Tape

tus

Default for tape acceleration flow control buffering is “automatic”
Adjusts to change in WAN latencies Adjusts to speed of the tape library

Flow control buffer can be manually set
Is not normally required Fixes buffer size, network or device performance changes will not be adjusted
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

22

FCIP Tape Read Acceleration
Server
Tape Read “N” Cmd Tape Read “N” Cmd

Tape

Tape Read “N+1” Cmd Tape Read “N+1” Cmd Tape Read “N+2” Cmd Tape Read “N+2” Cmd

CMD Data Status

After first Read command received, Tape-end FCIP sends Read commands to prefetch data Host-end FCIP buffers the prefetched data For all Tape Read commands from the host
Host-end FCIP sends the cached data for the read and sends a SCSI good status tape-end FCIP consumes the read command

When a non-Read command is received, Tape-end FCIP rewinds the tape
A non-sequential read is treated as a nonread by the MDS
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

45

Tape Acceleration Throughput Results

Performance Remains Linear with Increasing RTT or Distance Even at 10 ms or 500 KM, Tape Performance Without Tape Acceleration Is less than 10 MB
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

46

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

23

FCIP TA Design Requirements
Tape acceleration is not supported over port channels or equal cost FSPF paths
Requirement due to state information kept in interface

HA design requires costing parallel links so one path preferred over another
HA through redundant links

Can use separate VSAN and FCIP tunnel to allow port channels for write acceleration

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

SAN Extension Fabric Stability
Connecting existing SAN fabrics or extending a SAN fabric creates SAN design challenges
Minimize the impact of change in fabric services across geographically distributed sites Limit fabric control traffic such as RSCNs and build/reconfigure fabric (BF/RCF) to local VSANs Connecting SAN fabrics with the same domain IDs

Inter-VSAN Routing (IVR) can be used to address these challenges
IVR enables a layer-3 hierarchical layer design to SAN fabrics to isolate domains IVR only sends selective RSCNs to edge switches, preventing disruption of fabric services IVR with NAT allows two existing SAN fabrics with the same domain ID to be connected through a third transit VSAN
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

48

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

24

SAN Extension with IVR
Site A
Local VSAN_5

FC

Replication VSAN_10

Any failure in transit VSAN_20 (network equipment, physical or logical failure) will not disrupt VSAN_10 or VSAN_30 fabric Works with any transport service (FC, SONET/SDH, DWDM/CWDM, FCIP)
Host to Local Array Fabric is VSAN_5 Site A Replication Fabric is VSAN_10 Site B Replication Fabric is VSAN_30 SAN Extension Fabric is VSAN_20

Inter-VSAN Connection between Completely Isolated Fabrics

Transit VSAN_20 (IVR)

Site B

FC

Replication VSAN_30

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

QoS for FCIP SAN Extension
Most FCIP Implementations Use Dedicated Links, However:
Acceptance of FCIP generating interest in converged IP network for FCIP SAN Extension No generally accepted DSCP values for FCIP traffic (unlike voice with DSCP EF)
QoS—define marking and classification Mark DSCP according an agreed value Separate consideration of FCIP data and control packet

Bandwidth reservation
FCIP has no support for reservation protocol Simulated using min/max B/W command (can be considered as a type of reservation)

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

25

FCIP QoS Mapping Proposal
Synchronous data replication: bursty, high bandwidth
Can be mapped into mission critical (AF31/DSCP 26)

Asynchronous data: bursty, low to medium b/w
Can be mapped into transactional data (AF21/DSCP 18) Can also mapped into bulk data (AF11/DSCP 10)

Backup data: 150 ~ 500 ms, constant (during backup), medium b/w
Can be mapped into bulk data (AF11 / DSCP 10)

Control packets
Both control and data traffic can be assigned the same class If needed can assign CS6 or DSCP 48
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

51

Classification and Marking Design
Modified for SAN Traffic
Application Routing / FCIP control Voice Video Conferencing Streaming Video FCIP SYNC Call Signaling FCIP ASYNC Network Management FCIP backup /FCIP Async Scavenger Best Effort
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved.

L3 Classification IPP 6 5 4 4 3 3 2 2 1 1 0
Cisco Public

L2 DSCP 48 46 34 32 26 24 18 16 10 8 0 CoS 6 5 4 4 3 3 2 2 1 1 0
52

PHB CS6 EF AF41 CS4 AF31* CS3* AF21 CS2 AF11 CS1 0

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

26

Advanced FCIP Designs

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Advanced FCIP TA Design

Scaling TA Beyond 1 FCIP Tunnel/GE Interface

FCIP1 VSAN 10 FCIP2 VSAN 20 FCIP3 VSAN 30 MDS1 MDS2 VSAN 100 Eight Tape Libraries (Three Tapes Each) Distributed over 4 VSANs FCIP4 VSAN 40 MDS2

FCIP tape acceleration is limited to one parallel path, but … • Move the target tapes devices into separate VSANs

• Make each FCIP interface only a member of one tape target VSAN and ensure trunking turned off • Use IVR to allow all backup servers to access all tape devices in all target VSANs
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

54

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

27

Advanced FCIP TA Design
Bidirectional Multiple Path TA

FCIP1

VSAN 10

MDS2 VSAN 100 IVR Site 1 Backup Servers in VSAN 100 to Local and Remote Tapes in VSAN 10 and VSAN 20

FCIP2

VSAN 20

MDS2 VSAN 200 IVR Site 2 Backup Servers in VSAN 200 to Local and Remote Tapes in VSAN 10 and VSAN 20

Both Local and Remote Backup Servers Need to Access Tape in Two Locations • Move the target tapes devices into separate VSANs • Make each FCIP interface only a member of one tape target VSAN and ensure trunking turned off • Use IVR to allow all backup servers to access all tape devices in all target VSANs
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

55

FCIP WA and TA—Multiple Tunnels
Port channel FCIP interfaces 11 & 21 for replication only
VSAN 100 Replication

GigE1/1 GigE1/2

GigE1/1.100 GigE1/1.200 GigE1/2.100 GigE1/2.200

int fcip 11 int fcip 12 int fcip 21 int fcip 22

VSAN 100 VSAN 300 VSAN 100 VSAN 400

int fcip 11 int fcip 12 int fcip 21 int fcip 22

GigE1/1.100 GigE1/1.200 GigE1/2.100 GigE1/2.200

GigE1/1 GigE1/2

VSAN 100 Replication VSAN 300 Tape Drives VSAN 400 Tape Drives

VSAN 200 Backup Serv

Do not port channel FCIP interfaces 12 & 22 for tape backup

Using GE sub-interfaces, multiple FCIP tunnels and port channeling to enable high b/w FCIP Use separate VSANs for data replication (100), backup servers (200) and tape backup (300, 400) Port-channel FCIP tunnels for replication traffic Distribute tape drives over multiple VSANs and FCIP tunnels to permit tape acceleration, do not port channel or use equal cost paths Use IVR to allow access from backup servers (VSAN 200) to tape drives (VSANs 300, 400)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

56

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

28

FCIP WA and TA—Multiple Tunnels
Port Channel FCIP Interfaces 11 and 21 for Replication Only
VSAN 100 Replication

GigE1/1 GigE1/2

GigE1/1.100 GigE1/1.200 GigE1/2.100 GigE1/2.200

int fcip 11 int fcip 12 int fcip 21 int fcip 22

VSAN 100 VSAN 300 VSAN 100 VSAN 400

int fcip 11 int fcip 12 int fcip 21 int fcip 22

GigE1/1.100 GigE1/1.200 GigE1/2.100 GigE1/2.200

GigE1/1 GigE1/2

VSAN 100 Replication VSAN 300 Tape Drives VSAN 400 Tape Drives

VSAN 200 Backup Serv

Do not Port Channel FCIP Interfaces 12 and 22 for Tape Backup
fcip profile 11 ip address 10.10.100.1 tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 fcip profile 12 ip address 10.10.200.1 tcp max-bandwidth-mbps 250 min-available-bw-mbps 200 fcip profile 21 ip address 10.11.100.1 tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 fcip profile 22 ip address 10.11.200.1 tcp max-bandwidth-mbps 250 min-available-bw-mbps 200 interface fcip 11 switchport trunk mode off channel-group 100 force use profile 11 peer-info ipaddr 10.10.100.2 write-accelerator interface fcip 12 switchport trunk mode off use profile 12 peer-info ipaddr 10.10.200.2 write-accelerator tape-accelerator interface fcip 21 switchport trunk mode off channel-group 100 force use profile 21 peer-info ipaddr 10.11.100.2 write-accelerator interface fcip 22 switchport trunk mode off use profile 22 peer-info ipaddr 10.11.200.2 write-accelerator tape-accelerator interface GigabitEthernet 1/1 no shut interface Gigabit Ethernet 1/1.100 ip address 10.10.100.1 255.255.255.0 interface Gigabit Ethernet 1/1.100 ip address 10.10.200.1 255.255.255.0 interface GigabitEthernet 1/2 no shut interface Gigabit Ethernet 1/2.100 ip address 10.11.100.1 255.255.255.0 interface Gigabit Ethernet 1/2.100 ip address 10.11.200.1 255.255.255.0 interface port-channel 100 no shut switchport trunk mode off switchport mode E channel mode active vsan database vsan 100 interface fcip 11 vsan 100 interface fcip 21 vsan 100 interface portchannel 100 vsan 300 interface fcip 12 vsan 400 interface fcip 22
Cisco Public

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

57

FCIP WA and TA—Multiple Tunnels
Port Channel FCIP Interfaces 11 and 21 for Replication Only
VSAN 100 Replication

GigE1/1 GigE1/2

GigE1/1.100 GigE1/1.200 GigE1/2.100 GigE1/2.200

int fcip 11 int fcip 12 int fcip 21 int fcip 22

VSAN 100 VSAN 300 VSAN 100 VSAN 400

int fcip 11 int fcip 12 int fcip 21 int fcip 22

GigE1/1.100 GigE1/1.200 GigE1/2.100 GigE1/2.200

GigE1/1 GigE1/2

VSAN 100 Replication VSAN 300 Tape Drives VSAN 400 Tape Drives

VSAN 200 Backup Serv

Do not Port Channel FCIP Interfaces 12 and 22 for Tape Backup
fcip profile 11 ip address 10.10.100.2 tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 fcip profile 12 ip address 10.10.200.2 tcp max-bandwidth-mbps 250 min-available-bw-mbps 200 fcip profile 21 ip address 10.11.100.2 tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 fcip profile 22 ip address 10.11.200.2 tcp max-bandwidth-mbps 250 min-available-bw-mbps 200 interface fcip 11 switchport trunk mode off channel-group 100 force use profile 11 peer-info ipaddr 10.10.100.1 write-accelerator interface fcip 12 switchport trunk mode off use profile 12 peer-info ipaddr 10.10.200.1 write-accelerator tape-accelerator interface fcip 21 switchport trunk mode off channel-group 100 force use profile 21 peer-info ipaddr 10.11.100.1 write-accelerator
BRKSAN-2704 14571_04_2008_c1

interface fcip 22 switchport trunk mode off use profile 22 peer-info ipaddr 10.11.200.1 write-accelerator tape-accelerator interface GigabitEthernet 1/1 no shut interface Gigabit Ethernet 1/1.100 ip address 10.10.100.2 255.255.255.0 interface Gigabit Ethernet 1/1.100 ip address 10.10.200.2 255.255.255.0 interface GigabitEthernet 1/2 no shut interface Gigabit Ethernet 1/2.100 ip address 10.11.100.2 255.255.255.0 interface Gigabit Ethernet 1/2.100 ip address 10.11.200.2 255.255.255.0 interface port-channel 100 no shut switchport trunk mode off switchport mode E channel mode active vsan database vsan 100 interface fcip 11 vsan 100 interface fcip 21 vsan 100 interface portchannel 100 vsan 300 interface fcip 12 vsan 400 interface fcip 22

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

29

FCIP—Configuration

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

MDS FCIP Configuration Guidelines
TCP parameter settings Understanding FC buffers and FCIP Monitoring for FC Frame expiry and why it is important MDS packet shaping methods so not to over run network Understanding MTU and the WAN network

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

60

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

30

MDS FCIP—TCP Parameters
Select the proper bandwidth settings Use ips measure-rtt or ping command to set initial RTT
fcip profile 1 ip address 200.200.200.1 tcp max-bandwidth-mbps 800 min-available-bandwidth-mbps 500 round-trip-time-us 80

MDS9216-TOP# ips measure-rtt 200.200.200.2 int gig 2/4 Round trip time is 82 micro seconds (0.08 milli seconds) or MDS9216-TOP# ping 200.200.200.1 --- 200.200.200.1 ping statistics --rtt min/avg/max/mdev = 0.068/0.079/0.091 ms

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

MDS FCIP—TCP Parameters
Always use Selective ACK (SACK) If slow link (<T3/E3):
Keep tcp cwm burstsize low (<20 kB) Tune FC receive buffers—BB_Credits and performance buffers (fcrxbbcredit)

If shared link (other traffic):
Determine available bandwidth (link b/w – other traffic b/w) and set maxbandwidth = available bandwidth and min-available-bandwidth = 80–95% max-bandwidth Consider QoS policies; carving out bandwidth

If shared link (other FCIP tunnel):
Determine available bandwidth (link b/w – other traffic b/w) and set max-bandwidth = 50% available bandwidth and min-available-bandwidth = 80–95% max-bandwidth

If a dedicated link:
Set min=95% max and max=path bandwidth (FCIP will send at this rate) If GigE all the way, set tcp cwm burstsize to 100 kB
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

62

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

31

FCIP Configuration Example
10
FC

fcip profile 10 ip address 10.1.4.2 tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 80 roundtrip-time-ms 30 interface fcip50 switchport mode E no shutdown switchport trunk allowed vsan 100 use-profile 10 peer-info ipaddr 10.4.8.2 interface GigabitEthernet2/5 ip address 10.1.4.2 255.255.255.0 switchport mtu 2300 no shutdown

100 Shared 155-Mbps WAN Link (100 Mb for Storage) 30-ms RTT

10.1.4.2

RTT Will Autconfigure and Adapt to Network Changes During Idle Periods

Jumbo Frame MTU— 2300 Bytes Will Handle Largest FC Frame

Three steps for FCIP configuration—GigE interface, FCIP profile, and FCIP interface
10.4.8.2

FC

100

Min-bandwidth set to minimum bandwidth available (through QoS or other means); sender will start at this rate Peer FCIP interface configured similarly

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

Fibre Channel Frame Buffering
Traffic Flow
FC FC

BB_Credit Flow Control

BB_Credit Flow Control

BB_Credit Flow Control

FC Receive Buffers

FC Receive Buffers

FC frames buffered and queued in intermediate switches Buffer depth controlled by Rx BB_Credit parms
switchport fcrxbbcredit <x> switchport fcrxbbcredit performance-buffers <y> Performance buffers added by default unless configured otherwise

Hop-by-hop traffic flow paced by return of R_RDY frames Frames cannot sit in any switch for >500 ms (or they will expire)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

64

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

32

Frame Buffering: FCIP and FC
Traffic Flow
FC

BB_Credit Flow Control TCP Send Buffers FC Receive Buffers

TCP Windowing Flow Control

FC

BB_Credit Flow Control TCP Send Buffers

GigE

Slower WAN Link

GigE

Increase TCP Send Buffer if : Slow WAN link is preventing FC traffic destine to other devices not across FCIP link to have issues

FCIP Receive Buffers Backlog Here If Queue Can’t Drain Due to: Slow WAN link and long RTT Packet loss and retransmissions Many sources (only one shown) Buffer too big

FCIP presents a lower bandwidth pipe (If WAN link)
Drain rate (send rate) depends upon bandwidth and congestion

Slow ramp up of traditional TCP can cause FC frame expiry in some conditions
Mixture of slow link (e.g., <DS3/E3; retransmissions, many sources, big buffers)
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

65

Frame Expiration
500-ms timer is fixed (not configurable) Behavior possible whenever FC Rx buffers cannot drain at a sufficient rate (at least 2x #Rx buffers/second)
Caused by one or more of: too many sources (speed mismatch) slow downstream (slow receiver) FC Rx buffers too deep

Possible situations are:
1. Long optical links with low BB_Credits (i.e., can’t drain quickly)— misconfiguration 2. FCIP over long, slow links (e.g., T1 or T3 with 50 + ms RTT) with occasional packet loss/retransmission
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

66

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

33

Monitor FCIP Frame Timeouts
MDS9216-TOP# show ips stats dma int gigabitethernet 2/4 Dma-bridge ASIC Statistics for port GigabitEthernet2/4 Hardware Egress Counters 1030008889 Good, 0 bad protocol, 0 bad header cksum, 0 bad FC CRC Hardware Ingress Counters 1843269599 Good, 0 protocol error, 0 header checksum error 0 FC CRC error, 0 iSCSI CRC error, 0 parity error Software Egress Counters 1030008581 good frames, 0 bad header cksum, 0 bad FIFO SOP 0 parity error, 0 FC CRC error, 0 timestamp expired error 0 unregistered port index, 0 unknown internal type 0 RDL ok, 0 RDL drop (too big), 0 RDL ttl_1 1278210315 idle poll count, 116279 loopback 0 FCC PQ, 0 FCC EQ, 0 FCC generated Flow Control: 0 [0], 0 [1], 0 [2], 0 [3] Software Ingress Counters 1843269472 Good frames, 0 header cksum error, 0 FC CRC error 0 iSCSI CRC error, 0 descriptor SOP error, 0 parity error 141281 frames soft queued, 0 current Q, 732 max Q, 0 low memory 0 out of memory drop, 0 queue full drop 0 RDL ok, 0 RDL drop (too big) Flow Control: 0 [0], 141281 [1], 0 [2], 0 [3]
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Monitor for 500 ms Timeout Issues

67

TCP Window and Buffer with FCIP
You cannot configure the TCP window size directly. This value is automatically calculated from the product of the maximum bandwidth x RTT. In SAN-OS 2.0 and later, the RTT will dynamically adjust up to eight times the configured value in the FCIP profile. The TCP sender dynamically changes the maximum window size accordingly.

MDS9216-TOP# sh interface fcip 2 counters
--snip--

TCP Parameters Path MTU 1500 bytes Current retransmission timeout is 200 ms Round trip time: Smoothed 2 ms, Variance: 4 Advertized window: Current: 97 KB, Maximum: 97 KB, Scale: 5 Peer receive window: Current: 98 KB, Maximum: 98 KB, Scale: 5 Congestion window: Current: 52 KB, Slow start threshold: 103 KB Current Send Buffer Size: 97 KB, Requested Send Buffer Size: 0 KB CWM Burst Size: 50 KB
--snip--

IP compression statistics 4059579130004 rxbytes, 273626035160 rxbytes compressed 2287961989360 txbytes 633365852164 txbytes compressed, 0 txbytes non-compressed 3.61 tx compression ratio
BRKSAN-2704 14571_04_2008_c1

TCP Send Buffer Can Be Increased to Allow Faster Removal of Frame off FC Interface RX Buffer

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

68

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

34

TCP Send Buffer Interworkings
FC

Interface on Storage Array
FCIP Traffic

MDS9000
FC Interface FCIP Interface Virtual Output Queues

Traffic Transmitting over WAN at Capable Network Rate

Local Server Traffic

28 Buffers Total over All VOQ Default

TCP sendbuffer

IP Network
FC

HBA Attached Server

TCP Send Buffer Would Only Require Adjustment if Other Local Exchanges Were Occurring Off the Same FC Port That Is Being Used by FCIP Application; the Minimum Size is 14k and Is Dynamically Calculated from Bandwidth x rtt Product
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

69

FCIP— Capacity Planning

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

70

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

35

FCIP—Capacity Planning SAN Extension Tuner (SET)
SAN Extension Tuner—Lightweight Tool Built into the IPS and MPS to Assist in FCIP Tuning by Generating Various SCSI Traffic Flows
N-port: 10:00:00:00:00:00:00:01 Gig3/3 N-port: 11:00:00:00:00:00:00:03 Gig2/2

IP Network
FCIP Tunnel

Gig3/1

Gig2/1

Creates a virtual N-port on an IPS or MPS port that can act as both initiator and target to generate traffic Measures throughput and response time per I/O over the FCIP tunnels Model effect of storage data in a shared network
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

71

Using SAN Extension Tuner
San Extension-Tuner allows you to configure a unused iSCSI interface as a FC initiator and target Used to create virtual FC devices in the MDS to generate FC traffic FC traffic is sent across FCIP link (ISL) for WAN load test, help tune TCP parameters and exercise features; to achieve desired throughput across FCIP a number of parameters must be tuned to keep WAN pipe full
TCP parameters (Window size, SACK…) Outstanding SCSI I/Os allowed by application Transfer size used by application

Requires 2.0 SAN/OS or greater
Note: MSM 18+4/9222i require SAN-OS 3.3(1) or higher
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

72

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

36

SAN Extension Tuner
Only read-and-write SCSI commands are used, entire SCSI suite in not supported Multiply virtual N-ports can be created to overload FCIP link to stress WAN provisioning, compression results, troubleshoot issues without using FC real devices Use data provided with tool or use your own Simple Setup Wizard Wizard creates zone for the virtual N-ports

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

73

SET Use from Fabric Manager
Click on FCIP Link

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

74

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

37

SET—Monitor Results, Make Adjustments
Monitor ISL Link Utilization Tool

Auto Created Zone

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

75

FCIP—Troubleshooting

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

38

Troubleshooting Section Agenda
Tools available is SAN-OS Practical use of SPAN, PAA and Cisco analyzers in troubleshooting

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

77

FCIP CLI Commands
Useful Commands
show interface fcip nnn show interface gigabitethernet n/n show ips stats all show ips stats buffer show ips stats dma-bridge show ips stats hw-comp show ips stats icmp show ips stats ip show ips stats mac show ips stats tcp Show fcip interface statistics Show Gigabit Ethernet interface statistics Show all ips stats Show ips buffer stats Show dma-bridge stats Show hw compression stats Show icmp stats Show ip stats Show ethernet mac stats Show tcp stats

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

39

Command Line Debugging
Available debugs depend on features enabled in SAN-OS Many different options to select when turning on debugs
Interface and protocol level debugging available

Where is it output going?
Logfile—data file in switch memory Capture to direct to screen via console, telnet or ssh

Requires admin privileges to run debugs Debugs can only be run from cli, no debug interface in fabric manager or device manager
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

79

FCIP Debugs
You Must Attach to the Module You Are Debugging
module-2# debug ips fcip ? all Configure FCIP debugging ctlpkt Configure FCIP Control Pkt debugging ctlpkt-detail Configure verbose FCIP Control Pkt debugging datapkt Configure FCIP Data Pkt debugging datapkt-detail Configure verbose FCIP Data Pkt debugging err Configure FCIP Error debugging fsm Configure FCIP FSM debugging tape-acc Configure FCIP Tape accelerator tape-acc-det Configure FCIP Tape accelerator debug detail tape-acc-err Configure FCIP Tape accelerator error write-acc Configure FCIP Write accelerator write-acc-err Configure debug FCIP Write accelerator errors write-acc-event Configure debug FCIP Write accelerator events write-acc-pkt Configure debug FCIP Write accelerator packet)

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

80

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

40

Gathering Protocol Traces for Analysis
Using built-in FCanalyzer (CLI) Using Cisco Protocol and Traffic Analyzer on PC (local and remote) Using the MDS Port Analyzer Adapter (PAA) Using SPAN Using an external FCanalyzer All non-disruptive to switch operations and traffic on the SAN

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

81

MDS FCanalyzer
(SAN/OS Imbedded) Output is displayed to the console in readable sniffer like format Is only used to monitor Fibre Channel traffic to and from supervisor on the MDS9000
Traffic-like fabric logins, FSPF routing, switch-to -switch control traffic

Output can go direct to your console screen or to a workstation running a color Ethereal program
Note: SPAN is used not only for FC port-to-FC port monitoring, but also used to SPAN iSCSI and FCIP ports

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

82

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

41

FCanalyzer Options
Local or remote—where to send the trace; can be to local devices or remote PAA attached to different MDS switch Brief or detailed—header information vs. full output of frame including hex; detail is default Limit-captured-frames—number of frames to capture; default is fcanalyzer will trace 100 frames; specifying zero is unlimited frame capture Limit-capture-size—allows to capture N number of bytes of frame; useful for not capturing frame data when it is not relevant to troubleshooting
BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

83

Use of SPAN Feature
Used for FC port to FC port analyzing Same type of tool as used on Cisco Catalyst® products (Catalyst uses port monitor) Can be left configured on switch Ingress and egress ports are sent to an FC-port setup as a SPAN destination (SD-port type) No limits to where the ports are located on the MDS switch fabric Used to output to third-party test equipment or to Cisco Port Analyzer Adapter
BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

84

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

42

Using the FCanalyzer with FCIP
MDS9216-TOP# conf t Enter configuration commands, one per line. End with CNTL/Z. MDS9216-TOP(config)# fcanalyzer local brief display-filter mdshdr.vsan==300 Warning: Couldn't obtain netmask info (eth2: no IPv4 address assigned). Capturing on eth2 8.986146 8.986233 8.987140 8.987539 8.988273 8.988790 8.989438 8.991501 8.992965 8.993037 9.001146 9.001603 9.001719 ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd ff.ff.fd -> ff.ff.fd 0x46a6 0xffff SW_ILS ELP 0x46a6 0x469a FC Link Ctl, ACK1 0x469b 0xffff SW_ILS ELP 0x469b 0x46a7 FC Link Ctl, ACK1 0x469b 0x46a7 SW_ILS SW_RJT (ELP) 0x46a6 0x469a SW_ILS SW_ACC (ELP) 0x469b 0x46a7 FC Link Ctl, ACK1 0x46a6 0x469a FC Link Ctl, ACK1 0x46a8 0xffff SW_ILS EFP 0x46a8 0x469c FC Link Ctl, ACK1 0x469d 0xffff SW_ILS EFP 0x469d 0x46a9 FC Link Ctl, ACK1 0x46a8 0x469c SW_ILS SW_ACC (EFP)

Filter on VSAN 300 No shut of FCIP interface Complete view of standard ISL initialization and switch fabric merge You do not capture FCIP protocol; FCIP protocol is viewed with debugs and/or SPAN (monitor) of WAN network
85

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

SPAN of FCIP Interface to PAA and Cisco Protocol Analyzer

PC w/Cisco Protocol Analyzer (Ethereal)

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

86

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

43

PAA Capture Modes
No Truncate Mode (NTM) Fibre Channel Frames Are Encapsulated into Ethernet Frames Without any Modification to the Payload). Ethernet Devices must Support Jumbo Frames for this Mode to Work. Ethernet Truncate Mode (ETM) The PAA Truncates a Fibre Channel Frame to a Maximum Payload of 1496 Bytes. The Fibre Channel Payload Transmitted Is 1472 Bytes. Shallow Truncate Mode (STM) In STM, the PAA Truncates the Fibre Channel Frame to 256 Bytes. Deep Truncate Mode (DTM) In DTM, the PAA Truncates the Fibre Channel Frame to 64 Bytes. By Default, the PAA Is Configured in DTM Mode.
Ethernet Header 14 Bytes (EDA + SDA + Type) SOF-T VSAN 2 Bytes FC Frame 0–2164 Bytes
FC CRC 4 Bytes

EOF-T VSAN 4 Bytes

Enet CRC 4 Bytes

Ethernet Header 14 bytes (EDA + SDA + Type)

SOF-T VSAN 2 Bytes

FC Frame 0–1490 Bytes

FC CRC 4 Bytes

EOF-T VSAN 4 Bytes

Enet CRC 4 Bytes

Ethernet Header 14 Bytes (EDA + SDA + Type)

SOF-T VSAN 2 Bytes

FC Frame 0–256 Bytes

FC CRC 4 Bytes

EOF-T VSAN 4 Bytes

Enet CRC 4 Bytes

SOF-T VSAN (EDA + SDA + Type) 2 Bytes

Ethernet Header 14 Bytes

FC Frame 0–64 Bytes

FC CRC 4 Bytes

EOF-T VSAN 4 Bytes

Enet CRC 4 Bytes

Ethernet Frame Ethernet Payload FC Frame

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

87

Display Filters Can Be Applied Before Or After Capture

Right Click on Field to Filter

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

88

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

44

GiGE Monitoring of FCIP
FCIP GigE Interface Must Be Up and Healthy show ips stats ip interface gigabitethernet 2/2 detail show ips stats tcp interface gigabitethernet 2/2 detail show ips stats mac interface gigabitethernet 2/2

Device Manager Monitor Views Can Be Actively Kept Open and Graphed in Real Time
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

89

Q and A

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

90

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

45

Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store
BRKSAN-2704 14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

91

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

92

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

46

BRKSAN-2704 14571_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

93

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

47