You are on page 1of 40

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

1

Fibre Channel Storage Area Network Design

BRKSAN-2701

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

1

Agenda
Brief SAN Technology Overview
Fibre Channel Protocol Virtual SAN (VSAN), Zoning Port Channels, IOD

SAN Design Principles and Considerations
Design Factors Design Types Design Flexibility SAN Security Design Considerations Interoperability Design Considerations

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

SAN Technology Overview

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

4

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

2

SAN Technology Overview—Agenda
Fibre Channel Protocol
FC Communications Port types, ISL Addressing, Framing, Timers

Virtual SAN (VSAN), Zoning Port Channels, IOD Virtual Output Queuing (VOQ)

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

Fibre Channel Communications
Point-to-point oriented
Facilitated through device login

N_Port-to-N_Port connection
Logical node connection point

Flow controlled
Buffer-to-buffer credits and end-to-end basis

Acknowledged
For certain classes of traffic, none for others

Multiple connections allowed per device
Node
Transmitter N_Port Receiver Link
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Node
Transmitter N_Port Receiver

6

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

3

FCP—Protocol Stack
SCSI
SCSI Applications (File Systems, Databases) SCSI Block Commands SCSI Commands, Data, and Status Parallel SCSI Interfaces

FC
SCSI Applications (File Systems, Databases) SCSI Stream Commands SCSI Commands, Data, and Status Fibre Channel Protocol (FCP)

SCSI Bus

Fibre Channel

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

7

Fibre Channel Port Types
Fibre Channel Switch
Input Port Fabric X Output Port

Node

NL_Port

FL_Port

E_Port

E_Port

Fabric Switch

Node

NL_Port

G_Port

F_Port

N_Port

Node

Node

NL_Port

G_Port

F_Port

N_Port

Node

G_Port

F_Port

N_Port

Node

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

8

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

4

Inter-Switch Link (ISL)
EISL

The interconnection between switches is called the ISL
E_Port to E_Port (‘Expansion port)

Supports all classes of service
Class 1, 2, 3, and a special Class F (switch-to-switch)

FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput Cisco’s implementation is to dedicate an FC_ID pair and/or a given exchange to an ISL bundle member to guarantee in-order delivery for exchange/sequence frames Cisco Extended ISL (EISL, TE port)
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

9

Buffer to Buffer Credit Flow Control
BB_Credits and Distance
~ 2 km per Frame 1 Gbps FC

~ 1 km per Frame 2 Gbps FC

~ ½ km per Frame 4 Gbps FC

16 Km

BB_Credits are used to ensure enough FC frames in flight A full (2112 byte) FC frame is approx 2 km long at 1 Gbps, 1 km long at 2 Gbps and ½ km long at 4 Gbps As distance increases, the number of available BB_Credits need to increase as well Insufficient BB_Credits will throttle performance—no data will be transmitted until R_RDY is returned
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

5

FSPF Protocol
FSPF stands for Fabric Shortest Path First Path selection protocol used in Fibre Channel Based on link state protocol Fibre Channel standard defined in FC-SW2 Conceptually based on Open Shortest Path First (OSPF) Internet routing protocol

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

FC_ID Address Model
FC_ID address models help speed up routing Switches assign FC_ID addresses to N_Ports Some addresses are reserved for fabric services Private loop devices only understand 8-bit address (0x0000xx) FL_Port can provide proxy service for public address translation Maximum switch domains = 239 (based on standard)
8 Bits Switch Topology Model Private Loop Device Address Model Public Loop Device Address Model
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved.

8 Bits Area

8 Bits Device Arbitrated Loop Physical Address (AL_PA) Arbitrated Loop Physical Address (AL_PA)
12

Switch Domain 00 Switch Domain
Cisco Public

00

Area

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

6

Fibre Channel FC-2 Hierarchy
Multiple exchanges are initiated between initiators (hosts) and targets (disks) Each exchange consists of one or more bidirectional sequences Each sequence consists of one or more frames For the SCSI3 ULP, each exchange maps to a SCSI command

OX_ID and RX_ID

Exchange

SEQ_ID

Sequence

Sequence

Sequence

SEQ_CNT Frame Fields

Frame

Frame
ULP Information Unit

Frame

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

Fibre Channel Timers
Receiver-Transmitter Time-Out (R_T_TOV)
Used to time events at the link level Loss of synchronization Times responses for link reset protocol Default value: 100 ms

Error Detect Time-Out (E_D_TOV)
Timers for events and responses at the sequence level Missing ACK or R_RDY when buffer credit has reached zero N_Port logout Timer value is set at fabric login to accommodate the network environment based on delivery time of frames Default value: 2 sec

Resource Allocation Time-Out (R_A_TOV)
Time-out value for how long to hold resources associated with a failed operation Needed to free shared resources for reuse Value to determine how long a port needs to keep responding to a link service request before an error is detected 2xE_D_TOV in point-to-point and fabricwide from 1 to 231 – 1 Default value: 10 s
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

7

VSANs, Zoning, IVR Zones
Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single physical switch VSAN—Design Foundation

Zoning Fabric routing (Inter-VSAN Routing—IVR)
Ability to provide selected connectivity between virtual fabrics without merging them

Virtual Fabric Trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL or common group of ISLs

IVR zones
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

15

VSANs, Zoning, IVR Zones
Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single physical switch VSAN—Design Foundation

Zoning Fabric routing (Inter-VSAN Routing–IVR)
Ability to provide selected connectivity between virtual fabrics without merging them

Virtual Fabric Trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL or common group of ISLs

IVR zones
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

8

Virtual Storage Area Networks (VSAN)
VSANs address the limitations of common SAN deployments
VSANs are virtual fabrics SAN islands are virtualized onto a common SAN infrastructure Allocate ports within a physical fabric to create virtual fabrics Fabric disruption is limited to VSAN Traffic statistics are gathered per VSAN
Independent Physical SAN Islands Are Virtualized onto a Common SAN Infrastructure

Features include:
Dynamic provisioning and resizing Improved port utilization Shared ISL bandwidth
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

17

Virtual SANs—VSANs
Production SAN Tape SAN Test SAN

FC

FC

FC FC

FC

SAN A DomainID=1 DomainID=7

SAN B DomainID=2 DomainID=8

SAN C DomainID=3

SAN D DomainID=4

SAN E DomainID=5

SAN F Domain ID=6

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

9

Zoning
Devices within a zone can access each other
Zoning establishes access control Limiting access prevents unauthorized access
FC FC
FC

Zoneset Zone A Zone B
FC FC FC

FC

Soft zoning
Implemented in switch software and enforced by name server Name server responds to discovery queries with only devices found in requestor’s zone or zones “Soft zoning” used to be synonymous with “WWN zoning”
FC FC
FC

FC
FC

FC
FC

FC

Zone C

Hard zoning
Enforced by ACLs in port ASIC Applied to all data path traffic “Hard zoning” used to be synonymous with “port zoning”
FC
FC

Default Zone
FC

FC FC
FC

FC FC

FC
FC FC

FC
FC

WWN zoning in hardware
Both at source and destination ports Enhanced zoning
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Devices Not Assigned to a Zone Belong to the Default Zone
19

Zoning and VSANs
Hierarchical relationship
First assign physical ports to VSANs Then configure independent zones per VSAN VSANs only change when ports needed per virtual fabric Zones can change frequently (e.g., backup)
VSAN 2

Zones and VSANs Are Complementary Physical Topology
Active Zoneset A ZoneA
Host1 Disk3 Disk1 Disk2

ZoneC Default Zone ZoneB
Disk4 Host2

Zones provide added security and allow sharing of device ports Zone membership is configured by:
Port World Wide Name (pWWN)—device Fabric World Wide Name (fWWN)—fabric Fibre Channel Identifier (FCID) Fibre Channel Alias (FC_Alias) IP address Domain ID/port number Interface
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

VSAN 7

Active Zoneset D
Host4

Default Zone
Host3

ZoneD ZoneA
Disk6 Disk5

One Active Zoneset per VSAN

20

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

10

VSANs, Zoning, IVR Zones
Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single physical switch VSAN—Design foundation

Zoning Fabric routing (Inter-VSAN Routing—IVR)
Ability to provide selected connectivity between virtual fabrics without merging them

Virtual fabric trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL or common group of ISLs

IVR zones
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

21

VSANs—Routed Connectivity
Common Physical Fabric

Sharing a Common Resource such as Tape
Common Physical Fabric
Sales SAN HR SAN
MS

Sales SAN

MS MS

Marketing SAN

Marketing SAN

MS

HR SAN

Tape SAN

Production and DR Interconnect Without Merging Fabrics

SAN Extension Services

IP or FC

Tape Media Server
MS

Potentially Separate Administrators per SAN

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

22

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

11

Inter-VSAN Routing—IVR
Enables devices in different VSANs to communicate Allows selective routing between specific members of two or more VSANs
Traffic flow between selective devices

Resource sharing, i.e., tape libraries and disks
VSAN 10
FC FC
FC

IVR Zone
Media Server
FC

Tape Library
FC

VSAN 20
FC FC
FC

FC
FC

FC

FC
FC

FC
FC FC

FC
FC

FC
FC
FC

FC
FC

FC

FC
FC

FC FC FC
FC FC

FC

FC

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

IVR Zones
IVR zone
A container or access control, containing two or more devices in different VSANs
Physical Topology
VSAN 2
Disk2 ZoneA Host1 Disk3 Disk1 ZoneC

Standard zones are still used to provide intraVSAN access IVR Zoneset
A collection of IVR zones that must be activated to be operational

Disk4 ZoneB

Host2

VSAN 3
Host4 ZoneD ZoneA Host3 Disk6 Disk5

Inter-VSAN Zone
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

24

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

12

Port Channels
Port Aggregation Feature Used to Create a Single Logical ISL from 1–16 Physical ISLs
Increases bandwidth and availability Very granular load balancing— per exchange/src/dst or per src/dst (policy on a per VSAN basis) Interfaces can both be added and removed in a nondisruptive manner in production environments Preserved FC guarantee of in-order delivery (IOD)
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

4-Link Port Channel EISL

25

Port Channel Protocol (PCP)
Exchange-based, in-order load balancing
Mode 1: based on src/dst FC_ID/OX_ID/RX_ID Mode 2: based on src/dst FC_ID

Consistently detect misconfiguration Transition misconfigured ports to isolated state so as to be able to correct the misconfig Synchronize bring up of ports in a channel across peer switches Provide the ability for the system to automatically create Port Channels among compatible ports
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Up to 160 Gbps Port Channel with HA

26

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

13

VSANs, Trunking, Port Channels
Hierarchical relationship
Port Channels provide link aggregation to yield virtual ISL (E_Port) Single-link ISL or Port Channel ISL can be configured to become EISL— (TE_Port) VSANs can be selective grafted or pruned from EISL trunks
8 Gbps Port Channel Trunking E_Port (TE_Port)
p 20 cku AN Ba VS 10 AN VS

VSAN 10 20

Metric 100 50

VSAN Metric 10 50

All member links of a port channel must have same configuration prior to creating channel (e.g., TE_Port or E_Port, VSANs-enabled, etc.) Port Channel technology provides high availability and fast recovery for VSAN trunk (EISL) Multiple Port Channels yield multiple paths for custom traffic engineering
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

10 AN ly VS On

E_Port

Trunking E_Port (TE_Port)

E_Port

4-Link (8 Gbps) Port Channel Configured as EISL

27

IOD: In-Order Delivery— Frames Are Delivered In-Order
Frames of a flow (SID,DID) are received in the order they were sent
Some applications (FICON and HP CA EVA) can not tolerate out of order frames
Frames Left in Output Queues/Buffers at Peer Switch

MDS HW always keeps frames within same exchange on same path except during
FSPF route changes Port Channel (PC) membership changes

Frames Left in Output Queues/Buffers at Local Switch

5 6

2 1

4 7

3 8

With IOD the frame waits for 500ms (switch latency value where frames are dropped) to guarantee that all frames have been drained from all the VOQs in the peer switch Once the peer has performed this operation, it responds with an acknowledgement and traffic forwarding can begin immediately
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Recovery Path Could Carry Frames from Same Exchange Out-of-Order

28

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

14

Virtual Output Queuing (VOQ)
B C A C C C C C A B
Input Port Output Port
FC

A B

Switch with no VOQ
HOL blocking

Input Port

Output Port
FC

Input port

Output Port

C

--------------------------------------------------------------------------------------------------------------------------------------------------------------

C C C C B C A C C C C C A B B A
ARB
Input Port Output Port
FC

Input Port

Output Port

FC

A B C

Switch with VOQ support
No HOL blocking VOQ alleviates HOL
Input Port Output Port

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

SAN Design Principles and Considerations

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

15

Agenda
Brief SAN Technology Overview
Fibre Channel Protocol Zoning, Virtual SAN (VSAN) Port Channels, IOD

SAN Design Principles and Considerations
Design Factors Design Types Design Optimization SAN Security Design Considerations Interoperability Design Considerations

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

SAN Design—Principles and Considerations
Determine components to be used and how they will fit into your overall strategy Creation of technical infrastructure and how the pieces will fit together Determine how existing infrastructure and new one will be integrated Creation of the processes and procedures that will guide personnel in how the infrastructure is to be used
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Data Center Servers

Storage Network

Storage Disk and Tape

32

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

16

Design Factors

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

33

Early SAN Designs
1. First SANs hardly qualified as networks
SAN islands of two to four switches Fixed 8–16 port switches limited SAN growth

SAN Islands
Low Traffic Across ISLs
4 Host
Host

Host 1

Few Switches

2. No fabric segmentation such as VSANs
Host

3. Limited enhancements to FSPF
No Port Channeling No equal cost load balancing

4. Traffic management was not needed
No QoS because bandwidth was over provisioned

Switches Managed Separately
5

3

Single Routes

5. Management tools focus on element management
Not network management
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Faults Impact all Devices
2

Host

Host

Host

Host

Host

34

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

17

SAN Major Design Factors
1. Port density
How many now, how many later? topology
High Performance Crossbar 2 QoS, Congestion 4 Control, Reduce FSPF Routes
8 8 8 8 8 8 8 8 8 8 8 8

2. Network performance
What is acceptable? Unavoidable?

Large Port Count Directors

1

3. Traffic management
Preferential routing or resource allocation

4. Fault isolation
Consolidation while maintaining isolation
Host Host Host Host Host Host

3 Failure of One Device Has No Impact on Others

5. Management
Secure, simplified management
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

35

1. Scalability—Port Density, Topology Requirements
Number of ports for end devices How many ports are needed now? What is the expected life of the SAN? How many will be needed in the future? Hierarchical SAN design
High Performance Crossbar 2 QoS, Congestion 4 Control, Reduce FSPF Routes
8 8 8 8 8 8 8 8 8 8 8 8

Large Port Count Directors

1

Best Practice
Design to cater for future requirements Doesn’t imply “build it all now,” but means “cater for it” and avoids costly retrofits tomorrow
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Host

Host

Host

Host

Host

Host

3 Failure of One Device Has No Impact on Others

36

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

18

2. Network Performance— Oversubscription Design Considerations
All SAN Designs Have Some Degree of Oversubscription
Without oversubscription, SANs would be too costly Oversubscription is introduced at multiple points Switches are rarely the bottleneck in SAN implementations Device capabilities (peak and sustained) must be considered along with network oversubscription Must consider oversubscription during a network failure event Remember, all traffic flows towards targets—main bottlenecks
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Disk Oversubscription Disk Do Not Sustain Wire-Rate I/O with ‘Realistic’ I/O Mixtures Most Major Vendors Promote 12:1 host:disk Fan-Out 70 MBps Max/Port (Common) Typical Oversubscription in Two-Tier Design Can Approach 8:1, Sometimes Even Higher 7:1 O.S. (Common)

Tape Oversubscription Low Sustained I/O Rates LTO-2 Native Transfer Rate ~ 60 MBps 60 MBps Max/Port (Common)

Port Channels Help Reduce Oversubscription While Maintaining HA Requirements

40 MBps Max/HBA (Common) Host

Host Oversubscription Most Hosts Suffer from PCI Bus Limitations, OS, and Application Limitations Thereby Limiting Maximum I/O and Bandwidth Rate

37

3. Traffic Management
Do different apps/servers have different performance requirements?
Should bandwidth be reserved for specific applications? Is preferential treatment/ QoS necessary?
High Performance Crossbar 2 QoS, Congestion Control, Reduce FSPF Routes
8

Large Port Count Directors 4 1
8 8 8 8 8 8 8 8 8 8 8

Given two alternate paths for traffic between data centers, should traffic use one path in preference to the other?
Preferential routes
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Host

Host

Host

Host

Host

Host

3 Failure of One Device Has No Impact on Others

38

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

19

4. Fault Isolation
Consolidation of Storage into a Single Fabric = Increased Storage Utilization + Reduced Administration Overhead Major Drawback Is That Faults Are No Longer Isolated
Technologies such as VSANs enable consolidation and scalability while maintaining security and stability VSANs constrain fault impacts
Faults in one virtual fabric (VSAN) are contained and do not impact other virtual fabrics
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Physical SAN Islands Are Virtualized onto Common SAN Infrastructure

Fabric #3 Fabric #1

Fabric #2

39

5. Management
Consolidation and Large Scale SAN Becomes More Difficult to Manage. How Can It Be More Secure? How Can the SAN Traffic Be Monitored as Performance Requirements Increase?
FM Client FMS FC SNMP SNMP
FC

FM Clients

Tools and Features
RBAC on per VSAN basis

SNMP

FC-SP for switch-to-switch or device-to-switch security Fabric Manager Server Online traffic monitoring through Device Manager
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

40

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

20

Design Types

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Core-Edge
Traditional SAN design for growing SANs High density directors in core and fabric switches, directors or blade switches on edge Predictable performance Scalable growth up to core and ISL capacity
A B A B A B A B

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

21

Large Core-Edge Example
Large Core/Edge
(2240 Usable Ports per Fabric)
“A” Fabric Shown, Repeat for “B” Fabric

Traditional Core-Edge Design Is Ideal for Centralized Services and Consistent Host-Disk Performance Regardless of Location; Less Future Scalability
Requires third tier to scale Port bandwidth reservations guarantee performance for ISLs (edge side)
Ports Deployed: Used Ports: Storage Ports (2 G Dedicated): Host Ports (4 G Shared): ISL Oversubscription (Ports): Disk Oversubscription (Ports): 2520 2512 256 1984 15.5 : 1 15.5 : 1

128 Storage Ports at 2 G 64 ISL to Edge at 4 G

32 ISL to Core at 4 G 496 Host Ports at 4 G

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

Scaling Traditional Core-Edge with Three Tiers
Could Scale to ~ 40k Ports*
12 ISL to Core at 4 G

124 Storage Ports at 2 G 6 ISL to Backbone at 4 G 64 ISL to Edge at 4G

32 ISL to Core at 4 G 496 Host Ports at 4 G 15.5 : 1 Host to Core

* No software testing has been done to this level of scalability
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

22

Top of Rack Example— 9134 and 10 G ISLs
Top of Rack
(1088 Usable Ports per Fabric)

Top of Rack Design Utilizing 10 Gb ISLs; High Bandwidth ISLs Provides Ample Performance and Reduces Cabling

A 96 Storage Ports at 2 G 28 ISL to Edge at 10 G
2 ISL to Core at 10 G 32 Host Ports at 4 G

B

A B
Ports Deployed: Used Ports: Storage Ports (2 G Dedicated): Host Ports (4 G Shared): ISL Oversubscription (Ports): Disk Oversubscription (Ports): 1200 1200 192 896 6.4 : 1 9.3 : 1

14 Racks 32 Dual Attached Servers per Rack
Cisco Public

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

45

Blade Centers—4G ISLs
Top of Rack
(11200 Usable Ports per Fabric)

Blade Server Design Using 2 x 4 G ISL per Blade Switch; Oversubscription Can Be Reduced for Individual Blade Centers by Adding Additional ISLs as Needed

A 120 Storage Ports at 2 G 60 ISL to Edge at 4 G

B

A

B

Ports Deployed: Used Ports: Storage Ports (2 G Dedicated): Host Ports (4 G Shared): ISL Oversubscription (Ports): Disk Oversubscription (Ports):

1608 1440 240 960 8: 1 8: 1

2 ISL to Core at 4G 16 Host Ports at 4G

Five Racks 96 Dual Attached Blade Servers per Rack

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

46

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

23

Collapsed Core/Edge Design
Traditional Core-Edge Design Cisco MDS 9500 Director with Collapsed-Core Configuration Full Performance
(Non-Oversubscribed, Non-Blocking)

Host Optimized
(Oversubscribed, Non-Blocking)

Collapsed Core
Typically a lower oversubscription ratio Room to grow—empty slots = future port count growth While Director ports are more expensive than Fabric switch ports, Collapsed Core design has no wasted ports for ISLs—similar cost/usable port
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

47

Medium Scale Dual Fabric— Collapsed Core Design
Dual Director Switches
(Up to 528 Ports per Fabric) Medium Scale Design Leveraging 48-Port Modules with Port Bandwidth Reservations to Provide High-Density, Cost-Effective Solution
VSAN support Port bandwidth reservations guarantee performance for those devices that require it Port Channels with HA to other switches for future growth, scaling from a collapsed core to core/edge design
Ports Deployed: Usable Ports: Usable (Available) Ports: Design Efficiency: End-to-End Oversubscription 528 528 0 100% 10 : 1 (480 : 48)

96 Storage Ports .. 11 X 48-Port Modules 528 Ports Total 48 Ports for Storage 480 Ports for Hosts 10:1 Oversubscription

960 Host Ports

Within Each Port Group: One or Two Ports to Storage 11 or 10 Ports to Hosts Port Bandwidth Reservations Used

Storage (1x or 2x Dedicated)

Hosts (11x or 10x Shared)
48

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

24

Top-of-Rack Consolidated I/O
I/O Consolidation at Access
LAN Core SAN-A
8

SAN-B

Distribution MDS 9500

8

POD 1

POD N

Access
Nexus 5000

10 GE/FCoE CNA
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

iSCSI Design
Take advantage of what IP (IPv4, IPv6) and IPS have to offer Low cost with many options Proxy initiator iSCSI Server Load Balancing (iSLB)
Initiator Configured to See Targets at Virtual Address MDS9509-1 Real GigE Address IP: 10.0.0.101 MAC: 0005.3000.aabf FC SAN

Storage Array

IP Network Virtual Address IP: 10.1.10.100 MAC: 0000.5e00.0101

pWWN a
10.1.40.163
iSCSI

pWWN b
Real GigE Address IP: 10.0.0.102 MAC: 000e.38c5.b82d Initiator with NIC Teaming
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Primary/Secondary pwwn Alias and Trespassing MDS9509-2 50

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

25

SAN Extension—Design Considerations
Transport
Optical or IP WAN/MAN

Primary Data Center

High availability
Application availability, IVR

Optimal performance: latency and throughput
Application performance, tape and write acceleration

WAN/MAN

Resilience to WAN problems WAN bandwidth: optimal use and lowest cost
WAN bandwidth utilization, hardware compression

QoS: maintain and assure service
Traffic management, IVR, QoX, TCP tuning, IPv6

Data security in transit
FCIP encryption and FC-SP auth
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Backup Data Center
51

Design Optimization

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

52

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

26

Blocking—Impact on Design Performance
Performance can be adversely affected across an entire multiswitch FC Fabric by a single blocking port
HOL is a transitory event (until some BB_Credits are returned on the blocked port)

To help alleviate the blocking problem and enhance the design performance
Virtual Output Queuing (VoQ) on all ports Deep Buffers—255 BB_Credits per port Fibre Channel Congestion-Control (FCC)—Detects congested ports and throttles the port causing the congestion at its origin

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Advanced Traffic Management
Port bandwidth reservation
Dedicated mode—ports can act at any dedicated rate including line rate Shared mode Enhance utilization
Department/ Customer ‘A’ (Low Priority) Department/ Customer ‘B’ (High Priority)

Oversubscription
Round robin fairness Assured fairness
Potential Bottlenecks

VSANEnabled Fabric VSAN Trunks

Port Channel to scale connectivity
Bundle ISLs between switches Additional resiliency

In-order frame delivery guarantee (IOD)
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Shared Storage
54

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

27

Advanced Traffic Management
QoS allows traffic to be intelligently managed
Minimizes impact of oversubscription Allows more economical topologies Prioritize traffic by flow
VSANEnabled Fabric

Department/ Customer ‘A’ (Low Priority)

Department/ Customer ‘B’ (High Priority)

VOQ for switch performance
Slow flows to not disrupt fast flows Non-blocking frame forwarding
Potential Bottlenecks

VSAN Trunks

FCC mechanism can throttle back traffic at its origin
Manage congestion on ISLs

Shared Storage
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

55

Enhanced Quality of Service (QoS)
Arbiter-aware QoS
Requires Supervisor 2

Enabled within a switch or across the network
Allows QoS in a single switch configuration

No longer requires FCC to operate

User definable DiffServ

DWRR Priority Queue 2 Queue 3 Queue 4
PQ

Weight Absolute 60 10 30 Transmit Queue

DWRR 2 DWRR 3 DWRR 4
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

56

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

28

Enhancing SAN Design
Extending the effective distance for remote applications
SAN acceleration—Local Ack Write acceleration Tape acceleration Reduces WAN-induced latency Improves application performance over distance
Media Servers Increased distance Improved performance

SAN Extension with FC-WA

Backup design performance enhancement
Serverless backup—NASB Offloading the I/O and CPU work from Media Server

SSM SAN

Application Servers

Tape

Disk

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

N-Port Virtualizer (NPV)
Enabling Large-Scale Blade Server Deployments
Deployment Model—FC Switch Mode
Blade System
Blade N Blade 2 Blade 1 E-Port
BRKSAN-2701 14570_04_2008_c1

Deployment Model—HBA Mode
Blade Switch Configured as NPV (i.e., HBA Mode)
Blade System
Blade N Blade 2 Blade 1

Blade System
Blade N Blade 2 Blade 1

Blade System
Blade N Blade 2 Blade 1

FC Switch

FC Switch

NPV

NPV

NPV Enables Large Scale Blade Server Deployments By:
E-Port

N-Port

SAN

Reducing Domain ID usage Addressing switch interop issues Simplifying management
F-Port

SAN

Storage

Storage

Blade Switch Attribute FC Switch Mode (E-Port) One per FC Blade Switch Yes Medium
© 2008 Cisco Systems, Inc. All rights reserved.

Deployment Model # of Domain IDs Used Interoperability Issues with Multivendor Core SAN Switch Level of Management Coordination Between Server and SAN Administrators
Cisco Public

HBA Mode (N-Port) None (Uses Domain ID of Core Switch No Low
58

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

29

SAN Security

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

SAN Design Security Challenges
SAN design security is often overlooked as an area of concern
Application integrity and security is addressed, but not back-end storage network carrying actual data SAN extension solutions now push SANs outside datacenter boundaries

Not all compromises are intentional
Accidental breaches can still have the same consequences

FC

SAN design security is only one part of complete data center solution
Host access security—one-time passwords, auditing, VPNs Storage security—data-at-rest encryption, LUN security Data center physical security External Dos or Other Intrusion Privilege Escalation/ Unintended Privilege
FC FC FC
FC FC FC

Theft Unauthorized Connections (Internal) Data Tampering

Application Tampering (Trojans, etc.)

FC

FC FC
FC FC

SAN

FC

LAN
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

60

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

30

SAN Security—Design Considerations
Protecting data
Data integrity and encryption in transit or at rest Data Integrity and Secrecy
SAN Fabric Protocol Security

Securing against unauthorized user and device access
Target

User/device authorization and authentication Server and target access controls

SAN Fabric
Host

Guarding against malicious management misconfiguration
Management access controls Securing the SAN management information
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Target Access Security Fabric Access Security
iSCSI

SAN Management Security

IP Storage Security (iSCSI/FCIP)

61

Intelligent Fabric Applications

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

31

Intelligent Storage Applications
Delivered as a Transparent Fabric Service
Application Servers

Extend storage services to any device in the SAN Transparent to applications Nondisruptive deployment
No SAN reconfiguration No rewiring to insert appliances

MSM-18/4

MSM-18/4

Highly scalable performance Automatic load balance Reliable, highly available service Wizard-based provisioning
Storage Array

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

Cisco Storage Media Encryption (SME)
Application Server
Name: XYZ SSN: 1234567890 Amount: $123,456 Status: Gold

Encrypts storage media (data at rest)
IEEE compliant AES-256 encryption Integrated as transparent fabric service

Transparent Fabric Service
Key Management Center SME SME IP

Supports heterogeneous storage arrays, tape devices, and VTLs Compresses tape data Offers secure, comprehensive key management Allows offline media recovery

@!$%!%!%!%%^& *&^%$#&%$#$%*!^ @*%$*^^^^%$@*) %#*@(*$%%%%#@

Storage Array

Tape Library

Built upon FIPS Level 3 system architecture

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

64

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

32

Cisco Data Mobility Manager (DMM)
Application Servers

Migrates data between storage arrays for
Technology refreshes Workload balancing Storage consolidation

DMM offers
Data Mobility Manager

Online migration of heterogeneous arrays Simultaneous migration of multiple LUNs Unequal size LUN migration

Application I/O

Data Migration

Rate adjusted migration Verification of migrated data Secure erase Dual fabric support CLI and wizard-based management with Cisco Fabric Manager

Old Array

New Array

Utilizes Storage Services Modules (SSM)
BRKSAN-2701 14570_04_2008_c1

Requires no SAN reconfiguration or rewiring
65

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Network Assisted Storage Applications
SAN Tap
Enables appliance-based storage applications without compromising SAN integrity About SAN Tap
MDS delivers a copy of primary I/O to an appliance
Copy of Primary I/O

Initiator

Initiator

Target I/O

SAN

Appliance

Appliance provides the storage application Examples of applications include Continuous Data Protection (CDP), replication, etc.

Target

Key customer benefits
Preserve integrity, availability, and performance of primary I/O No service disruption

= SAN Tap
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Investment protection
66

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

33

Storage Virtualization for Cisco MDS 9000
Control Processor

Independent control path Program the data path Processes exceptions

Services
Volume management Data migration Point-in-time copies

FAIS

High-performance fast path Integrated, HA architecture Multiprotocol integration

SSM
Data Path

Comprehensive security Troubleshooting and diags

Virtual to Physical Mapping
Data Traffic
BRKSAN-2701 14570_04_2008_c1

Control Traffic
Cisco Public

© 2008 Cisco Systems, Inc. All rights reserved.

67

SecureErase on Offline Storage Arrays
Configuration
Deploy a SSM Create VIs Zone VIs and target ports Expose target LUNS to VI Create and start SE sessions
Virtual Initiators on SSM
Cisco Fabric Mgr
FC FC

Key Benefits In-place SecureErase
No window of vulnerability

Vendor agnostic
Heterogeneous Storage Arrays

Compliance
DoD, RCMP, and Gutmann

Easy to deploy and cost effective
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

68

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

34

Interoperability

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

Standard Fibre Channel Interoperability
Switch Interoperability Is Available Between the Cisco MDS Platforms and Non-Cisco ‘Standards Compliant’ Switches
Provides way to redeploy smaller edge switches Both McDATA and Brocade switches must be in ‘interoperability mode’
On both products this results in the loss of some functionality No trunking No port-based zoning No full zone-set exchanges Restricted # of domains and their ID

TE

E

Non-Cisco Switch in ‘Interoperability’ Mode Cisco RA_T_O_V ED_T_O_V Brocade McDATA

Must also choose fabric timer values that are the same and available across all vendors— must be the same fabric-wide Must enable ‘interoperability’ mode on MDS 9000
No loss of functionality on MDS 9000 Enabled per-VSAN—‘Interop mode 1’
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

500040001000100000 ms 120000 ms 120000 ms 10001000DS_TOV ms 20000 ms
v2.4.1 and v3.0.1a

20060000 ms
v04.01.00 12

70

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

35

Third-Party Switch Native Mode
Enables MDS9000 family to interoperate with legacy fabric switches in ‘native mode’— Brocade and McDATA (SANOS3.1)
Reuse existing legacy fabric switches as edge devices No impairment to Cisco fabric— all advanced services available No change required on legacy switches—simply connect switches
VSAN 50 VSAN 40

Cisco MDS 9000 Family

Legacy Brocade Switches

Legacy McDATA Switches

Three additional modes
Interop mode 2 Interop mode 3 Interop mode 4
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Configurable on a VSAN-byVSAN Basis on MDS 9000

71

Closing Remarks

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

72

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

36

Closing Remarks—SAN Design
Simple SAN design
Small port count No interswitch links Dual fabric Minimal SAN/OS feature use

Scalable SAN design
Core/edge Collapsed core/edge

Director class collapsed CORE SAN
Large port count High availability design Port Channels, FSPF routing Multi-SAN/OS feature use

Core director SAN with blade server and edge switches
Multidomains Interop
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

73

BRKSAN-2704: SAN Extension BRKSAN-2719: From Storage Component Virtualization to End-to-End Virtualization BRKSAN-2705: iSCSI SAN Design and Operation BRKSAN-2702: SAN Migration and Interoperability BRKSAN-3707: Advanced SAN Design

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

74

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

37

Additional Information
Cisco Storage Networking
http://www.cisco.com/go/storagenetworking

Cisco Data Center Networking
http://www.cisco.com/go/datacenter

Storage Network Industry Association (SNIA)
http://www.snia.org

Internet Engineering Task Force—IP Storage
http://www.ietf.org/html.charters/ips-charter.html

ANSI T11—Fibre Channel
http://www.t11.org/index.htm
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

75

Q and A

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

38

Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store
BRKSAN-2701 14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

77

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

39

BRKSAN-2701 14570_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

79

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

40