You are on page 1of 40

BRKSAN-2701

14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1

Fibre Channel Storage
Area Network Design

BRKSAN-2701

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1
Presentation_ID.scr
Agenda

ƒ Brief SAN Technology Overview
Fibre Channel Protocol
Virtual SAN (VSAN), Zoning
Port Channels, IOD

ƒ SAN Design Principles and Considerations
Design Factors
Design Types
Design Flexibility
SAN Security Design Considerations
Interoperability Design Considerations

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

SAN Technology
Overview

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2
Presentation_ID.scr
SAN Technology Overview—Agenda

ƒ Fibre Channel Protocol
FC Communications
Port types, ISL
Addressing, Framing, Timers

ƒ Virtual SAN (VSAN), Zoning
ƒ Port Channels, IOD
ƒ Virtual Output Queuing (VOQ)

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

Fibre Channel Communications
ƒ Point-to-point oriented
Facilitated through device login
ƒ N_Port-to-N_Port connection
Logical node connection point
ƒ Flow controlled
Buffer-to-buffer credits and end-to-end basis
ƒ Acknowledged
For certain classes of traffic, none for others
ƒ Multiple connections allowed per device
Node Node
Transmitter Transmitter
N_Port N_Port
Receiver Receiver

Link

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3
Presentation_ID.scr
FCP—Protocol Stack

SCSI FC

SCSI Applications (File SCSI Applications (File
Systems, Databases) Systems, Databases)

SCSI Block Commands SCSI Stream
Commands

SCSI Commands, Data, SCSI Commands, Data,
and Status and Status

Parallel SCSI Fibre Channel Protocol
Interfaces (FCP)

SCSI Bus
Fibre Channel

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Fibre Channel Port Types
Fibre Channel Switch
Input Fabric Output
Port X Port

Fabric
Node NL_Port FL_Port E_Port E_Port Switch

G_Port F_Port N_Port Node
Node NL_Port

G_Port F_Port N_Port Node
Node NL_Port

G_Port F_Port N_Port Node

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4
Presentation_ID.scr
Inter-Switch Link (ISL)
EISL

ƒ The interconnection between switches is called the ISL
E_Port to E_Port (‘Expansion port)

ƒ Supports all classes of service
Class 1, 2, 3, and a special Class F (switch-to-switch)

ƒ FC-PH permits consecutive frames of a sequence to be routed
over different ISL links for maximum throughput
ƒ Cisco’s implementation is to dedicate an FC_ID pair and/or a given
exchange to an ISL bundle member to guarantee in-order delivery
for exchange/sequence frames
ƒ Cisco Extended ISL (EISL, TE port)
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

Buffer to Buffer Credit Flow Control
BB_Credits and Distance
~ 2 km per Frame
1 Gbps FC

~ 1 km per Frame
2 Gbps FC

~ ½ km per Frame
4 Gbps FC

16 Km

ƒ BB_Credits are used to ensure enough FC frames in flight
ƒ A full (2112 byte) FC frame is approx 2 km long at 1 Gbps,
1 km long at 2 Gbps and ½ km long at 4 Gbps
ƒ As distance increases, the number of available BB_Credits
need to increase as well
ƒ Insufficient BB_Credits will throttle performance—no data will be
transmitted until R_RDY is returned
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5
Presentation_ID.scr
FSPF Protocol

ƒ FSPF stands for Fabric Shortest Path First
ƒ Path selection protocol used in Fibre Channel
ƒ Based on link state protocol
ƒ Fibre Channel standard defined in FC-SW2
ƒ Conceptually based on Open Shortest Path First
(OSPF) Internet routing protocol

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

FC_ID Address Model
ƒ FC_ID address models help speed up routing
ƒ Switches assign FC_ID addresses to N_Ports
ƒ Some addresses are reserved for fabric services
ƒ Private loop devices only understand 8-bit address (0x0000xx)
ƒ FL_Port can provide proxy service for public address translation
ƒ Maximum switch domains = 239 (based on standard)
8 Bits 8 Bits 8 Bits

Switch Topology Switch
Area Device
Model Domain

Private Loop Device Arbitrated Loop
Address Model 00 00 Physical Address
(AL_PA)
Public Loop Device Arbitrated Loop
Switch
Address Model Area Physical Address
Domain
(AL_PA)
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6
Presentation_ID.scr
Fibre Channel FC-2 Hierarchy
ƒ Multiple exchanges are initiated between initiators (hosts) and
targets (disks)
ƒ Each exchange consists of one or more bidirectional sequences
ƒ Each sequence consists of one or more frames
ƒ For the SCSI3 ULP, each exchange maps to a SCSI command

OX_ID and
RX_ID Exchange

SEQ_ID Sequence Sequence Sequence

SEQ_CNT Frame Frame Frame

Frame Fields
ULP Information Unit
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Fibre Channel Timers
ƒ Receiver-Transmitter Time-Out (R_T_TOV)
Used to time events at the link level
Loss of synchronization
Times responses for link reset protocol
Default value: 100 ms
ƒ Error Detect Time-Out (E_D_TOV)
Timers for events and responses at the sequence level
Missing ACK or R_RDY when buffer credit has reached zero
N_Port logout
Timer value is set at fabric login to accommodate the network environment
based on delivery time of frames
Default value: 2 sec
ƒ Resource Allocation Time-Out (R_A_TOV)
Time-out value for how long to hold resources associated with a failed operation
Needed to free shared resources for reuse
Value to determine how long a port needs to keep responding to a link service request
before an error is detected
2xE_D_TOV in point-to-point and fabricwide from 1 to 231 – 1
Default value: 10 s
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7
Presentation_ID.scr
VSANs, Zoning, IVR Zones

ƒ Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single
physical switch
VSAN—Design Foundation
ƒ Zoning
ƒ Fabric routing (Inter-VSAN Routing—IVR)
Ability to provide selected connectivity between virtual fabrics
without merging them
ƒ Virtual Fabric Trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL
or common group of ISLs
ƒ IVR zones
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

VSANs, Zoning, IVR Zones

ƒ Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single
physical switch
VSAN—Design Foundation
ƒ Zoning
ƒ Fabric routing (Inter-VSAN Routing–IVR)
Ability to provide selected connectivity between virtual fabrics
without merging them
ƒ Virtual Fabric Trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL
or common group of ISLs
ƒ IVR zones
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8
Presentation_ID.scr
Virtual Storage Area Networks (VSAN)
ƒ VSANs address the limitations of
common SAN deployments
VSANs are virtual fabrics
SAN islands are virtualized onto a
common SAN infrastructure
Allocate ports within a physical fabric to Independent Physical SAN
Islands Are Virtualized onto a
create virtual fabrics Common SAN Infrastructure
Fabric disruption is limited to VSAN
Traffic statistics are gathered per VSAN

ƒ Features include:
Dynamic provisioning and resizing
Improved port utilization
Shared ISL bandwidth
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Virtual SANs—VSANs
Production SAN Tape SAN Test SAN

FC

FC FC
FC

FC

SAN C SAN D SAN E SAN F
SAN B DomainID=3 DomainID=4 DomainID=5 Domain ID=6
SAN A
DomainID=1 DomainID=2
DomainID=7 DomainID=8

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9
Presentation_ID.scr
Zoning
Zoneset
ƒ Devices within a zone can access
each other Zone A
Zoning establishes access control FC
FC Zone B
Limiting access prevents unauthorized access FC
FC
FC
FC

ƒ Soft zoning FC

Implemented in switch software and FC FC
enforced by name server FC
FC FC FC
Name server responds to discovery FC

queries with only devices found in Zone C FC

requestor’s zone or zones
“Soft zoning” used to be synonymous
with “WWN zoning”
ƒ Hard zoning Default Zone
Enforced by ACLs in port ASIC FC FC
FC FC
Applied to all data path traffic FC

FC
FC
FC
“Hard zoning” used to be synonymous FC
FC

with “port zoning” FC
FC

ƒ WWN zoning in hardware FC

Both at source and destination ports Devices Not Assigned to a Zone
Enhanced zoning Belong to the Default Zone
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Zoning and VSANs
ƒ Hierarchical relationship Zones and VSANs Are Complementary
First assign physical ports to VSANs Physical Topology
Then configure independent VSAN 2 Active Zoneset A
zones per VSAN ZoneA
VSANs only change when ports Host1 Disk1
Disk3 Disk2
needed per virtual fabric
ZoneC
Zones can change frequently (e.g., backup) ZoneB
Default Host2
ƒ Zones provide added security and Zone Disk4

allow sharing of device ports
VSAN 7 Active Zoneset D
ƒ Zone membership is configured by:
Host4
Port World Wide Name (pWWN)—device
ZoneD
Fabric World Wide Name (fWWN)—fabric Default
Zone ZoneA
Disk5
Fibre Channel Identifier (FCID) Host3
Disk6
Fibre Channel Alias (FC_Alias)
IP address
Domain ID/port number One Active Zoneset per VSAN
Interface
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10
Presentation_ID.scr
VSANs, Zoning, IVR Zones

ƒ Fabric virtualization—VSAN
Provide independent (‘virtual’) fabric services on a single
physical switch
VSAN—Design foundation
ƒ Zoning
ƒ Fabric routing (Inter-VSAN Routing—IVR)
Ability to provide selected connectivity between virtual fabrics
without merging them
ƒ Virtual fabric trunking (VSAN Trunking)
Ability to transport multiple virtual fabrics over a single ISL
or common group of ISLs
ƒ IVR zones
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

VSANs—Routed Connectivity
Common Physical Fabric
Sharing a Common
Resource such as Tape
Common Physical Fabric
Sales
SAN

HR
SAN
MS Sales Marketing
MS SAN
SAN MS
Marketing
SAN

SAN IP
MS Tape Production and or
SAN Extension
HR DR Interconnect Without Services FC
SAN Merging Fabrics

Tape Media Server Potentially Separate
MS Administrators per SAN

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11
Presentation_ID.scr
Inter-VSAN Routing—IVR
ƒ Enables devices in different VSANs to communicate
ƒ Allows selective routing between specific members of two or
more VSANs
Traffic flow between selective devices

ƒ Resource sharing, i.e., tape libraries and disks

VSAN 10 IVR Zone VSAN 20
Media Server Tape Library
FC FC
FC
FC
FC FC
FC

FC

FC FC
FC
FC FC

FC FC
FC FC

FC
FC FC
FC

FC FC
FC FC FC FC

FC
FC FC FC

FC FC

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

IVR Zones
ƒ IVR zone Physical Topology

A container or access VSAN 2
control, containing two or Disk2 Disk3

more devices in different ZoneA Host1
Disk1

VSANs ZoneC

ƒ Standard zones are still Disk4 Host2
ZoneB
used to provide intra-
VSAN access
VSAN 3

ƒ IVR Zoneset Host4

ZoneD
A collection of IVR zones ZoneA
Disk5
that must be activated to be Host3

operational Disk6

Inter-VSAN Zone
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12
Presentation_ID.scr
Port Channels
Port Aggregation Feature Used to Create a
Single Logical ISL from 1–16 Physical ISLs
ƒ Increases bandwidth and
availability
ƒ Very granular load balancing—
per exchange/src/dst or
per src/dst (policy on a per
VSAN basis)
ƒ Interfaces can both be
added and removed in a
nondisruptive manner in
4-Link Port Channel
production environments
EISL
ƒ Preserved FC guarantee
of in-order delivery (IOD)
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Port Channel Protocol (PCP)
ƒ Exchange-based, in-order
load balancing
Mode 1: based on src/dst
FC_ID/OX_ID/RX_ID
Mode 2: based on src/dst FC_ID
ƒ Consistently detect misconfiguration Up to 160 Gbps
Port Channel
ƒ Transition misconfigured ports to with HA
isolated state so as to be able to
correct the misconfig
ƒ Synchronize bring up of ports in a
channel across peer switches
ƒ Provide the ability for the system to
automatically create Port Channels
among compatible ports
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13
Presentation_ID.scr
VSANs, Trunking, Port Channels
ƒ Hierarchical relationship VSAN
10
Metric
100
VSAN Metric
10 50
Port Channels provide link aggregation to 20 50
yield virtual ISL (E_Port)
Single-link ISL or Port Channel ISL
can be configured to become EISL—
(TE_Port)
VSANs can be selective grafted or
pruned from EISL trunks 8 Gbps
Port Channel
ƒ All member links of a port channel p
Trunking

VS On
20 cku
must have same configuration prior E_Port

AN ly
E_Port AN Ba
VS 10

10
to creating channel (e.g., TE_Port (TE_Port) AN
VS
or E_Port, VSANs-enabled, etc.)
ƒ Port Channel technology provides Trunking E_Port
E_Port
high availability and fast recovery (TE_Port)
for VSAN trunk (EISL)
4-Link (8 Gbps) Port
ƒ Multiple Port Channels yield Channel Configured
multiple paths for custom traffic as EISL
engineering
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

IOD: In-Order Delivery—
Frames Are Delivered In-Order
ƒ Frames of a flow (SID,DID) are
received in the order they were sent
Some applications (FICON and HP CA Frames Left in Output
Queues/Buffers at Peer Switch
EVA) can not tolerate out of order frames
ƒ MDS HW always keeps frames Frames Left in Output 5 2
within same exchange on same Queues/Buffers 6 1
at Local Switch
path except during
FSPF route changes
4 3
Port Channel (PC) membership changes
7 8
ƒ With IOD the frame waits for 500ms
(switch latency value where frames
are dropped) to guarantee that all
frames have been drained from
Recovery Path Could Carry Frames
all the VOQs in the peer switch from Same Exchange Out-of-Order
ƒ Once the peer has performed
this operation, it responds with
an acknowledgement and traffic
forwarding can begin immediately
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14
Presentation_ID.scr
Virtual Output Queuing (VOQ)
B C A C C C C C A B Input Output FC
A
Port Port

Input Output B
Port Port
ƒ Switch with no VOQ
HOL blocking FC
Input Output C
port Port

--------------------------------------------------------------------------------------------------------------------------------------------------------------

C C C C
Input Output FC
B C A C C C C C A B B Port Port A
A ARB
Input Output B
Port Port
ƒ Switch with VOQ support
FC
No HOL blocking
Input Output C
VOQ alleviates HOL Port Port

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

SAN Design Principles
and Considerations

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15
Presentation_ID.scr
Agenda

ƒ Brief SAN Technology Overview
Fibre Channel Protocol
Zoning, Virtual SAN (VSAN)
Port Channels, IOD

ƒ SAN Design Principles and Considerations
Design Factors
Design Types
Design Optimization
SAN Security Design Considerations
Interoperability Design Considerations

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

SAN Design—Principles and
Considerations
ƒ Determine components to
Data Center Servers
be used and how they will
fit into your overall strategy
ƒ Creation of technical
infrastructure and how
the pieces will fit together Storage
Network
ƒ Determine how existing
infrastructure and new one
will be integrated
ƒ Creation of the processes Storage
and procedures that will Disk and Tape

guide personnel in how the
infrastructure is to be used
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16
Presentation_ID.scr
Design Factors

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Early SAN Designs
SAN Islands
1. First SANs hardly qualified Low Traffic
Across ISLs
as networks
4 Host
SAN islands of two to four switches Host Host

Fixed 8–16 port switches limited SAN growth 1 Few Switches

2. No fabric segmentation such
as VSANs
Host
3. Limited enhancements to FSPF
No Port Channeling
No equal cost load balancing

4. Traffic management was Switches
Managed
not needed Separately 3

No QoS because bandwidth 5 Single
was over provisioned Routes

5. Management tools focus Faults Impact
all Devices
on element management Host
Host Host Host Host

2
Not network management
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17
Presentation_ID.scr
SAN Major Design Factors
1. Port density
High
How many now, how many Performance
Crossbar
later? topology Large Port
2
Count
2. Network performance QoS,
Directors

Congestion
What is acceptable? Control, 4
Unavoidable? Reduce FSPF 1
Routes
3. Traffic management 8 8 8 8 8 8 8 8 8 8 8 8

Preferential routing or
resource allocation
4. Fault isolation
Host Host Host Host Host Host
Consolidation while
maintaining isolation 3
Failure of One Device Has
5. Management No Impact on Others

Secure, simplified management
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

1. Scalability—Port Density,
Topology Requirements
ƒ Number of ports for end devices
High
ƒ How many ports are Performance
needed now? Crossbar
Large Port
2
Count
ƒ What is the expected Directors
life of the SAN? QoS,
Congestion
Control, 4
ƒ How many will be needed in Reduce FSPF 1
the future? Routes
8 8 8 8 8 8 8 8 8 8 8 8
ƒ Hierarchical SAN design

Best Practice
ƒ Design to cater for future
requirements Host Host Host Host Host Host

ƒ Doesn’t imply “build it all now,” 3
but means “cater for it” and Failure of One Device Has
avoids costly retrofits tomorrow No Impact on Others

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18
Presentation_ID.scr
2. Network Performance—
Oversubscription Design Considerations
All SAN Designs Have Some
Degree of Oversubscription Disk Oversubscription
Disk Do Not Sustain
ƒ Without oversubscription, Wire-Rate I/O with
‘Realistic’ I/O Mixtures Tape Oversubscription Low
Sustained I/O Rates
SANs would be too costly Most Major Vendors
LTO-2 Native Transfer
Promote 12:1 host:disk
Fan-Out Rate ~ 60 MBps
ƒ Oversubscription is introduced
70 MBps 60 MBps
at multiple points Max/Port Max/Port
(Common) (Common)
ƒ Switches are rarely the bottleneck
in SAN implementations Typical
Oversubscription
Port Channels
Help Reduce
in Two-Tier Design Oversubscription
ƒ Device capabilities (peak and Can Approach 8:1,
Sometimes Even
While Maintaining
HA Requirements
sustained) must be considered Higher

along with network 7:1 O.S.
(Common)
oversubscription
ƒ Must consider oversubscription
during a network failure event
40 MBps Host Oversubscription

ƒ Remember, all traffic flows Max/HBA
(Common)
Most Hosts Suffer from PCI Bus
Limitations, OS, and Application
Limitations Thereby Limiting Maximum
towards targets—main Host I/O and Bandwidth Rate
bottlenecks
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

3. Traffic Management
ƒ Do different apps/servers
High
have different performance Performance
requirements? Crossbar
Large Port
2
Count
Should bandwidth be Directors
QoS,
reserved for specific Congestion 4
applications? Control,
Reduce FSPF 1
Is preferential treatment/ Routes

QoS necessary? 8 8 8 8 8 8 8 8 8 8 8 8

ƒ Given two alternate paths
for traffic between data
centers, should traffic use Host Host Host Host Host Host
one path in preference to
the other? 3
Failure of One Device Has
Preferential routes No Impact on Others

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19
Presentation_ID.scr
4. Fault Isolation
Consolidation of Storage into
a Single Fabric = Increased
Storage Utilization + Reduced
Administration Overhead
Major Drawback Is That Faults
Are No Longer Isolated Physical SAN Islands Are
Virtualized onto Common
SAN Infrastructure
ƒ Technologies such as VSANs
enable consolidation and
scalability while maintaining
security and stability
Fabric
ƒ VSANs constrain fault impacts #3

Fabric
Faults in one virtual fabric (VSAN) #1 Fabric
#2
are contained and do not impact
other virtual fabrics

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

5. Management
Consolidation and Large Scale SAN Becomes
More Difficult to Manage. How Can It Be More
Secure? How Can the SAN Traffic Be Monitored
as Performance Requirements Increase?
FM Client FMS FM Clients
FC

SNMP FC

SNMP
SNMP
Tools and Features
ƒ RBAC on per VSAN basis
ƒ FC-SP for switch-to-switch or device-to-switch security
ƒ Fabric Manager Server
ƒ Online traffic monitoring through Device Manager
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20
Presentation_ID.scr
Design Types

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Core-Edge
ƒ Traditional SAN design for
growing SANs
ƒ High density directors in
core and fabric switches, A B
directors or blade switches
on edge
ƒ Predictable performance
A B
A B
ƒ Scalable growth up to core A B
and ISL capacity

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21
Presentation_ID.scr
Large Core-Edge Example
Large Core/Edge
(2240 Usable Ports per Fabric) “A” Fabric Shown,
Repeat for “B” Fabric
Traditional Core-Edge Design
Is Ideal for Centralized
Services and Consistent
Host-Disk Performance 128 Storage
Regardless of Location; Ports at 2 G
Less Future Scalability 64 ISL to
Edge at 4 G
ƒ Requires third tier to scale
ƒ Port bandwidth
reservations guarantee 32 ISL to
Core at 4 G
performance for ISLs
496 Host
(edge side) Ports at 4 G

Ports Deployed: 2520
Used Ports: 2512
Storage Ports (2 G Dedicated): 256
Host Ports (4 G Shared): 1984
ISL Oversubscription (Ports): 15.5 : 1
Disk Oversubscription (Ports): 15.5 : 1

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

Scaling Traditional Core-Edge
with Three Tiers
Could Scale to ~ 40k Ports*
12 ISL to
Core at 4 G

124 Storage
Ports at 2 G
6 ISL to
Backbone at 4 G
64 ISL to
Edge at 4G

32 ISL to
Core at 4 G
496 Host
Ports at 4 G
15.5 : 1 Host
to Core

* No software testing has been done to this level of scalability
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22
Presentation_ID.scr
Top of Rack Example—
9134 and 10 G ISLs
Top of Rack
(1088 Usable Ports per Fabric)
Top of Rack Design Utilizing
10 Gb ISLs; High Bandwidth
ISLs Provides Ample A B
Performance and 96 Storage
Reduces Cabling Ports at 2 G
28 ISL to Edge
at 10 G
2 ISL to
Core at 10 G
32 Host Ports
at 4 G

A
B
Ports Deployed: 1200
Used Ports: 1200
Storage Ports (2 G Dedicated): 192
14 Racks
Host Ports (4 G Shared): 896
32 Dual
ISL Oversubscription (Ports): 6.4 : 1 Attached
Disk Oversubscription (Ports): 9.3 : 1 Servers per
Rack
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Blade Centers—4G ISLs
Top of Rack
(11200 Usable Ports per Fabric)
Blade Server Design Using 2
x 4 G ISL per Blade Switch;
Oversubscription Can Be A B
Reduced for Individual Blade 120 Storage
Centers by Adding Additional Ports at 2 G
ISLs as Needed 60 ISL to Edge
at 4 G

A B

2 ISL to
Core at 4G
Ports Deployed: 1608
16 Host
Used Ports: 1440 Ports at 4G Five Racks
Storage Ports (2 G Dedicated): 240 96 Dual
Host Ports (4 G Shared): 960 Attached
ISL Oversubscription (Ports): 8: 1 Blade Servers
Disk Oversubscription (Ports): 8: 1
per Rack

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23
Presentation_ID.scr
Collapsed Core/Edge Design
Traditional Cisco MDS 9500 Director with
Core-Edge Design Collapsed-Core Configuration

Full Performance
(Non-Oversubscribed,
Non-Blocking)

Host Optimized
(Oversubscribed,
Non-Blocking)

Collapsed Core
ƒ Typically a lower oversubscription ratio
ƒ Room to grow—empty slots = future port count growth
ƒ While Director ports are more expensive than Fabric switch ports,
Collapsed Core design has no wasted ports for ISLs—similar
cost/usable port
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

Medium Scale Dual Fabric—
Collapsed Core Design
Dual Director Switches 96
(Up to 528 Ports per Fabric) Storage
Ports
Medium Scale Design Leveraging ..
48-Port Modules with Port
Bandwidth Reservations to
11 X 48-Port Modules
Provide High-Density, 528 Ports Total
Cost-Effective Solution 48 Ports for Storage
480 Ports for Hosts
ƒ VSAN support 10:1 Oversubscription

ƒ Port bandwidth reservations
guarantee performance for
those devices that require it 960 Host Ports
ƒ Port Channels with HA to other
switches for future growth,
scaling from a collapsed core
to core/edge design
Storage
Ports Deployed: 528 Within Each (1x or 2x Dedicated)
Usable Ports: 528 Port Group:
One or Two Ports
Usable (Available) Ports: 0
to Storage
Design Efficiency: 100% 11 or 10 Ports
End-to-End Oversubscription 10 : 1 to Hosts
(480 : 48) Port Bandwidth Hosts
Reservations Used (11x or 10x Shared)

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24
Presentation_ID.scr
Top-of-Rack Consolidated I/O
I/O Consolidation at Access

LAN Core
SAN-A SAN-B

8

Distribution 8

MDS 9500

POD 1 POD N Access
Nexus 5000

10 GE/FCoE
CNA

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

iSCSI Design
ƒ Take advantage of what IP (IPv4, IPv6) and IPS have to offer
ƒ Low cost with many options
ƒ Proxy initiator
ƒ iSCSI Server Load Balancing (iSLB)
Initiator Configured to See
Targets at Virtual Address MDS9509-1
Real GigE Address
IP: 10.0.0.101 Storage Array
MAC: 0005.3000.aabf
IP Network
FC SAN
pWWN a
Virtual Address
10.1.40.163 IP: 10.1.10.100
iSCSI
MAC:
0000.5e00.0101

pWWN b

Real GigE Address
IP: 10.0.0.102
MAC: 000e.38c5.b82d Primary/Secondary pwwn
Initiator with Alias and Trespassing
NIC Teaming MDS9509-2
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25
Presentation_ID.scr
SAN Extension—Design Considerations
ƒ Transport Primary Data Center
Optical or IP WAN/MAN
ƒ High availability
Application availability, IVR
ƒ Optimal performance: latency and
throughput
Application performance, tape and write
acceleration WAN/MAN
ƒ Resilience to WAN problems
ƒ WAN bandwidth: optimal use and
lowest cost
WAN bandwidth utilization, hardware compression
ƒ QoS: maintain and assure service
Traffic management, IVR, QoX, TCP tuning, IPv6
ƒ Data security in transit Backup Data Center
FCIP encryption and FC-SP auth
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

Design Optimization

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26
Presentation_ID.scr
Blocking—Impact on
Design Performance

ƒ Performance can be adversely affected across an
entire multiswitch FC Fabric by a single blocking port
HOL is a transitory event (until some BB_Credits are returned
on the blocked port)

ƒ To help alleviate the blocking problem and enhance
the design performance
Virtual Output Queuing (VoQ) on all ports
Deep Buffers—255 BB_Credits per port
Fibre Channel Congestion-Control (FCC)—Detects congested
ports and throttles the port causing the congestion at its origin

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Advanced Traffic Management
Department/ Department/
ƒ Port bandwidth reservation Customer ‘A’ Customer ‘B’
(Low Priority) (High Priority)
Dedicated mode—ports can act at
any dedicated rate including line rate
Shared mode
Enhance utilization

ƒ Oversubscription VSAN-
Enabled
Round robin fairness Fabric
Assured fairness
Potential VSAN
ƒ Port Channel to scale Bottlenecks Trunks

connectivity
Bundle ISLs between switches
Additional resiliency

ƒ In-order frame delivery
guarantee (IOD) Shared Storage
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27
Presentation_ID.scr
Advanced Traffic Management
Department/ Department/
ƒ QoS allows traffic to be Customer ‘A’ Customer ‘B’
(Low Priority) (High Priority)
intelligently managed
Minimizes impact of
oversubscription
Allows more economical
topologies
Prioritize traffic by flow VSAN-
Enabled
ƒ VOQ for switch performance Fabric

Slow flows to not disrupt fast flows Potential VSAN
Bottlenecks Trunks
Non-blocking frame forwarding

ƒ FCC mechanism can throttle
back traffic at its origin
Manage congestion on ISLs

Shared Storage
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

Enhanced Quality of Service (QoS)
ƒ Arbiter-aware QoS ƒ Enabled within a switch or
Requires Supervisor 2 across the network
Allows QoS in a single switch
ƒ No longer requires FCC to configuration
operate
ƒ User definable DiffServ

DWRR Weight
Priority Absolute
Queue 2 60
Queue 3 10
Queue 4 30

PQ Transmit
Queue
DWRR 2

DWRR 3

DWRR 4
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28
Presentation_ID.scr
Enhancing SAN Design
ƒ Extending the effective ƒ Increased distance
distance for remote ƒ Improved performance
applications
SAN acceleration—Local Ack
Write acceleration
Tape acceleration
SAN Extension with FC-WA
Reduces WAN-induced latency
Improves application performance
over distance Media
Servers
ƒ Backup design
performance enhancement SSM Application
SAN Servers
Serverless backup—NASB
Offloading the I/O and CPU work
from Media Server Tape Disk

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

N-Port Virtualizer (NPV)
Enabling Large-Scale Blade Server Deployments
Deployment Model—FC Switch Mode Deployment Model—HBA Mode
Blade System Blade System Blade Switch Blade System Blade System
Configured as
NPV (i.e.,
Blade N

Blade N
Blade N

Blade N
Blade 2

Blade 2
Blade 1

Blade 2

Blade 1

Blade 2
Blade 1

Blade 1

… HBA Mode)

FC Switch FC Switch NPV NPV

E-Port
NPV Enables Large Scale
N-Port
Blade Server Deployments By:
SAN ƒ Reducing Domain ID usage SAN
E-Port
F-Port
ƒ Addressing switch interop issues
ƒ Simplifying management

Storage Storage

Blade Switch Attribute
FC Switch Mode (E-Port) Deployment Model HBA Mode (N-Port)
None
One per FC Blade Switch # of Domain IDs Used
(Uses Domain ID of Core Switch
Interoperability Issues with
Yes No
Multivendor Core SAN Switch
Level of Management Coordination
Medium Between Server and SAN Low
Administrators
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29
Presentation_ID.scr
SAN Security

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

SAN Design Security Challenges
ƒ SAN design security is often overlooked as an area of concern
Application integrity and security is addressed, but not back-end storage network carrying actual data
SAN extension solutions now push SANs outside datacenter boundaries
ƒ Not all compromises are intentional FC
Accidental breaches can still have the same consequences
ƒ SAN design security is only one part of complete data center solution
Host access security—one-time passwords, auditing, VPNs
Storage security—data-at-rest encryption, LUN security
Data center physical security
Theft
Privilege Escalation/
External Dos
Unintended Privilege
or Other Data
Unauthorized
Intrusion Tampering
Application Connections
FC Tampering (Internal)
FC
FC (Trojans, etc.)
FC
FC
FC

FC
FC SAN FC
FC

FC

FC

LAN
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30
Presentation_ID.scr
SAN Security—Design Considerations
Data Integrity and Secrecy
ƒ Protecting data
SAN Fabric Protocol
Data integrity and encryption Security
in transit or at rest

ƒ Securing against
unauthorized user and
device access
Target
User/device authorization SAN Fabric
and authentication Host

Server and target
access controls
Target Access
ƒ Guarding against Security
malicious management Fabric Access
Security
misconfiguration iSCSI

IP Storage
Management access controls Security
SAN Management (iSCSI/FCIP)
Securing the SAN
Security
management information
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

Intelligent Fabric
Applications

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31
Presentation_ID.scr
Intelligent Storage Applications
Delivered as a Transparent Fabric Service

Application Servers ƒ Extend storage services to
any device in the SAN
ƒ Transparent to applications
ƒ Nondisruptive deployment
No SAN reconfiguration

MSM-18/4 MSM-18/4
No rewiring to insert appliances

ƒ Highly scalable performance
ƒ Automatic load balance
ƒ Reliable, highly available
service
ƒ Wizard-based provisioning
Storage Array

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

Cisco Storage Media Encryption (SME)

Application
ƒ Encrypts storage media
Server (data at rest)
IEEE compliant AES-256 encryption
Name: XYZ
SSN: 1234567890
Amount: $123,456 Integrated as transparent fabric service
Status: Gold
ƒ Transparent Fabric Service
Key Management
Center ƒ Supports heterogeneous storage
SME SME IP arrays, tape devices, and VTLs
ƒ Compresses tape data
ƒ Offers secure, comprehensive key
@!$%!%!%!%%^&
management
*&^%$#&%$#$%*!^
@*%$*^^^^%$@*)
%#*@(*$%%%%#@
ƒ Allows offline media recovery
ƒ Built upon FIPS Level 3 system
Storage Tape architecture
Array Library

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32
Presentation_ID.scr
Cisco Data Mobility Manager (DMM)

Application Servers ƒ Migrates data between storage
arrays for
Technology refreshes
Workload balancing
Storage consolidation

ƒ DMM offers
Online migration of heterogeneous arrays
Data Mobility Manager
Simultaneous migration of multiple LUNs
Unequal size LUN migration
Application Data Rate adjusted migration
I/O Migration
Verification of migrated data
Secure erase
Dual fabric support
Old Array New Array CLI and wizard-based management
with Cisco Fabric Manager
Utilizes Storage Services
Modules (SSM) ƒ Requires no SAN reconfiguration or
rewiring
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

Network Assisted Storage Applications
SAN Tap

ƒ Enables appliance-based
Initiator storage applications without
compromising SAN integrity
Initiator ÅÆ Target I/O ƒ About SAN Tap
SAN MDS delivers a copy of primary
I/O to an appliance
Appliance provides the storage
Copy of Appliance
Primary application
I/O
Examples of applications include
Continuous Data Protection
(CDP), replication, etc.

Target ƒ Key customer benefits
Preserve integrity, availability, and
performance of primary I/O
No service disruption
= SAN Tap
Investment protection
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33
Presentation_ID.scr
Storage Virtualization for
Cisco MDS 9000
ƒ Independent
Control control path Services
Processor ƒ Program the ƒ Volume management
data path ƒ Data migration
ƒ Processes ƒ Point-in-time copies
exceptions

FAIS
ƒ High-performance fast path
SSM ƒ Integrated, HA architecture
ƒ Multiprotocol integration
Data Path
ƒ Comprehensive security
ƒ Troubleshooting and diags

Virtual to Physical Mapping

Data Traffic Control Traffic
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

SecureErase on Offline Storage Arrays
ƒ Configuration
Deploy a SSM
Create VIs
Zone VIs and target ports
Expose target LUNS to VI
Create and start SE sessions

Virtual Initiators on SSM Key Benefits
Cisco Fabric
Mgr
ƒ In-place SecureErase
FC FC No window of vulnerability

ƒ Vendor agnostic
Heterogeneous Storage Arrays
ƒ Compliance
DoD, RCMP, and Gutmann

ƒ Easy to deploy and cost effective
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34
Presentation_ID.scr
Interoperability

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

Standard Fibre Channel Interoperability
Switch Interoperability Is Available Between the Cisco MDS
Platforms and Non-Cisco ‘Standards Compliant’ Switches
ƒ Provides way to redeploy smaller
edge switches
ƒ Both McDATA and Brocade switches
must be in ‘interoperability mode’
On both products this results in TE E
the loss of some functionality
No trunking
No port-based zoning Non-Cisco Switch in
No full zone-set exchanges ‘Interoperability’ Mode
Restricted # of domains and their ID
Cisco Brocade McDATA
ƒ Must also choose fabric timer
values that are the same and 5000- 4000- 1000-
RA_T_O_V
available across all vendors— 100000 ms 120000 ms 120000 ms
must be the same fabric-wide 1000- 1000- 200-
ED_T_O_V
DS_TOV ms 20000 ms 60000 ms
ƒ Must enable ‘interoperability’
mode on MDS 9000 v2.4.1 and v04.01.00 12
v3.0.1a
No loss of functionality on MDS 9000
Enabled per-VSAN—‘Interop mode 1’
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35
Presentation_ID.scr
Third-Party Switch Native Mode
ƒ Enables MDS9000 family to
Cisco MDS 9000 Family
interoperate with legacy fabric
switches in ‘native mode’—
Brocade and McDATA
(SANOS3.1)
Reuse existing legacy fabric switches
as edge devices
No impairment to Cisco fabric—
all advanced services available VSAN 50 VSAN 40

No change required on legacy Legacy Brocade Legacy McDATA
switches—simply connect switches Switches Switches

ƒ Three additional modes Configurable on a VSAN-by-
Interop mode 2 VSAN Basis on MDS 9000
Interop mode 3
Interop mode 4

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

Closing Remarks

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36
Presentation_ID.scr
Closing Remarks—SAN Design
ƒ Simple SAN design
Small port count
No interswitch links
Dual fabric
Minimal SAN/OS feature use

ƒ Scalable SAN design
Core/edge
Collapsed core/edge

ƒ Director class collapsed CORE SAN
Large port count
High availability design
Port Channels, FSPF routing
Multi-SAN/OS feature use

ƒ Core director SAN with blade server and edge switches
Multidomains
Interop
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

BRKSAN-2704: SAN Extension
BRKSAN-2719: From Storage Component Virtualization
to End-to-End Virtualization
BRKSAN-2705: iSCSI SAN Design and Operation
BRKSAN-2702: SAN Migration and Interoperability
BRKSAN-3707: Advanced SAN Design

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37
Presentation_ID.scr
Additional Information

ƒ Cisco Storage Networking
http://www.cisco.com/go/storagenetworking

ƒ Cisco Data Center Networking
http://www.cisco.com/go/datacenter

ƒ Storage Network Industry Association (SNIA)
http://www.snia.org

ƒ Internet Engineering Task Force—IP Storage
http://www.ietf.org/html.charters/ips-charter.html

ƒ ANSI T11—Fibre Channel
http://www.t11.org/index.htm
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

Q and A

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38
Presentation_ID.scr
Recommended Reading

ƒ Continue your Cisco Live
learning experience with further
reading from Cisco Press
ƒ Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

Complete Your Online
Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39
Presentation_ID.scr
BRKSAN-2701
14570_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

© 2006, Cisco Systems, Inc. All rights reserved. 40
Presentation_ID.scr