You are on page 1of 42

Introduction to Storage

Area Networks (SAN)

BRKSAN-1701

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Session Agenda
Main Topics for Today
ƒ Storage Basics
What is it?

ƒ Fibre Channel Basics
What makes it tick?

ƒ Introduction to SAN Design
… and a little bit of the how-to

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Storage Basics

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Storage Basics

ƒ SCSI Primer
ƒ Storage topologies
DAS, NAS, and SAN

ƒ SCSI Transport
Fibre Channel (FC) , FCIP, iSCSI

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

SCSI Primer

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Standards

ƒ SCSI has evolved since it was introduced as SASI in
1979 by Shugart Associates—it was approved as a
standard by ANSI in 1986 and is now referred to as
SCSI-1
ƒ SCSI-2 was approved by X3 in 1990 and by ANSI
in 1994
ƒ SCSI-3 refers to a collection of standards, each of
which defines a very specific part of SCSI: physical
interface, transport interface, command interface,
architecture model, programming interface, etc.
Also known as SCSI Architecture Model 3 (SAM-3)

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

SCSI Architecture Model

SAN

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Small Computer System Interface—SCSI

ƒ SCSI is a standard that defines an interface between
an initiator (usually a computer) and a target (usually
a storage device such as a hard disk)
ƒ Interface refers to connectors, cables, electrical signals,
optical signals and the command protocol that allow
initiators and targets to communicate
ƒ Logical Unit Number (LUN): A 64-bit field within SCSI
that identifies the Logically addressable Unit within a
target SCSI device

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

SCSI Command Protocol

ƒ SCSI command protocol is the de facto standard that
is used extensively in high-performance storage
applications
ƒ The command part of SCSI can be transported over
various types of interconnecting technologies
ƒ A basic understanding of SCSI communications is
beneficial in understanding storage networks.

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
The SCSI I/O Transaction

ƒ The SCSI protocol forms the basis of an I/O transaction
ƒ The channel provides connectivity between
communicating devices in a SCSI transaction
The following shows two sample SCSI exchanges:

Host (Initiator) Disk (Target)
SCSI READ OPERATION
DATA STATUS
DATA DATA READ

SCSI I/O Channel

Host (Initiator) Disk (Target)
SCSI WRITE OPERATION
READY
STATUS DATA WRITE
DATA DATA

SCSI I/O Channel
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

The SCSI I/O Channel—Starting Point
ƒ SCSI I/O Channel provides
half-duplex pipe for SCSI Applications
Half-Duplex
command, data, and status File System I/O Channel

ƒ SCSI I/O channel can be Block Device SI
SC
internal or external to host
SCSI Generic

ƒ Multiple SCSI I/O channels
can exist within host TCP/IP Stack
SCSI

ƒ A network approach can scale NIC Driver Adapter Driver

the I/O channel in many areas
NIC Adapter SCSI Adapter
(length, devices, throughput) SI
SC

SCSI
Initiator
SCSI
Target

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
The Parallel SCSI I/O Channel
Host (Initiator)
ƒ The SCSI Channel is used to transmit SCSI
commands, data, and status
ƒ Multiple devices can exist on the channel
each playing the role of SCSI initiator or SCSI
SCSI
SCSI target
ƒ Most common channel is the basic parallel

I/O Channel
Channel
SCSI bus which can be internal or external Controller
to a host
ƒ Parallel SCSI Specifications:
Up to 25m bus length
Shared channel bandwidth up to 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation
Target 1 Target 2
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Networking the I/O Channel
Host (Initiator)
ƒ Same SCSI protocol carried over a network
transport via serial implementation Initiator

ƒ Transport must not jeopardize SCSI payload SCSI
SCSI
(security, integrity, latency)
ƒ Two primary transports to choose from
I/O Channel
Networked

today, namely IP and Fibre Channel Channel
Controller
ƒ A networked I/O channel allows for
multiple improvements:
Network
Distance limitations greatly increased
Dedicated bandwidth (not shared)
High # of addressable devices
Bandwidth increase (including link bundling)

Target
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel I/O Networking
Host System
ƒ Fibre Channel provides high speed
transport for SCSI payload via Host Buss Initiator

Adapter (HBA)
SCSI
SCSI
ƒ Fibre Channel overcomes many
shortcomings of Parallel I/O including:
Addressing for up to 16 million nodes Fibre Channel
HBA
Loop (shared) and fabric (switched) transport
Host speeds of 100 to 400 MBps (1–4 Gbps)
Fibre Channel
Segments of up to 10km (without extenders) Fabric

Support for multiple protocols

ƒ Combines best attributes of a channel
and a network together

Target
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

Consolidated I/O Networking
ƒ Traditional Ethernet with FCoE
Host
Fibre Channel over Ethernet (Initiator)
FCoE
FCoE
ƒ Consolidated Network Ethernet
Network
Adapter – (CAN)
FCoE CNA
Native Ethernet NIC FCoE
(Target)
Native FC HBA

ƒ Relies on Lossless Ethernet
PAUSE per IEEE 802.1p
SCSI
SCSI
ƒ Able to build Ethernet-based FC
SANs using FCoE arrays Fabric FC HBA
Attached
Host
(Initiator)
Storage
(Target)

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
TCP/IP I/O Networking
ƒ IP access to open iSCSI-Enabled
Host (Initiator)
systems iSCSI and
iSCSI
iSCSI
Fibre Channel storage
ƒ iSCSI driver is loaded onto Ethernet
hosts on Ethernet network iSCSI NIC
(Target)
ƒ Able to consolidate servers
IP
via iSCSI onto existing Network
storage arrays
iSCSI
ƒ Able to build Ethernet-based Gateway
SANs using iSCSI arrays SCSI
SCSI

ƒ Storage can be mapped from FC
Fabric FC HBA
iSCSI to Fibre Channel Attached
Host
(Initiator)
Storage
(Target)
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Storage Topologies

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Storage Topologies

IP
Network SAN

IP Front-End Network SAN Back-End Network
ƒ Host to Host ƒ Host to Storage
ƒ Application to file system ƒ Block I/O to Storage
ƒ Client to Server ƒ Storage to Storage
ƒ NFS, SMB, CIFS, NCP ƒ SCSI, IDE, NTFS, FAT
ƒ NAS, WAFS ƒ SAN
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Direct Attached Storage (DAS)
ƒ Hosts directly access
storage as block-level
devices Clients
ƒ Storage is captive ‘behind’
the server, limited mobility
LAN Application
ƒ Limited scalability due Servers
to limited devices
Win2k Linux Unix Win2k Linux Unix
ƒ No efficient storage
sharing possible
SCSI FC FC
ƒ Costly to scale; complex
to manage

Tape
Direct Attached Storage

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Storage Area Network (SAN)
ƒ Storage is accessed at a
block-level via SCSI Clients

ƒ High performance interconnect
providing high I/O throughput LAN
ƒ Lower TCO relative to direct
attached storage, storage can Servers
be shared
ƒ Limited vendor interoperability Fibre
Channel
SAN
ƒ Complex management
Block
Storage
Devices
Separation of Storage from the Server

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

Network Attached Storage (NAS)
ƒ Storage is accessed at a
file level via NFS or CIFS
ƒ Storage is accessed
over an IP network
ƒ Storage devices can be LAN
shared between servers
ƒ Files can be shared Win2k Linux Unix
between users Generic Generic

ƒ TCP can be tuned to
optimize for file transport Application
Servers
NAS Appliances
or
NAS Head Ends
NAS = Optimized for File I/O
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
iSCSI Storage Network
iSCSI-Enabled
ƒ Storage is accessed at Hosts (Initiators)
iSCSI
a block-level via iSCSI iSCSI
iSCSI
ƒ Good performance via
standard Ethernet NIC iSCSI
Appliance iSCSI
ƒ Enhanced performance (Target)
with TCP Offload
Engine (TOE) IP
Network
ƒ Lower TCO relative to
iSCSI
direct Fibre Channel Gateway
HBA/Fabric
FC
ƒ Standards based Fabric
FC HBA
Attached
Host
(Initiator)
Storage
Pool (Target)
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

DAS, SAN, iSCSI, NAS Comparison
iSCSI iSCSI NAS NAS
DAS SAN
Appliance Gateway Appliance Gateway
Computer System Computer System Computer System Computer System Computer System Computer System

Application Application Application Application Application Application

Host/ File System File System File System File System File System File System

Server Volume Manager Volume Manager Volume Manager
SCSI Device Driver
iSCSI Driver
Volume Manager
SCSI Device Driver
iSCSI Driver
I/O Redirector
NFS/CIFS
I/O Redirector
NFS/CIFS
SCSI Device Driver SCSI Device Driver TCP/IP Stack TCP/IP Stack
TCP/IP stack TCP/IP stack
SCSI Bus Adapter FC HBA NIC NIC NIC NIC

Block I/O File I/O

SAN IP IP IP IP
Storage
Transport NIC
TCP/IP Stack
NIC
TCP/IP Stack
NIC
TCP/IP Stack
NIC
TCP/IP Stack
iSCSI Layer iSCSI Layer File System File System
SCSI FC Bus Adapter FC HBA Device Driver FC HBA

Storage FC Block I/O FC

Media
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
SCSI Transport

Parallel SCSI
Fibre Channel (FC), FCoE, FCIP, iSCSI

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Network stack comparison

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP

TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
The Parallel SCSI I/O Channel
Host (Initiator)
ƒ The SCSI Channel is used to transmit
SCSI commands, data, and status
ƒ Multiple devices can exist on the channel
each playing the role of SCSI initiator or SCSI
SCSI
SCSI target
ƒ Most common channel is the basic

I/O Channel
Channel
parallel SCSI bus which can be internal Controller
or external to a host
ƒ Parallel SCSI Specifications:
Up to 25m bus length
Shared channel bandwidth up to 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation
Target 1 Target 2
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

Fibre Channel Stack

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP

TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
What Is It? Fibre Channel
Channels Networks
ƒ Connection service ƒ Connectionless
ƒ Physical circuits ƒ Logical circuits
ƒ Reliable transfers ƒ Unreliable transfers
ƒ High speed
ƒ Low latency
ƒ Fibre Channel ƒ High connectivity
ƒ Higher latency
ƒ Short distance
ƒ Hardware intense
ƒ Circuit and packet ƒ Longer distance
ƒ Software intense
switched
ƒ Reliable transfers
ƒ High data integrity
ƒ High data rates
ƒ Low latency
ƒ High connectivity
ƒ Long distance

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Fibre Channel Protocol Architecture

ULP (Upper Level Protocol)
SCSI-3 IP FICON
(sample – there are more)

Link FC Single
FC-4 SNMP Mapping SCSI-3 Mapping
Encapsulation Byte Command
(FC-SNMP) (SCSI-FCP)
(FC-LE) Sets

FC-3 Common Services

FC-2 Framing Protocol Fibre Channel FC-AL FC-AL-2 FC-SW-2
Physical & Signaling
FC-1 Encode / Decode Interface 8B/10B Encoding
(FC-PH, FC-PH2,
FC-0 Physical Interface FC-PH3) Copper Optical

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel Mapping
SCSI Architecture to Fibre Channel Mapping

FCP Goes
Here (FC-4)

FC-3
FC-2
FC-AL
FC-1
FC-0

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

Fibre Channel Functions
Structure Is Divided into Five Levels of Functionality
ƒ FC-0
Defines the physical interface characteristics
Signaling rates, cables, connectors, distance capabilities, etc.
ƒ FC-1
Defines how characters are encoded/decoded for transmission
Transmission characters are given desirable characters
ƒ FC-2
Defines how information is transported
Frames, sequences, exchanges, login sessions
ƒ FC-3
Place holder for future functions
ƒ FC-4
Defines how different protocols are mapped to use Fibre Channel
SCSI, IP, Virtual Interface architecture, FICON, others

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel Frame Format
ƒ All FC-2 frames follow the general frame format as shown below
ƒ Idles are ‘Ordered Sets’ used for synchronization and
basic signaling
ƒ SOF—Start-of-Frame, EOF—End-of-Frame

General FC-2 Frame Format

Frame Content

Frame
Idles* SOF Data Field CRC EOF Idles*
Header
(4) (24) (0-2112) (4) (4)

0-528 Transmission Word

*Six idle words (24 bytes) required by TX; two idle words (8 bytes) guaranteed to RX
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Fibre Channel FC-2 Hierarchy
ƒ Multiple exchanges are initiated between initiators (hosts) and
targets (disks)
ƒ Each exchange consists of one or more bi-directional sequences
ƒ Each sequence consists of one or more frames
ƒ For the SCSI3 ULP, each exchange maps to a SCSI command

OX_ID &
RX_ID Exchange

SEQ_ID Sequence Sequence Sequence

SEQ_CNT Frame Frame Frame
Frame
Fields ULP Information Unit

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Example: Fibre Channel Exchange
ƒ One SCSI command is issued—a single exchange is
created consisting of multiple sequences
ƒ Each exchange is identified with unique OX_ID/RX_ID
ƒ Each sequence is identified with unique SEQ_ID

Fibre Channel Fabric
Initiator
Data Returned in
Sequence Multiple Sequences
#1 Target
SEQ#1 , CN
T#1
Read Command Issued
New Exchange Issued
New Sequence Issued
SEQ#2,
CNT#2
CNT#2
Exchange
SEQ#3,
CNT#1 CNT#2
Sequence #2 SEQ#2, SEQ#4,
CNT#1 CNT#2
SEQ#3, SEQ#5,
Sequence #3 SEQ#4,
CNT#1
Sequence #4 SEQ#5,
CNT#1 Status Returned in
Sequence #5 Additional Sequence
CNT#1
SEQ#6,

Sequence #6

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

Fibre Channel over Ethernet Stack

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP

TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
FCoE: FC over Ethernet

ƒ FCoE is I/O consolidation of FC storage traffic over
Ethernet
FC traffic shares Ethernet links with other traffics
Requires a lossless Ethernet fabric

Ethernet

Fibre
Channel
Traffic

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

Storage Protocols—FCoE
FCoE Frame Format
ƒ From a Fibre Channel standpoint it’s FC connectivity
over a new type of cable called an Ethernet cloud
ƒ From an Ethernet standpoints it’s yet another ULP
(Upper Layer Protocol) to be transported
ƒ FC frame not changed

FCoE
FC
SCSI
Data

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel over IP Stack

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP

TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

Storage Protocols—FCIP
FCIP Frame Format
ƒ Primary use is for Storage-to-Storage connectivity via an IP
WAN/MAN
ƒ The FCIP Link carries encapsulated fibre channel traffic between
Link End Points over an IP network by using TCP on port 3225
ƒ The result is a virtual Inter Switch Link (ISL) between FC Fabrics
ƒ FC frame not changed

IP
TCP
FCIP
FC
SCSI
Data

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Internet SCSI Stack

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP

TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Storage Protocols - iSCSI
iSCSI Frame Format
ƒ Primary use is for Host-to-Storage connectivity via an
IP LAN
ƒ SCSI command and data are encapsulated into iSCSI
by adding a special header
ƒ iSCSI data is encapsulated into a TCP packet
ƒ IP is the transport protocol

IP
TCP
iSCSI
SCSI
Data

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Network stack comparison

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP

TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

SCSI Transport Summary
ƒ Parallel SCSI—Legacy host to storage bus topology
ƒ Fibre Channel—Robust serial transport network for SCSI
Host-to-Storage Connectivity
Primary transport for Storage Area Networks
ƒ FCoE – FC-2 encapsulation in Ethernet
Host-to-SAN Connectivity
Typically used for unified server I/O
ƒ FCIP—FC-2 encapsulation in TCP/IP
SAN-to-SAN Connectivity
Typically used in the WAN/MAN
ƒ iSCSI—TCP/IP transport for SCSI protocol
Host-to-SAN Connectivity
Typically used in the LAN
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel Basics

Fibre Channel Physical Attributes
Fibre Channel Operational Characteristics

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Physical

The main physical objects in Fibre Channel are:
ƒ Ports
ƒ Links
ƒ Nodes
ƒ Fabric

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel Port Types
ƒ ‘N’ port: Node ports used to connect devices to N N

switched fabric or point to point configurations
ƒ ‘F’ port: Fabric ports residing on switches
connecting ‘N’ port devices
N F

ƒ ‘L’ port: Loop ports are used in arbitrated loop
configurations to build networks without FC
switches; these ports often also have ‘N’ port
capabilities and are called ‘NL’ ports NL FL

ƒ ‘E’ port: Expansion ports are essentially
trunk ports used to connect two Fibre L L

Channel switches
ƒ ‘GL’ port: A generic port capable of operating
as either an ‘E’ or ‘F’ port; it’s also capable of E E

acting in an ‘L’ port capacity; Auto Discovery

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

Port Type Examples

Fabric Switch

Node NL_Port FL_Port E_Port E_Port

G_Port F_Port N_Port Node
Node NL_Port

G_Port F_Port N_Port Node
Node NL_Port

G_Port F_Port N_Port Node

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
N_Port

Device
Host
Host/ Device
Interface

FC
N_Port

Can Be More than One Serial Data Out Serial Data In
N_Port on the Device

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

Inter-Switch Link (ISL)

ƒ The interconnection between switches is called the
Inter-Switch Link (ISL)
E_Port to E_Port

ƒ Supports all classes of service
Class 1, 2, 3, and a special Class F

ƒ FC-PH permits consecutive frames of a sequence to be
routed over different ISL links for maximum throughput

E E

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel Link

ƒ A link consists of
Two unidirectional “fibers” transmitting in opposite directions
May be either:
Optical fiber or Copper

ƒ Transmitters may be:
Long wave laser
Short wave laser
LED
Electrical

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

Node
ƒ The equipment which contains one or more
ƒ N_Port or NL_Port (topology dependent)
ƒ May be:
Computer (HBA)
Controller (Port on Disk Subsystem)
Device (SCSI FC to Parallel converter)
ƒ Is not a switch fabric device

Controller

N_Port N_Port N_Port N_Port

Link Link Link Link
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
FC Communications Model

ƒ Point to point
ƒ N_Port to N_Port
ƒ Flow controlled
ƒ Acknowledged Node Node

Host Host

TX RX
N_Port
FC FC

RX TX

Link

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Fibre Channel Fabric

ƒ The entity which interconnects N_Ports
ƒ Provides routing based on destination address
ƒ Fabric may be:
Point to point: No routing required
Arbitrated loop: Routing is distributed across ports
Switched: Routing provided by switch

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Arbitrated Loop
NL_Node
“A”

Link

NL_Node
“B”

Communications Model—Source to Destination Based on
Address Routing Distributed in the NL_Ports on the Loop

A B Still Pt. to Pt. Communication
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

Switched Fabric
N_Port N_Port N_Port N_Port N_Port N_Port

A B

Switch

Fabric
Communications Model—Source to Destination Based on
Address Routing through the Fabric; Still a Point to Point
Connection; FSPF routing required when more then two
switches make up the Fabric
A B
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
FC Operational
Characteristics

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

Operational

The Main Operational Subsystems in Fibre Channel are:
ƒ Worldwide Names
ƒ Fabric Addressing
ƒ Directory Server
ƒ Fabric Controller
ƒ Principle Switch
ƒ Fabric Routing
ƒ Zoning
ƒ Virtual SANs

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Worldwide Names

ƒ Each switch element is assigned a WWN at time
of manufacture
ƒ Each switch port is assigned a WWN at the
time of manufacture
ƒ During FLOGI the switch identifies the WWN
in the service parameters of the accept frame
and assigns a Fibre Channel ID (FCID)
ƒ These address assignments can then correlate
each fabric port with the switch element

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

Fabric Addressing

ƒ The 24-bit FCID address is partitioned into three fields
Device
Area
Domain

ƒ This partitioning helps speed up routing
ƒ Switch element assigns the address to N_Ports
ƒ Address portioning is transparent to N_Ports

8 bits 8 bits 8 bits
Switch Topology
Switch Domain Area Device

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Directory Server

ƒ Repository of information regarding the components
that make up the Fibre Channel network
ƒ Located at address ‘FF FF FC’ (some readings call this
the name server)
ƒ Components can register their characteristics with the
directory server
ƒ An N_Port can query the directory server for specific
information
Query can be the address identifier, WWN and volume names
for all SCSI targets

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

Fabric Controller

ƒ Each switch has a fabric controller
ƒ Assigned address ‘FF FF FD’
Every fabric controller in the fabric has the same address
It is the N_Port within the switch
Responsible for managing fabric, initialization, routing, setup
and teardown of Class-1 connections

ƒ Responsible to receive request and generate
responses for the switch fabric
Information must be consistent independent of which fabric
controller responds to a request

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fabric Configuration: PS Selection
ƒ A principal switch shall be selected whenever at least one
inter-switch link (a link between two E_Port) is established
ƒ The selection process chooses a principal switch, which
is then designated to assign domain identifier to all the
switches in the fabric, and any who join later the fabric
later on
ƒ The principal switch selection can be triggered by anyone
of the following events
Switch Boot and Exchange Fabric Parameters (EFP)
Build Fabric (BF)
Reconfigure Fabric (RCF)

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

Fabric Configuration
Disruptive/Non-Disruptive
ƒ One of the following three conditions can trigger BF
(non-disruptive) or RCF (disruptive)
Two disjoint fabrics are combined together
A principal ISL fails (upstream or downstream)
A switch with Domain_ID request for another Domain_ID

ƒ Whenever a switch receives a BF/RCF, the switch starts
F_S_TOV timer and enters the BF/RCF state; it forwards
BF/RCF out of all E_ports except the incoming port (only
once) and wait for the timer to expire
ƒ When the timer expires, BF/RCF propagation state is left
and principal switch selection begins
ƒ BF is not a disruptive process
ƒ RCF is a disruptive process

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fabric Routing: FSPF

ƒ FSPF routes traffic based on destination domain ID
ƒ For FSPF a domain ID identifies a single switch
This limits the max number of switches that can support
in the Fabric to 239 when FSPF is supported

ƒ FSPF performs hop-by-hop routing
ƒ FSPF uses total cost as the metric to determine most
efficient path
ƒ FSPF supports hierarchical path selection
Provides the scalable routing tables in large topologies

ƒ Static routes can be applied

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

Fabric Routing: FSPF

ƒ Everyone says HELLO to their neighbor, on all
initialized ISLs
ƒ The neighbors say HELLO back, unless they are dead
ƒ When the HELLO packet is received with both
originator and recipient domain id, the two way
communication is done and:
The ISL is active
The ISL may be available as a two-way path for frames

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Fibre Channel Fabric Zoning
ƒ Zoning operation
Zone members “see” only other members of the zone
Zones can be configured dynamically based on WWN
Devices can be members of more than one zone
FC-AL zoning allows the creation of private loops on a single hub
Switched fabric zoning can take place at the port or device level
Based on physical switch port
Based on device WWN SAN
Disk2
Based on LUN ID Disk3
Disk1
ZoneA Host1

ƒ Benefits ZoneC

Secured device access Disk4 Host2
ZoneB
Allows operating system co-existence
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

Zoning—Enforcement
ƒ Zoning is used to control access in a SAN
ƒ Soft zoning
Enforced by name server query responses
Name server sends membership list to N_Port
N-port accesses members only

ƒ Hard zoning
Enforced by hardware (forwarding ASIC) at wire speed pWWN, fWWN, FC_ID, FC_Alias

Soft Zone Hard Zone
Host Host
FC FC
Zone-1 Zone-1
Array Array

MDS MDS Zone-2 MDS MDS Zone-2
Host Host
Array FC Array FC
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Introducing Virtual SANs (VSANs)
A Virtual SAN (VSAN) Provides
Cisco MDS 9000
a Method to Allocate Ports Family with VSAN Service
within a Physical Fabric to
Create Virtual Fabrics
ƒ Analogous to VLANs in Ethernet
ƒ Virtual fabrics created from larger cost- Physical SAN
effective redundant physical fabric Islands Are
Virtualized onto
ƒ Reduces wasted ports of island approach Common SAN
Infrastructure
ƒ Fabric events are isolated per VSAN—
maintains isolation for HA (i.e. RSCNs)
ƒ Hardware-based isolation—traffic is
explicitly tagged across interswitch links
with VSAN membership info
ƒ Statistics can be gathered per VSAN

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

Virtualizing the Fabric—The Full Solution
To Build a Cost-Saving Fabric Virtual Fabric Service Model
Virtualization Solution, Seven
Intervirtual Fabric Routing
Key Services Are Required: Virtualized Fabric Management
1. Virtual fabric attachment—the ability to assign Virtualized Fabric Security Policies
virtual fabric membership at the port level
Virtualized Fabric Diagnostics
2. Multiprotocol extensions—the ability to extend virtual
Virtualized Fabric Services
fabric service to iSCSI, FCIP, FICON, etc.
Multiprotocol Transport Extensions
3. Virtual fabric services—the ability to create fabric
services per virtual fabric (routing, zones, RSCNs, Virtualized Fabric Attachment
QoS, etc.)
4. Virtual fabric diagnostics—the ability to troubleshoot
per virtual fabric problems
5. Virtual fabric security—the ability to define
separate security policies per virtual fabric
6. Virtual fabric management—the ability to map ISL
and manage virtual fabrics independently
7. Interfabric routing—the ability to provide MDS MDS
9000 9000
connectivity across virtual fabrics—without
Family Family
merging the fabrics
Full Service End-to-End Virtual
Fabric Implementation
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
VSANs and Zones—Complimentary
Virtual SANs and Fabric Zoning
Are Very Complimentary
ƒ Hierarchical relationship— Relationship of VSANs to Zones
First assign physical ports to VSANs Physical Topology
Then configure independent zones VSAN 2
per VSAN
Disk2 Disk3
ƒ VSANs divide the physical Host1 Disk1
ZoneA
infrastructure ZoneC
ƒ Zones provide added security and Disk4 Host2
allow sharing of device ports ZoneB

ƒ VSANs provide traffic statistics
VSAN 3 ZoneD
ƒ VSANs only changed when ports
Host4
needed per virtual fabric
ZoneA
ƒ Zones can change frequently
Host3 Disk5
(e.g. backup) Disk6
ƒ Ports are added/removed non-
disruptively to VSANs
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

ISL PortChanneling
A PortChannel Is a Logical Bundling of Identical Links
ƒ Criteria for forming a PortChannel
Same speed links
Same modes (auto, E, etc.) and states
Between same two switches
Same VSAN membership
E.g. 4 Gbps
ƒ Treated as one logical ISL by upper layer PortChannel
protocols (FSPF) (Two x
2Gbps)
ƒ Can use up to 16 links in a PortChannel (32Gbps max)
ƒ Can be formed from any ports on any modules— E.g. 8 Gbps
PortChannel
HA enabled (Four x
ƒ Exchange-based in-order load balancing 2Gbps)
Mode 1: Based on src/dst FC_IDs
Mode 2: Based on src/dst FC_ID/OX_ID
ƒ Much faster recovery than FSPF-based balancing
ƒ Given logical interface name with aggregated bandwidth
and derived routing metric
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
PortChannel vs. Trunking
ƒ ISL = inter-switch link
ƒ PortChannel = E_Ports and ISLs
ƒ Trunk = ISLs that support VSANs
ƒ Trunking = TE_Ports and EISLs

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

Fibre Channel Basics Summary

ƒ Fibre Channel is a very robust, hierarchical standard
ƒ Fibre Channel utilizes a Point-to-Point communications
model irrespective of the topology
ƒ Fibre Channel includes a full set of services for naming,
addressing, building, and managing fabrics
ƒ Fibre Channel utilizes FSPF, an OSPF like routing
protocol to route traffic
ƒ Fibre Channel Zoning is a method of logically grouping
devices within a given fabric

We Have Only Scratched the Surface
of FC Protocol Here!
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Introduction to
SAN Design

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

Type of SAN Networks

ƒ Two primary designs for SANs; Core-Edge and
Collapsed Core-Edge
ƒ Each design provides different advantages
ƒ SAN still driven by oversubscription; host to disk, ISL,
chassis
ƒ As SANs increase in size, some modification to each
model may be required
ƒ SAN network services need to be considered in SAN
design

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Core-Edge

ƒ Traditional SAN design for growing
SANs
ƒ High density directors in core and A B
fabric switches, directors or blade
switches on edge
ƒ Predictable performance A B
A B

A B

ƒ Scalable growth up to core and ISL
capacity

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

Collapsed Core-Edge

ƒ SAN design to take advantage of
high density directors
A B
ƒ Most traffic localized, reducing
number of ISLs
A B

ƒ Oversubscription primarily in
chassis and linecards A B

ƒ Potential to scale further than
traditional Core-Edge design

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Additional Information

ƒ Cisco Storage Networking
http://www.cisco.com/go/storagenetworking

ƒ Cisco Data Center Networking
http://www.cisco.com/go/datacenter

ƒ Storage Network Industry Association (SNIA)
http://www.snia.org

ƒ Internet Engineering Task Force—IP Storage
http://www.ietf.org/html.charters/ips-charter.html

ƒ ANSI T11—Fibre Channel
http://www.t11.org/index.htm
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

Q and A

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
Recommended Reading

ƒ Continue your Cisco Live
learning experience with further
reading from Cisco Press®
ƒ Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

Complete Your Online
Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes; winners announced daily your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center Solutions or visit
www.cisco-live.com

BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr