3.3K views

BRKSAN-2821

Uploaded by jeffgrantinct

Copyright: Attribution Non-Commercial (BY-NC)

Download as PDF, TXT or read online from Scribd

Flag for inappropriate content

save

Jump to Page

You are on page 1of 53

Search inside document

BRKSAN-2821

I/O Consolidation
in the Data Center

BRKSAN-2821

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1
Presentation_ID.scr
Agenda

 Section 1: What is I/O Consolidation
Section 2: Enabling Technologies
Section 3: FCoE (Fibre Channel over Ethernet)
Section 4: I/O Consolidation Use Cases
Challenges
Closing Remarks

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Section 1
What Is I/O
Consolidation

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2
Presentation_ID.scr
What Is I/O Consolidation

 IT organizations operate multiple parallel networks
IP and other LAN protocols over an Ethernet network
SAN over a Fibre Channel network
HPC/IPC over an InfiniBand network

 I/O consolidation supports all three types of traffic onto
a single network
Servers have a common interface adapter that supports
all three types of traffic

IPC: Inter-Process Communication
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

I/O Consolidation in the Network

Processor Processor

Memory Memory

I/O I/O I/O I/O Subsystem
Storage

Storage
LAN
IPC

LAN
IPC

IPC: Inter-Process Communication
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3
Presentation_ID.scr
I/O Consolidation in the Host

 Fewer CNAs (Converged Network Adapters) instead
of NICs, HBAs, and HCAs
Limited number of interfaces for Blade Servers

FC HBA FC Traffic
FC HBA FC Traffic
CNA All Traffic
NIC Enet Traffic Goes over
10 GE
NIC Enet Traffic CNA

NIC Enet Traffic
HCA IPC Traffic
HCA IPC Traffic
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Cabling and I/O Consolidation

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4
Presentation_ID.scr
I/O Consolidation: Benefits to Customers

FC Traffic
FC Traffic FCoE
FCoE SAN A
FCoE

Enet Traffic FCoE SAN B
Enet Traffic

FCoE SAN

Fewer CNAs and Cables Storage Keeps the Same
Management Model as Native FC

Display
FCoE
Adapter

FC Storage FC Switch FCoE Server
Switch

No Storage Gateway Less Power and Cooling

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

Merging the Requirements

IPC
LAN/IP Storage (Inter-Process
Communication)

 Must be Ethernet Must follow the Doesn’t care of
Too much
Fibre Channel the underlying
investment model network, provided
that:
Too many Losing frames is
applications that not an option It is cheap
assume Ethernet It is low latency
It supports APIs
like OFED, RDS,
MPI, sockets

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5
Presentation_ID.scr
Why Has It Not Succeeded Yet?

Previous attempts
Fibre Channel
Never credible
InfiniBand
Not Ethernet
iSCSI
Not Fibre Channel

 Before PCI-Express there was not enough I/O
bandwidth in the servers
It needs to be Ethernet, but…
1 GE didn’t have enough bandwidth
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

PCI-Express

 PCI Express (PCI-E or PCIe)
A computer expansion card interface format designed to
replace PCI, PCI-X, and AGP

PCIe 1.1
Serial links at 2.5 Gbps (2 Gbps at the Datalink)
Speeds from 2 Gbps (1x) to 32 Gbps (16x)
8x is required for 10 GE

 PCIe 2.0 (aka PCIe Gen 2)
Doubles the bandwidth per serial link from 2 Gbit/s to 4 Gbit/s
Spec available since January 2007
Products are making their way into the market
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6
Presentation_ID.scr
10 GE

 2008 will be the year of 10GE
10 GE has enough bandwidth
Merging example
2 x 1 GE Ethernet NIC
1 x 4 Gbps FC (really 3.2 Gbps)
Total 5.2 Gbps over a 10 Gbps link

 CNAs will all be dual-ported for HA
20 Gbps usable bandwidth per server with a single CNA

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Evolution of Ethernet Physical Media
Role of Transport in Enabling These Technologies

Mid-1980s Mid-1990s Early-2000s Late-2000s
SFP+ to SFP+
10 Mb 100 Mb 1SFP+
Gb Cu 10 Gb

UTP Cat 5 UTP Cat 5
SFP Fiber

Power Transceiver
Technology Cable Distance (Each Side) Latency (Link)

SFP+ CU
Twinax 10 m 0.1W 0.1 μs
Copper

SFP+ USR MM OM2 10 m
1W 0
Ultra Short Reach MM OM3 100 m

SFP+ SR MM 62.5 μm 82 m
Short Reach MM 50 μm 300 m 1W 0

Cat6 55 m 8W 2.5 μs
10GBASE-T Cat6a/7 100 m 8W 2.5 μs
Cat6a/7 30 m 4W 1.5 μs

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7
Presentation_ID.scr
Twin-ax Copper Cable
Low power consumption SAN A LAN SAN B

 Low cable cost
Low transceiver latency
Application Server

Low error rate (10–17) Application Server

Application Server

Thinner cable with higher Application Server

16x10 GE Cables

16x10 GE Cables
Application Server

bend radius Application Server

Application Server

Easier to manage cabling Application Server

Application Server
solution reduces Application Server

deployment time Application Server

Application Server

All copper cables are Application Server

Application Server

contained within rack Application Server

Application Server

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

Drivers for 10GE to the Servers
Multicore CPU Architectures Allowing Bigger and
Multiple Workloads on the Same Machine

Server Virtualization Driving the Need for More
Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the
Demand for Higher Network Bandwidth to the Server

Multicore CPUs and Server Virtualization Driving the
Demand for Higher Bandwidth Network Connections

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8
Presentation_ID.scr
Section 1
Enabling Technologies

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Three Challenges + One

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9
Presentation_ID.scr
Why Are Frames Lost?

Transmission
Collision Congestion
Error

 No longer present Very rare in the Most common
in full duplex data center cause
Ethernet
Congestion is a
switch issue, not a
link issue
A full duplex IEEE
802.3 link does not
lose frames
It must be dealt
with in the
bridge/switch
By IEEE 802.1

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Can Ethernet Be Lossless?

Yes, with Ethernet PAUSE Frame

Ethernet Link

STOP PAUSE Queue Full

Switch A Switch B

 Defined in IEEE 802.3—Annex 31B
The PAUSE operation is used to inhibit transmission of
data frames for a specified period of time

 Ethernet PAUSE transforms Ethernet into a
lossless fabric
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10
Presentation_ID.scr
How PAUSE Works

Threshold

A B
PAUSE
Frame
Start
Stop Sending
Frames
FramesforAgain
This
Interval of Time
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

Let’s Compare PAUSE with
FC Buffer to Buffer Credit

Eight credits preagreed

A B

R_RDY

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11
Presentation_ID.scr
How PAUSE Propagates

Threshold

PAUSE PAUSE

S1 S2 S3

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

PAUSE Frame Format
PAUSE Frame
A standard Ethernet frame, not
01:80:C2:00:00:01 tagged

Source Station MAC
EtherType = 0x8808 means MAC
Control Frame
EtherType = 0x8808
Opcode = 0x0001
Pause_Time
Opcode = 0x0101 means PAUSE
Pause_Time is the time the link
needs to remain paused in
Pad
Pause Quanta (512-bits time)
42 Bytes
There is a single Pause_Time for
the whole link
…
CRC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12
Presentation_ID.scr
Why Is PAUSE Not Widely Deployed?

Inconsistent implementations
Standard allows for asymmetric implementations
Easy to fix

 PAUSE applies to the whole links
Single mechanism for all traffic classes

 This may cause “traffic interference”
e.g., Storage traffic paused due to a congestion on IP traffic

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Priority Flow Control (PFC)
a.k.a. PPP (Per Priority Pause)
PFC enables PAUSE functionality per Ethernet priority
IEEE 802.1Q defines eight priorities
Traffic classes are mapped to different priorities:
No traffic interference
IP traffic may be paused while storage traffic is being forwarded
Or, vice versa
Requires independent resources per priority (buffers)
High level of industry support
Cisco distributed proposal
Standard track in IEEE 802.1Qb

16 3 1 12 Bits
EtherType = IEEE 802.1Q Priority CFI VLAN ID

IEEE 802.1Q Tag
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13
Presentation_ID.scr
Priority Flow Control in Action

Transmit Queues Receive Queues
Ethernet Link
One One
Two Two
Three Three
Eight
Four Four Priorities
Five Five
Six STOP PAUSE Six
Seven Seven
Eight Eight

Switch A Switch B

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

PFC Frame Format
Priority Flow Control
Similar to the PAUSE frame
01:80:C2:00:00:01
Opcode = 0x0101 is used to
Source Station MAC
distinguish PFC from PAUSE
EtherType = 0x8808 Class vector indicates for which
Opcode = 0x0101
Class Enable Vector priorities the frame carries valid
Time (Class 0)
Time (Class 1)
Pause information
Time (Class 2)
Time (Class 3) There are eight Time fields,
Time (Class 4)
Time (Class 5)
one per priority
Time (Class 6)
Time (class 7)
Pad
28 Bytes
…
CRC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14
Presentation_ID.scr
Is Lossless Better?

 Frames are TCP relies on Congestion
not dropped losses spreading and
head of line
FC over lossless We can run it on
blocking
Ethernet works a priority where
well we do not enable
Pause

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Is Anything Else Required?

In Order to Build a Deployable I/O Consolidation
Solution, the Following Additional Components
Are Required:

 Discovery protocol (DCBX)
Bandwidth manager
Congestion management

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15
Presentation_ID.scr
Discovery Protocol

DCBX:
Data Center
Bridging eXchange
Data Center
Ethernet Links

DCBX Data Center Ethernet
Links with Partial
Enhancements

DCBX

Legacy Ethernet
Links
Data Center
Ethernet Cloud

XP
Legacy Ethernet
Network

BC
DC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

DCBX

 Hop-by-hop negotiation for:
Priority Flow Control (PFC)
Bandwidth management
Congestion management (BCN/QCN)
Applications
Logical link-down

 Based on LLDP (Link Level Discovery Protocol)
Added reliable transport

 Allows either full configuration or configuration checking
Link partners can choose supported features and willingness to
accept configuration from peer
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16
Presentation_ID.scr
Bandwidth Management

 IEEE 802.1Q defines priorities, but not a simple,
effective, and consistent scheduling mechanism
Products typically implement some form of Deficit
Weighted Round Robin (DWRR)
Configuration and interworking is problematic

 Proposal for HW-efficient, two-level DWRR with strict
priority support
Consistent behavior and configuration across network elements

Standard track in IEEE 802.1Qaz

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Priority Groups
Priority Groups
Are Then
Scheduled
LAN

Priorities Are
Assigned to
Individual SAN
Traffic Classes

IPC Final Link
Behavior
Priority
Groups

First Level of Scheduling
Inside Each Group
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17
Presentation_ID.scr
Goals
BW assignment for each Cannot compromise low-
“Priority Group” latency application due
Example: 40% LAN, to convergence
40% SAN, 20% IPC Allow strict, high priority
scheduling of IPC (and
Should allow multiple traffic equivalent) traffic
classes within a “Priority
Group” Should provide
management infrastructure
Allow these traffic classes to
share BW without hard (MIBs)
configuration Defining scheduling algorithms
Example: VoIP and bulk traffic is too restrictive and not
to share 40% LAN BW necessary
Interoperability for
management is important

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

Example of Link Bandwidth Allocation
Offered Traffic 10 GE Link Realized Traffic Utilization

3 Gbs 3 Gbs 2 Gbs (30%) HPC Traffic (20%)
(30%)

LAN Traffic
(30%) (50%)
3 Gbs 4 Gbs 6 Gbs (40%)

Storage Traffic
(30%) (30%)
3 Gbs 3 Gbs 3 Gbs (30%)

T1 T2 T3 T1 T2 T3

HPC Traffic—Priority Class “High”—20% Guaranteed Bandwidth
LAN Traffic—Priority Class “Medium”—50% Guaranteed Bandwidth
Storage Traffic—Priority Class “Medium-High”—30% Default Bandwidth
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18
Presentation_ID.scr
Congestion Management

 Layer 2, end-to-end congestion management
Standards track in IEEE 802.1Qau
a.k.a. BCN (Backward Congestion Notification) or
QCN (Quantized Congestion Notification)
Switch Switch

Switch

Transmit Queue
Thro
ttle Receive Buffer
Switch Switch

e
ottl
Thr

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

Congestion Management Principles
Move congestion to network edges to avoid congestion
spreading
Use rate-limiters at the edge to shape flows causing
congestion
Tune rate-limiter parameters based on feedback coming from
congestion points
Inspired by:
TCP
AIMD (Additive Increase, Multiplicative Decrease) rate control
TCP window increases linearly in absence of congestion
Decreases exponentially (gets halved) at every congestion
indication (either implicit or explicit)
FCC (Fibre Channel Congestion Control)
A feature on Cisco MDS switches
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19
Presentation_ID.scr
Congestion Point and Reaction Point

 Roles and responsibilities
Reaction Points (RP) shape
traffic entering the network
RP
Congestion Points (CP)
indicate congested state
of queuing points
CP
RP

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

DCB: Data Center Bridging

 Industry consensus term to indicate
Priority flow control
Bandwidth management
Congestion management
Discovery (DCBX)

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20
Presentation_ID.scr
DCE: Data Center Ethernet

 Cisco term used to indicate Cisco switches that
implement the DCB features, plus
Layer 2 multipathing
Fibre Channel over Ethernet

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Layer 2 Multipathing

Increase bandwidth
of L2 networks via
multiple active links

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21
Presentation_ID.scr
L2 Multipathing

 Multiple paths are used, reclaiming network bandwidth
L3 multipathing is common in IP networks
Important when there is limited or no differentiation in
speed between access links and backbone links
Reduces latency
L2 multipathing
Eliminates Spanning Tree from the backbone
No packet flooding
Small forwarding tables

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

Layer 2 Multipathing

Cisco DCE is:
A precursor of TRILL, an IETF project for Layer 2 multipath
Inspired to FSPF (Fibre Channel Shortest Path First)

Cisco DCE
9 Computes topology and forwarding via IS-IS
9 Provides optimal pair-wise unicast forwarding
9 Provides multipathing for unicast and multicast frames
9 Provides seamless interoperability with existing devices

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22
Presentation_ID.scr
FCoE: Fibre Channel over Ethernet

 FCoE is the protocol
used to carry Fibre
SAN LAN SAN
Channel over CEE/DCE
Allows storage
I/O consolidation
FCoE

 It’s in an advanced state
of definition in INCITS
T11 FC-BB-5 WG

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Delayed Drop

 Delayed Drop is a mechanism that:
Allows a switch buffer to virtually extend to previous hop
This reduces packet drop for transient congestions
Is enabled per priority

 It is implemented by asserting PFC on the priority for
a short time
After that time, traffic can flow again or can be dropped

Delayed Drop Is a Means of Using PFC to Mitigate
the Effects of Short-Term Traffic Bursts While
Maintaining Packet Drop for Long-Term Congestion

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23
Presentation_ID.scr
Delay Drop and Proxy Queue
Actual Queue
Traffic
Flow

PAUSE

Proxy Queue

UNPAUSE

 During short-term congestion, both queues drain fast enough that
the actual queue releases the PAUSE on its own
During long-term congestion, the proxy queue fills to its high-water
mark, and it releases the PAUSE; the actual queue begins
dropping packets, and the congestion is managed through higher-
level protocols
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

Section 3
FCoE: Fibre Channel
over Ethernet

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24
Presentation_ID.scr
FCoE: FC over Ethernet

 FCoE is I/O consolidation of FC storage traffic
over Ethernet
FC traffic shares Ethernet links with other traffics
Requires a lossless Ethernet fabric

Ethernet

Fibre Channel
Traffic

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

FCoE Protocol Stack
From a Fibre Channel
standpoint, its FC connectivity
over a new type of cable
called an Ethernet cloud
From an Ethernet standpoint,
it’s yet another ULP (Upper
Layer Protocol) to be
transported

1, 10…Gbps

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25
Presentation_ID.scr
FCoE Benefits

 FCoE benefits are the same of any I/O consolidation
solution
Fewer cables
Both block I/O and Ethernet traffic coexist on same cable
Fewer adapters needed
Overall less power

 Plus additional advantages of being FC
Seamless integration with existing FC SANs
No gateway

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

FCoE Benefits
LAN SAN-A SAN-B LAN SAN-A SAN-B
4 4

2
2

Nearly Half
the Cables

16 Servers Enet FC Total 16 Servers Enet FC Total

Adapters 16 16 32 Adapters 16 0 16
Switches 2 2 4 Switches 2 0 2
Cables 36 36 72 Cables 36 4 40
Mgmt Pts 2 2 4 Mgmt Pts 2 0 2
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26
Presentation_ID.scr
FCoE Is Fibre Channel
FCoE Is Fibre Channel at the Host and Switch Level

Easy to Completely Based
Understand on the FC Model
Same Host-to-Switch and
Same
Operational Model Switch-to-Switch Behavior
of FC
Same Techniques of e.g., in Order Delivery or
Traffic Management FSPF Load Balancing

Same Management WWNs, FC-IDs, Hard/Soft
and Security Models Zoning, DNS, RSCN

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Protocol Organization

FCoE Itself FIP
(FCoE Initialization Protocol)
Is the data plane protocol It is the control plane protocol
It is used to carry most of the It is used to discover the FC
FC frames and all the SCSI entities connected to an
traffic Ethernet cloud
It is also used to login to and
logout from the FC fabric

The Two Protocols Have:
Two different EtherTypes
Two different frame formats
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27
Presentation_ID.scr
FCoE Frame Size
Ethernet
Header 12 Bytes (MAC Addresses) +
4 Bytes (802.1Q Tag)
FCoE
Header
16 Bytes
FC
Header

24 Bytes
Tota
l: 2180
FC Payload

Byte
Up to 2112 Bytes s

4 Bytes

1 Byte (EOF) + 3 Bytes (Padding)
CRC
EOF
FCS
4 Bytes
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

FCoE Frame Format
Destination MAC Address

Source MAC Address
IEEE 802.1Q Tag
ET = FCoE Ver Reserved
Reserved
Reserved
Reserved SOF

Encapsulated FC Frame
(Including FC-CRC)

EOF Reserved
FCS
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28
Presentation_ID.scr
ENode: Simplified Model

 ENode (FCoE Node): a Fibre Channel HBA
implemented within an Ethernet NIC
a.k.a. CNA (Converged Network Adapter)

FC Node

FCoE FCoE

Enet Enet
Port Port

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

FCoE Switch: Simplified Model

 FCF (Fibre Channel Forwarder), the forwarding entity
inside an FCoE switch

FC
Port
FCoE Switch
FC
FCF Port
FCoE
FC
Port

Ethernet Bridge FC
Port

Eth Eth Eth Eth Eth Eth Eth Eth
Port Port Port Port Port Port Port Port

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29
Presentation_ID.scr
FC-BB-5 Terminology

 Unchanged from previous FC standard
VN_Port: Virtual N_Port
VF_Port: Virtual F_Port
VE_Port: Virtual E_Port

 Added to support FCoE
FCoE_LEP (FCoE link endpoint): The data forwarding
component that handles FC frame encapsulation/decapsulation,
and transmission/reception of FCoE frames
FCoE Controller: the entity that implement the FIP protocol

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

ENode: Complete Model

FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s

FC FC FC FC
VN_Port VN_Port VN_Port VN_Port
Entity Entity Entity Entity

FCoE FCoE FCoE FCoE
FCoE_LEP FCoE_LEP FCoE_LEP FCoE_LEP
Entity Entity Entity Entity
FCoE FCoE
Controller Controller

Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port

DISC
F FLOGI
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30
Presentation_ID.scr
Means
FCoE Switch: Complete Model Optional

FC Fabric Interface E_Port E_Port E_Port F_Port F_Port F_Port

FC Switching Element

FC FC FC FC
VE_Port Entity VF_Port Entity VE_Port Entity VF_Port Entity

FCoE FCoE_LEP FCoE FCoE FCoE_LEP FCoE
FCoE_LEP Entity FCoE_LEP Entity FCoE_LEP Entity FCoE_LEP Entity
FCoE_LEP FCoE_LEP
FCoE FCoE
Controller Controller

Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port

Ethernet_Port Ethernet_Port
Lossless Ethernet Lossless Ethernet
Bridging Element Bridging Element
Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

FCoE: Initial Deployment
SAN A SAN B
10 GE
Backbone

VF_Ports

VN_Ports

10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31
Presentation_ID.scr
FCoE: Adding Blade Servers
SAN A SAN B
10 GE
Backbone

VF_Ports

VN_Ports
10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

FCoE: Adding Native FCoE Storage
SAN A SAN B
10 GE
Backbone

VN_Ports

VF_Ports

VN_Ports
10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32
Presentation_ID.scr
FCoE: Adding VE_Ports
SAN A SAN B
10 GE
Backbone

VE_Ports

VF_Ports

VN_Ports
10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

FCoE Addressing and Forwarding

 FCoE frames have:
MAC addresses (hop-by-hop)
FC addresses (end-to-end)

FC Storage FCID 1.1.1
FC Domain 3 FC Domain 1 MAC C
FCID 7.1.1 FC Domain 7 MAC A MAC B
Ethernet Ethernet
Fabric Fabric
FC Fabric FC Fabric

Dest. = MAC B Dest. = MAC C
D_ID = FC-ID (1.1.1) D_ID = FC-ID (1.1.1) Srce. = MAC A Srce. = MAC B
S_ID = FC-ID (7.1.1) S_ID = FC-ID (7.1.1) FCoE
D_ID = FC-ID (1.1.1) D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1) S_ID = FC-ID (7.1.1) Frame
FC Frame FC Frame

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33
Presentation_ID.scr
FCoE MAC Addresses

 VE_Ports and VF_Ports always use MAC addresses
derived from the switch pool
VN_Ports may use two types of MAC addresses
SPMA (Server Provided MAC Addresses)
FPMA (Fabric Provided MAC Addresses)

 MAC Addresses are negotiated in FIP
Initial deployment will use FPMA only

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

The Mapped MAC Addresses

 A dedicated MAC address for each FC-ID
Assigned by the FC fabrics
Consistent with the FC model
OUIs with U/L = 1 (Local addressing), called FC-MAPs
Multiple FC-MAPs may be supported (one per FC fabric)

24 Bits 24 Bits
FC-MAP FC-ID
(ex 02-12-34) 7.8.9

MAC FC-MAP FC-ID
Address (ex 02-12-34) 7.8.9
48 Bits
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34
Presentation_ID.scr
Initial Login Flow Ladder
ENode FCoE Switch
Solicitati
on
Discovery Discovery
Advertisement
FIP:
FCoE
Initialization
Protocol
FLOGI/FDISC FLOGI/FDISC Accept

FC Command FCOE
FC Command Responses Protocol

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

FIP Frame: Contains FIP Operation

Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag
ET = FIP Ver Reserved

Encapsulated FIP Operation
(Self-Describing Length)

PAD to Minimum Length or Mini-Jumbo Length
Ethernet FCS

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35
Presentation_ID.scr
FIP Descriptors (1)
Type = 1 Len = 4 Reserved Priority

Type = 2 Len = 8
MAC Address

Type = 3 Len = 8 Reserved
FC-MAP

Type = 4 Len = 12 Reserved

Switch_Name

Type = 5 Len = 12 Reserved

Fabric_Name

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

FIP Descriptors (2)
Type = 6 Len = 12 Reserved

Port_Name

Type = 7 Len = XX Reserved

FLOGI Request, FLOGI LS_ACC/LS_RJT
NPIV FDISC Request, FDISC LS_ACC/LS_RJT
Fabric LOGO Request, LOGO LS_ACC/LS_RJT
(No SOF/EOF / FC-CRC?)

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36
Presentation_ID.scr
FIP
Multicast Solicitation from H2
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

All-MACs
Solicitation identifies VF_Port capable FCF-MACs
MAC(H2)
with compatible addressing capabilities
Solicitation (FIP)
Other parameters may include ENode’s
[F=0, S=0, MAC(H2),
Capability, Other]
Port_Name for optional duplicate MAC
address detection
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

FIP
Unicast Advertisements from A and B
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

MAC(H2) MAC(H2)

FCF-MAC(A) FCF-MAC(B)
H2’s FCF list: FCF not meeting
Mini-jumbo Mini-jumbo
Advertisement (FIP) Advertisement (FIP) FCF-MAC(A) [J] capability of ENode
[S=1, F=1, Priority, FC-MAP, [S=1, F=1, Priority, FC-MAP, FCF-MAC(B) [J] does not reply
FCF-MAC(A), Switch_Name, FCF-MAC(B), Switch_Name,
Fabric_Name, Capability, Fabric_Name, Capability,
Other] Other]
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37
Presentation_ID.scr
FIP
FLOGI Request
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

FCF-MAC(A) FCF-MAC(B)

MAC (H2) MAC(H2)
Capability agreed
FLOGI Request (FIP) FLOGI Request (FIP)
during discovery
[FC Header, FLOGI data, [FC Header, FLOGI data,
Proposed MAC’(H2)] Proposed MAC’’(H2)]

FIP
FLOGI LS_ACC
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

MAC (H2) MAC(H2)

FCF-MAC(A) FCF-MAC(B)
ENode uses MAC address
in FIP FLOGI LS_ACC as the
FLOGI LS-ACC (FIP) FLOGI LS-ACC (FIP) VN_Port MAC address for the
[FC Header, LS_ACC data, [FC Header, LS_ACC data, FC-ID contained in the FLOGI data
Approved MAC(H2)’] Approved MAC(H2)’’] for subsequent FCoE frames

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2)
MAC (H2)’ FCF-MAC (B)
MAC (H2)’’

FCF-MAC(A) FCF-MAC(B)
All subsequent FCoE frames use
MAC(H2)’ MAC(H2)’’ granted MAC address and assigned
Fibre Channel Frame Fibre Channel Frame FC-ID
(FCoE) (FCoE) FIP frames continue to use MAC(H2)
For SPMA, MAC(H2)’ = MAC(H2)’’ = MAC(H2)
[FC SOF, FC Header, FC [FC SOF, FC Header, FC For FPMA, MAC(H2)’ and MAC(H2)’’ use
Data, FC CRC FC EOF] Data, FC CRC, FC EOF] FC-IDs as low order 24 bits
and FC-MAP for upper 24 bits
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

The Most Asked Question:
Is FCoE Routable?

FCoE

1, 2, 4, (8), 10 Gbps 1, 10 . . . Gbps 10, 20 Gbps

© 2006, Cisco Systems, Inc. All rights reserved. 39
Presentation_ID.scr
Is FCoE Routable?
Most folks mean “Is FCoE IP-routable”
The answer is NO, there is no IP layer in FCoE
This was a design goal to keep FCoE simple
FC-BB-5 contains FCIP that is “IP-routable”

FCoE is FC-routable
FCoE switches may forward FC frames across different
Ethernet clouds
FCoE switches may forward FC frames over the Internet using FCIP

FCoE FCIP FCIP FCoE
IP Cloud

CNA: Converged Network Adapter
LAN HBA CNA
10GbEE

10GbEE
10GbE
10GbE

HBA

Link Link Link
Fibre Channel

Fibre Channel
Ethernet

Ethernet

PCIe PCIe PCIe

Ethernet Fibre Channel Ethernet Fibre Channel
Drivers Drivers Drivers Drivers

© 2006, Cisco Systems, Inc. All rights reserved. 40
Presentation_ID.scr
View from Operating System
Standard drivers
Same management
Operating system sees:
Dual-port, 10 Gigabit
Ethernet adapter
Dual-port, 4 Gbps Fibre
Channel HBAs

Open-FCoE Software
HBA FCoE

HBA Mgmt Plane FCoE Mgmt Plane

Linux Kernel Linux Kernel

File System layers File System layers

SCSI Layer
SCSI Layer

OpenFC Layer

HBA Driver FCoE Layer

Net Device
HBA Ethernet Driver
Ethernet

Fibre Server Ethernet Server

 Open source project
Open-FCoE.org—source Git trees
Or Open-FCoE source package—TBD
Install a Linux Red Hat EL5, Fedora Core 7,
or SuSE 10 distribution
Update kernel to 2.6.23 or later
Install: see Quick Start Guide at open-fcoe.org
Use switch, soft-target, or gateway

Wireshark

 Once known as Ethereal
Captures and displays network traffic
Available from: http://wireshark.org/
Sample trace file
/common/openfc/traces/fcoe-t11.cap

 Use tcpdump to capture
tcpdump –i eth0 –s 0 –w /tmp/fcoe.cap

Screenshots/demo

Section 4
Case Studies

LAN Core
SAN-A SAN-B

Distribution
MDS 9500
4
Cisco Catalyst® 6509

4 x 4G FC 1 x 10 GE STP BLK

POD 1 POD N Access
MDS 9500
Cisco Catalyst 6509

NIC Teaming
Active/Standby

Discrete
1 GE NICs
and
FC HBA
Server Cabinet 1 Server Cabinet N Server Cabinet 1 Server Cabinet N

Top-of-Rack Consolidated I/O
I/O Consolidation at Access

LAN Core
SAN-A SAN-B
8
Distribution
8 MDS 9500
Cisco Catalyst 6509

STP BLK

POD 1 POD N Access
Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

LAN Core
SAN-A SAN-B
8
Distribution
8 MDS 9500
Cisco Catalyst 6509

Ethernet Host Virtualizer
Active/Active

POD 1 POD N Access
Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

Top-of-Rack Consolidated I/O
VSS Support at Aggregation

LAN Core
SAN-A SAN-B
8
VSS Distribution
MDS 9500
8
Cisco Catalyst 6509

VSS Support
at Aggregation

POD 1 POD N Access
Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

4x10 GE
Ports

4x10 GE 6x4 GFC
Ports 6x4 GFC

Nexus 5000

40x10 GE Ports
per Switch

Physical Topology—200 Servers
40 4GFC
SAN-A SAN-B
LAN

4x10 GE
Ports 6x4 GFC

Distribution
MDS 9500
8 Cisco Catalyst 6509

POD 1 POD N Access
Nexus Family

Blade Server
Copper
Pass-Through

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Blade Server 1 Blade Server N Blade Server 1 Blade Server N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93

Blade Servers with
Ethernet Switches
DC Core
MDS MDS
SAN-Core SAN-Core
8

Distribution
MDS 9500
8 Cisco Catalyst 6509

POD 1 POD N Access
Nexus Family

Blade Server
Ethernet-Only
Switch

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Blade Server 1 Blade Server 1 Blade Server 1 Blade Server 1
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 94

2x10 GE
Ports

2x10 GE 4x4 GFC
Ports 4x4 GFC

Nexus 5000

Topology—100 Blade Servers, ToR
LAN SAN-A SAN-B

4x10 GE
Ports

8x4 GFC
8x4 GFC
4x10 GE
Ports

Distribution
MDS 9500
8 Nexus 7000

Layer 2 Multipath

POD 1 POD N Access
Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

Consolidation in the Distribution Layer
LAN Core
SAN-A SAN-B
8

Distribution
MDS 9500
8 Nexus 7000

Layer 2 Multipath

POD 1 POD N Access
Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

Challenges

 FCoE redefines consolidated scenarios
Ethernet switch manufacturers will try to enter the FC
switching market
FC switch manufacturers will try to enter the Ethernet
switching market
HBA manufacturers will try to enter the NIC market
NIC manufacturers will try to enter the HBA market

Deep integration with virtualization will take some time

IEEE 802.3
http://standards.ieee.org/getieee802/802.3.html

Improvements to Ethernet
http://www.nuovasystems.com/EthernetEnhancements-Final.pdf

 IEEE 802.1 activities
http://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf
http://www.ieee802.org/1/files/public/docs2007/new-cm-pelissier-enabling-block-storage-0705-v01.pdf
http://www.ieee802.org/1/files/public/docs2007/au-ko-fabric-convergence-0507.pdf
http://www.ieee802.org/1/pages/802.1au.html
http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf

FCoE
http://www.fcoe.com/
http://www.t11.org/
http://www.open-fcoe.org/
http://www.fibrechannel.org/OVERVIEW/FCIA_SNW_FCoE_WP_Final.pdf

Thank You

 Continue your Cisco Live
learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books

Complete Your Online
Session Evaluation
Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKSAN-2821

Related titles

BRKSAN-2821

Related Interests

Documents Similar To BRKSAN-2821

More From jeffgrantinct

Popular in Science