3.3K views

BRKSAN-2821

Uploaded by jeffgrantinct

Copyright: Attribution Non-Commercial (BY-NC)

Download as PDF, TXT or read online from Scribd

Flag for inappropriate content

save

Related titles

Jump to Page

You are on page 1of 53

Search inside document

BRKSAN-2821 14572_05_2008_c1

Cisco Public

I/O Consolidation in the Data Center

BRKSAN-2821

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Agenda
Section 1: What is I/O Consolidation Section 2: Enabling Technologies Section 3: FCoE (Fibre Channel over Ethernet) Section 4: I/O Consolidation Use Cases Challenges Closing Remarks

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Section 1 What Is I/O Consolidation

BRKSAN-2821 14572_05_2008_c1

Cisco Public

What Is I/O Consolidation
IT organizations operate multiple parallel networks
IP and other LAN protocols over an Ethernet network SAN over a Fibre Channel network HPC/IPC over an InfiniBand network

I/O consolidation supports all three types of traffic onto a single network Servers have a common interface adapter that supports all three types of traffic

IPC: Inter-Process Communication
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

I/O Consolidation in the Network

Processor Memory

I/O
Storage

I/O

I/O Subsystem
Storage

LAN

IPC

IPC: Inter-Process Communication
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

LAN

IPC

I/O Consolidation in the Host
Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs Limited number of interfaces for Blade Servers
FC HBA FC HBA NIC NIC NIC HCA HCA
BRKSAN-2821 14572_05_2008_c1

FC Traffic FC Traffic
CNA

Enet Traffic Enet Traffic Enet Traffic IPC Traffic IPC Traffic
Cisco Public

CNA

All Traffic Goes over 10 GE

Cabling and I/O Consolidation

BRKSAN-2821 14572_05_2008_c1

Cisco Public

I/O Consolidation: Benefits to Customers
FC Traffic FC Traffic

FCoE FCoE

FCoE

SAN A SAN B

Enet Traffic Enet Traffic

FCoE

FCoE SAN

Fewer CNAs and Cables

Storage Keeps the Same Management Model as Native FC

Display FCoE Adapter

FC Storage

FC Switch

FCoE Switch

Server

No Storage Gateway
BRKSAN-2821 14572_05_2008_c1

Less Power and Cooling

Cisco Public

Merging the Requirements
LAN/IP
Must be Ethernet
Too much investment Too many applications that assume Ethernet

Storage
Must follow the Fibre Channel model Losing frames is not an option

IPC
(Inter-Process Communication)

Doesn’t care of the underlying network, provided that:
It is cheap It is low latency It supports APIs like OFED, RDS, MPI, sockets

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Why Has It Not Succeeded Yet?
Previous attempts
Fibre Channel Never credible InfiniBand Not Ethernet iSCSI Not Fibre Channel

Before PCI-Express there was not enough I/O bandwidth in the servers It needs to be Ethernet, but…
1 GE didn’t have enough bandwidth
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

PCI-Express
PCI Express (PCI-E or PCIe)
A computer expansion card interface format designed to replace PCI, PCI-X, and AGP

PCIe 1.1
Serial links at 2.5 Gbps (2 Gbps at the Datalink) Speeds from 2 Gbps (1x) to 32 Gbps (16x) 8x is required for 10 GE

PCIe 2.0 (aka PCIe Gen 2)
Doubles the bandwidth per serial link from 2 Gbit/s to 4 Gbit/s Spec available since January 2007 Products are making their way into the market
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

10 GE
2008 will be the year of 10GE 10 GE has enough bandwidth Merging example
2 x 1 GE Ethernet NIC 1 x 4 Gbps FC (really 3.2 Gbps) Total 5.2 Gbps over a 10 Gbps link

CNAs will all be dual-ported for HA
20 Gbps usable bandwidth per server with a single CNA

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Evolution of Ethernet Physical Media
Role of Transport in Enabling These Technologies
Mid-1980s 10 Mb Mid-1990s 100 Mb
UTP Cat 5

SFP+ to SFP+

Early-2000s

Late-2000s 10 Gb

SFP+ 1 Gb Cu
UTP Cat 5 SFP Fiber

Technology SFP+ CU Copper SFP+ USR
Ultra Short Reach

Cable Twinax MM OM2 MM OM3 MM 62.5 μm MM 50 μm
Cat6 Cat6a/7 Cat6a/7

Distance 10 m 10 m 100 m 82 m 300 m
55 m 100 m 30 m

Power (Each Side) 0.1W

Transceiver Latency (Link) 0.1 μs

0 0
2.5 μs 2.5 μs 1.5 μs

SFP+ SR
Short Reach

1W
8W 8W 4W

10GBASE-T

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Twin-ax Copper Cable
Low power consumption Low cable cost Low transceiver latency
Application Server

SAN A

LAN

SAN B

Low error rate (10–17)
16x10 GE Cables

Application Server Application Server

Application Server Application Server Application Server Application Server Application Server Application Server Application Server Application Server

Easier to manage cabling solution reduces deployment time All copper cables are contained within rack
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Application Server Application Server Application Server Application Server

16x10 GE Cables

Thinner cable with higher bend radius

Application Server

Drivers for 10GE to the Servers
Multicore CPU Architectures Allowing Bigger and Multiple Workloads on the Same Machine

Server Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server

Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections
BRKSAN-2821 14572_05_2008_c1

Cisco Public

Section 1 Enabling Technologies

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Three Challenges + One

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Why Are Frames Lost?
Collision
No longer present in full duplex Ethernet

Transmission Error
Very rare in the data center

Congestion
Most common cause Congestion is a switch issue, not a link issue
A full duplex IEEE 802.3 link does not lose frames

It must be dealt with in the bridge/switch
By IEEE 802.1

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Can Ethernet Be Lossless?
Yes, with Ethernet PAUSE Frame
Ethernet Link

STOP
Switch A

PAUSE

Queue Full
Switch B

Defined in IEEE 802.3—Annex 31B
The PAUSE operation is used to inhibit transmission of data frames for a specified period of time

Ethernet PAUSE transforms Ethernet into a lossless fabric
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

How PAUSE Works
Threshold

A PAUSE Frame Start Sending Stop Frames for This Frames Again Interval of Time
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Let’s Compare PAUSE with FC Buffer to Buffer Credit
Eight credits preagreed

A
R_RDY

BRKSAN-2821 14572_05_2008_c1

Cisco Public

How PAUSE Propagates
Threshold

PAUSE

BRKSAN-2821 14572_05_2008_c1

Cisco Public

PAUSE Frame Format
PAUSE Frame
01:80:C2:00:00:01

A standard Ethernet frame, not tagged EtherType = 0x8808 means MAC Control Frame Opcode = 0x0101 means PAUSE Pause_Time is the time the link needs to remain paused in Pause Quanta (512-bits time) There is a single Pause_Time for the whole link

Source Station MAC EtherType = 0x8808 Opcode = 0x0001 Pause_Time

Pad 42 Bytes

…
CRC

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Why Is PAUSE Not Widely Deployed?
Inconsistent implementations
Standard allows for asymmetric implementations Easy to fix

PAUSE applies to the whole links
Single mechanism for all traffic classes

This may cause “traffic interference”
e.g., Storage traffic paused due to a congestion on IP traffic

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Priority Flow Control (PFC)
a.k.a. PPP (Per Priority Pause) PFC enables PAUSE functionality per Ethernet priority
IEEE 802.1Q defines eight priorities Traffic classes are mapped to different priorities: No traffic interference IP traffic may be paused while storage traffic is being forwarded Or, vice versa Requires independent resources per priority (buffers)

High level of industry support
Cisco distributed proposal Standard track in IEEE 802.1Qb

16
EtherType = IEEE 802.1Q

12 Bits
VLAN ID

Priority CFI

IEEE 802.1Q Tag
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Priority Flow Control in Action
Transmit Queues Ethernet Link
One Two Three Four Five Six Seven Eight
STOP PAUSE

Receive Queues
One Two Three Four Five Six Seven Eight

Eight Priorities

Switch A
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Switch B
27

PFC Frame Format
Priority Flow Control
01:80:C2:00:00:01

Similar to the PAUSE frame Opcode = 0x0101 is used to distinguish PFC from PAUSE Class vector indicates for which priorities the frame carries valid Pause information There are eight Time fields, one per priority

Source Station MAC EtherType = 0x8808
Opcode = 0x0101 Class Enable Vector Time (Class 0) Time (Class 1) Time (Class 2) Time (Class 3) Time (Class 4) Time (Class 5) Time (Class 6) Time (class 7)

Pad 28 Bytes

…

CRC

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Is Lossless Better?

Frames are not dropped FC over lossless Ethernet works well

TCP relies on losses We can run it on a priority where we do not enable Pause

Congestion spreading and head of line blocking

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Is Anything Else Required?
In Order to Build a Deployable I/O Consolidation Solution, the Following Additional Components Are Required:

Discovery protocol (DCBX) Bandwidth manager Congestion management

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Discovery Protocol
DCBX: Data Center Bridging eXchange
Data Center Ethernet Links DCBX Data Center Ethernet Links with Partial Enhancements

DCBX

Legacy Ethernet Links Legacy Ethernet Network
XP

Data Center Ethernet Cloud
BC DC

BRKSAN-2821 14572_05_2008_c1

Cisco Public

DCBX
Hop-by-hop negotiation for:
Priority Flow Control (PFC) Bandwidth management Congestion management (BCN/QCN) Applications Logical link-down

Based on LLDP (Link Level Discovery Protocol)
Added reliable transport

Allows either full configuration or configuration checking
Link partners can choose supported features and willingness to accept configuration from peer
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Bandwidth Management
IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism Products typically implement some form of Deficit Weighted Round Robin (DWRR)
Configuration and interworking is problematic

Proposal for HW-efficient, two-level DWRR with strict priority support
Consistent behavior and configuration across network elements

Standard track in IEEE 802.1Qaz

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Priority Groups
Priority Groups Are Then Scheduled LAN

Priorities Are Assigned to Individual Traffic Classes

SAN

IPC Priority Groups First Level of Scheduling Inside Each Group
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Final Link Behavior

Goals
BW assignment for each “Priority Group”
Example: 40% LAN, 40% SAN, 20% IPC

Cannot compromise lowlatency application due to convergence
Allow strict, high priority scheduling of IPC (and equivalent) traffic

Should allow multiple traffic classes within a “Priority Group”
Allow these traffic classes to share BW without hard configuration Example: VoIP and bulk traffic to share 40% LAN BW

Should provide management infrastructure (MIBs)
Defining scheduling algorithms is too restrictive and not necessary Interoperability for management is important

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Example of Link Bandwidth Allocation
Offered Traffic
3 Gbs 3 Gbs 2 Gbs

10 GE Link Realized Traffic Utilization (30%) HPC Traffic (30%) LAN Traffic (40%) (20%)

3 Gbs

4 Gbs

6 Gbs

(30%)

(50%)

3 Gbs

(30%)

Storage Traffic (30%) T2

(30%)

HPC Traffic—Priority Class “High”—20% Guaranteed Bandwidth LAN Traffic—Priority Class “Medium”—50% Guaranteed Bandwidth Storage Traffic—Priority Class “Medium-High”—30% Default Bandwidth
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Congestion Management
Layer 2, end-to-end congestion management Standards track in IEEE 802.1Qau a.k.a. BCN (Backward Congestion Notification) or QCN (Quantized Congestion Notification)
Switch Switch

Switch Transmit Queue Switch Switch

Thro ttle

Receive Buffer

e ottl Thr

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Congestion Management Principles
Move congestion to network edges to avoid congestion spreading Use rate-limiters at the edge to shape flows causing congestion
Tune rate-limiter parameters based on feedback coming from congestion points

Inspired by:
TCP AIMD (Additive Increase, Multiplicative Decrease) rate control TCP window increases linearly in absence of congestion Decreases exponentially (gets halved) at every congestion indication (either implicit or explicit) FCC (Fibre Channel Congestion Control) A feature on Cisco MDS switches
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Congestion Point and Reaction Point
Roles and responsibilities
Reaction Points (RP) shape traffic entering the network Congestion Points (CP) indicate congested state of queuing points
RP

BRKSAN-2821 14572_05_2008_c1

Cisco Public

DCB: Data Center Bridging
Industry consensus term to indicate
Priority flow control Bandwidth management Congestion management Discovery (DCBX)

BRKSAN-2821 14572_05_2008_c1

Cisco Public

DCE: Data Center Ethernet
Cisco term used to indicate Cisco switches that implement the DCB features, plus
Layer 2 multipathing Fibre Channel over Ethernet

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Layer 2 Multipathing
Increase bandwidth of L2 networks via multiple active links

BRKSAN-2821 14572_05_2008_c1

Cisco Public

L2 Multipathing
Multiple paths are used, reclaiming network bandwidth L3 multipathing is common in IP networks Important when there is limited or no differentiation in speed between access links and backbone links Reduces latency L2 multipathing
Eliminates Spanning Tree from the backbone No packet flooding Small forwarding tables

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Layer 2 Multipathing
Cisco DCE is:
A precursor of TRILL, an IETF project for Layer 2 multipath Inspired to FSPF (Fibre Channel Shortest Path First)

Cisco DCE
Computes topology and forwarding via IS-IS Provides optimal pair-wise unicast forwarding Provides multipathing for unicast and multicast frames Provides seamless interoperability with existing devices

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE: Fibre Channel over Ethernet
FCoE is the protocol used to carry Fibre Channel over CEE/DCE
Allows storage I/O consolidation
FCoE

SAN

LAN

SAN

It’s in an advanced state of definition in INCITS T11 FC-BB-5 WG

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Delayed Drop
Delayed Drop is a mechanism that:
Allows a switch buffer to virtually extend to previous hop This reduces packet drop for transient congestions Is enabled per priority

It is implemented by asserting PFC on the priority for a short time
After that time, traffic can flow again or can be dropped Delayed Drop Is a Means of Using PFC to Mitigate the Effects of Short-Term Traffic Bursts While Maintaining Packet Drop for Long-Term Congestion

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Delay Drop and Proxy Queue
Actual Queue Traffic Flow
PAUSE

Proxy Queue

UNPAUSE

During short-term congestion, both queues drain fast enough that the actual queue releases the PAUSE on its own During long-term congestion, the proxy queue fills to its high-water mark, and it releases the PAUSE; the actual queue begins dropping packets, and the congestion is managed through higherlevel protocols
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Section 3 FCoE: Fibre Channel over Ethernet

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE: FC over Ethernet
FCoE is I/O consolidation of FC storage traffic over Ethernet
FC traffic shares Ethernet links with other traffics Requires a lossless Ethernet fabric

Ethernet Fibre Channel Traffic

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE Protocol Stack
From a Fibre Channel standpoint, its FC connectivity over a new type of cable called an Ethernet cloud From an Ethernet standpoint, it’s yet another ULP (Upper Layer Protocol) to be transported

1, 10…Gbps

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE Benefits
FCoE benefits are the same of any I/O consolidation solution
Fewer cables Both block I/O and Ethernet traffic coexist on same cable Fewer adapters needed Overall less power

Plus additional advantages of being FC
Seamless integration with existing FC SANs No gateway

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE Benefits
LAN
4

SAN-A

SAN-B

LAN
4

SAN-A

SAN-B

Nearly Half the Cables
16 Servers Adapters Switches Cables Mgmt Pts
BRKSAN-2821 14572_05_2008_c1

Enet 16 2 36 2

FC 16 2 36 2

Total 32 4 72 4
Cisco Public

16 Servers Adapters Switches Cables Mgmt Pts

Enet 16 2 36 2

FC 0 0 4 0

Total 16 2 40 2
52

FCoE Is Fibre Channel
FCoE Is Fibre Channel at the Host and Switch Level
Easy to Understand Same Operational Model Same Techniques of Traffic Management Same Management and Security Models Completely Based on the FC Model Same Host-to-Switch and Switch-to-Switch Behavior of FC e.g., in Order Delivery or FSPF Load Balancing WWNs, FC-IDs, Hard/Soft Zoning, DNS, RSCN

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Protocol Organization

FCoE Itself Is the data plane protocol It is used to carry most of the FC frames and all the SCSI traffic

FIP (FCoE Initialization Protocol) It is the control plane protocol It is used to discover the FC entities connected to an Ethernet cloud It is also used to login to and logout from the FC fabric

The Two Protocols Have:
Two different EtherTypes Two different frame formats
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE Frame Size
Ethernet Header FCoE Header

12 Bytes (MAC Addresses) + 4 Bytes (802.1Q Tag) 16 Bytes

FC Header

24 Bytes

Tota l:

Up to 2112 Bytes 4 Bytes

2180 Byte s

CRC EOF FCS
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved.

FCoE Frame Format
Destination MAC Address

EOF FCS
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FC Payload
ET = FCoE

1 Byte (EOF) + 3 Bytes (Padding) 4 Bytes
Cisco Public

Source MAC Address IEEE 802.1Q Tag Ver Reserved Reserved Reserved Encapsulated FC Frame (Including FC-CRC) Reserved SOF Reserved

ENode: Simplified Model
ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC
a.k.a. CNA (Converged Network Adapter)

FC Node FCoE FCoE

Enet Port

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE Switch: Simplified Model
FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch
FC Port

FCoE Switch FCF FCoE

FC Port FC Port FC Port Eth Port Eth Port Eth Port

Ethernet Bridge

Eth Port

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FC-BB-5 Terminology
Unchanged from previous FC standard
VN_Port: Virtual N_Port VF_Port: Virtual F_Port VE_Port: Virtual E_Port

Added to support FCoE
FCoE_LEP (FCoE link endpoint): The data forwarding component that handles FC frame encapsulation/decapsulation, and transmission/reception of FCoE frames FCoE Controller: the entity that implement the FIP protocol

BRKSAN-2821 14572_05_2008_c1

Cisco Public

ENode: Complete Model

FC-3/FC-4s

VN_Port

FC Entity

VN_Port

FC Entity

VN_Port

FC Entity

VN_Port

FC Entity

FCoE_LEP

FCoE Entity FCoE Controller

FCoE_LEP

FCoE Entity

FCoE_LEP

FCoE Entity FCoE Controller

FCoE_LEP

FCoE Entity

Lossless Ethernet MAC

Ethernet_Port

Lossless Ethernet MAC

Ethernet_Port

SC DI FLOGI F
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE Switch: Complete Model
FC Fabric Interface
E_Port E_Port E_Port F_Port F_Port

Means Optional

F_Port

FC Switching Element

VE_Port

FC Entity

VF_Port

FC Entity

VE_Port

FC Entity

VF_Port

FC Entity

FCoE_LEP

FCoE Entity FCoE Controller

FCoE_LEP FCoE FCoE_LEP Entity FCoE_LEP

FCoE_LEP

FCoE Entity FCoE Controller

FCoE_LEP FCoE FCoE_LEP Entity FCoE_LEP

Lossless Ethernet MAC

Ethernet_Port

Lossless Ethernet MAC

Ethernet_Port

Lossless Ethernet Bridging Element
Ethernet_Port Ethernet_Port Ethernet_Port

Ethernet_Port

Lossless Ethernet Bridging Element
Ethernet_Port Ethernet_Port Ethernet_Port

Ethernet_Port

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE: Initial Deployment
SAN A SAN B 10 GE Backbone

VF_Ports VN_Ports

10 GE 4/8 Gbps FC
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE: Adding Blade Servers
SAN A SAN B 10 GE Backbone

VF_Ports

VN_Ports
10 GE 4/8 Gbps FC
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE: Adding Native FCoE Storage
SAN A SAN B 10 GE Backbone

VN_Ports VF_Ports

VN_Ports
10 GE 4/8 Gbps FC
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE: Adding VE_Ports
SAN A SAN B 10 GE Backbone

VE_Ports VF_Ports

VN_Ports
10 GE 4/8 Gbps FC
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE Addressing and Forwarding
FCoE frames have:
MAC addresses (hop-by-hop) FC addresses (end-to-end)

FC Storage
FCID 7.1.1 FC Fabric FC Domain 7 FC Fabric FC Domain 3 MAC A Ethernet Fabric FC Domain 1 MAC B Ethernet Fabric

FCID 1.1.1 MAC C

D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

Dest. = MAC B Srce. = MAC A D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

Dest. = MAC C Srce. = MAC B D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)

FC Frame

FCoE Frame

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE MAC Addresses
VE_Ports and VF_Ports always use MAC addresses derived from the switch pool VN_Ports may use two types of MAC addresses
SPMA (Server Provided MAC Addresses) FPMA (Fabric Provided MAC Addresses)

MAC Addresses are negotiated in FIP Initial deployment will use FPMA only

BRKSAN-2821 14572_05_2008_c1

Cisco Public

The Mapped MAC Addresses
A dedicated MAC address for each FC-ID
Assigned by the FC fabrics Consistent with the FC model OUIs with U/L = 1 (Local addressing), called FC-MAPs Multiple FC-MAPs may be supported (one per FC fabric)
24 Bits FC-MAP (ex 02-12-34) 24 Bits FC-ID 7.8.9

MAC Address
BRKSAN-2821 14572_05_2008_c1

FC-MAP (ex 02-12-34) 48 Bits

FC-ID 7.8.9

Cisco Public

Initial Login Flow Ladder
ENode
Discovery
Solicitati on
Advertisement

FCoE Switch
Discovery FIP: FCoE Initialization Protocol

FLOGI/FDISC

FLOGI/FDISC Accept

FC Command

FC Command Responses

FCOE Protocol

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FIP Frame: Contains FIP Operation
Destination MAC Address Source MAC Address IEEE 802.1Q Tag ET = FIP Ver Reserved

Encapsulated FIP Operation (Self-Describing Length) PAD to Minimum Length or Mini-Jumbo Length Ethernet FCS

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FIP Descriptors (1)
Type = 1 Type = 2 Len = 4 Len = 8 MAC Address Type = 3 Len = 8 FC-MAP Type = 4 Len = 12 Switch_Name Type = 5 Len = 12 Fabric_Name
BRKSAN-2821 14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Reserved

Priority

Reserved

FIP Descriptors (2)
Type = 6 Len = 12 Port_Name Reserved

Type = 7

Len = XX

Reserved

FLOGI Request, FLOGI LS_ACC/LS_RJT NPIV FDISC Request, FDISC LS_ACC/LS_RJT Fabric LOGO Request, LOGO LS_ACC/LS_RJT (No SOF/EOF / FC-CRC?)

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FIP
Multicast Solicitation from H2
MAC (H1) FCF-MAC (A)

H1 Lossless Ethernet Bridge

FCF A FC Fabric FCF A
MAC (H2) FCF-MAC (B)

All-MACs MAC(H2) Solicitation (FIP)
[F=0, S=0, MAC(H2), Capability, Other]

Solicitation identifies VF_Port capable FCF-MACs with compatible addressing capabilities Other parameters may include ENode’s Port_Name for optional duplicate MAC address detection
Cisco Public

BRKSAN-2821 14572_05_2008_c1

FIP
Unicast Advertisements from A and B
MAC (H1) FCF-MAC (A)

H1 Lossless Ethernet Bridge

FCF A FC Fabric FCF A
MAC (H2) FCF-MAC (B)

MAC(H2) FCF-MAC(A) Mini-jumbo Advertisement (FIP)
[S=1, F=1, Priority, FC-MAP, FCF-MAC(A), Switch_Name, Fabric_Name, Capability, Other]
BRKSAN-2821 14572_05_2008_c1

MAC(H2) FCF-MAC(B)

H2’s FCF list:
Mini-jumbo Advertisement (FIP)
[S=1, F=1, Priority, FC-MAP, FCF-MAC(B), Switch_Name, Fabric_Name, Capability, Other]
Cisco Public

FCF-MAC(A) [J] FCF-MAC(B) [J]

FCF not meeting capability of ENode does not reply

FIP
FLOGI Request
MAC (H1) FCF-MAC (A)

H1 Lossless Ethernet Bridge

FCF A FC Fabric FCF A
MAC (H2) FCF-MAC (B)

FCF-MAC(A) MAC (H2) FLOGI Request (FIP)
[FC Header, FLOGI data, Proposed MAC’(H2)]

FCF-MAC(B) MAC(H2) FLOGI Request (FIP)
[FC Header, FLOGI data, Proposed MAC’’(H2)]

Capability agreed during discovery

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FIP
FLOGI LS_ACC
MAC (H1) FCF-MAC (A)

H1 Lossless Ethernet Bridge

FCF A FC Fabric FCF A
MAC (H2) FCF-MAC (B)

MAC (H2) FCF-MAC(A) FLOGI LS-ACC (FIP)
[FC Header, LS_ACC data, Approved MAC(H2)’]

MAC(H2) FCF-MAC(B) FLOGI LS-ACC (FIP)
[FC Header, LS_ACC data, Approved MAC(H2)’’]

ENode uses MAC address in FIP FLOGI LS_ACC as the VN_Port MAC address for the FC-ID contained in the FLOGI data for subsequent FCoE frames
76

BRKSAN-2821 14572_05_2008_c1

Cisco Public

FCoE
Data Transfer
MAC (H1) FCF-MAC (A)

H1 Lossless Ethernet Bridge

FCF A FC Fabric FCF A
MAC (H2) MAC (H2)’ MAC (H2)’’ FCF-MAC (B)

FCF-MAC(A) MAC(H2)’ Fibre Channel Frame (FCoE)
[FC SOF, FC Header, FC Data, FC CRC FC EOF]
BRKSAN-2821 14572_05_2008_c1

FCF-MAC(B) MAC(H2)’’ Fibre Channel Frame (FCoE)
[FC SOF, FC Header, FC Data, FC CRC, FC EOF]

All subsequent FCoE frames use granted MAC address and assigned FC-ID
FIP frames continue to use MAC(H2) For SPMA, MAC(H2)’ = MAC(H2)’’ = MAC(H2) For FPMA, MAC(H2)’ and MAC(H2)’’ use FC-IDs as low order 24 bits and FC-MAP for upper 24 bits
77

Cisco Public

The Most Asked Question:
Is FCoE Routable?

FCoE

1, 2, 4, (8), 10 Gbps

1, 10 . . . Gbps

10, 20 Gbps

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Is FCoE Routable?
Most folks mean “Is FCoE IP-routable”
The answer is NO, there is no IP layer in FCoE This was a design goal to keep FCoE simple FC-BB-5 contains FCIP that is “IP-routable”

FCoE is FC-routable
FCoE switches may forward FC frames across different Ethernet clouds FCoE switches may forward FC frames over the Internet using FCIP

FCoE

FCIP

IP Cloud

FCIP

FCoE

BRKSAN-2821 14572_05_2008_c1

Cisco Public

CNA: Converged Network Adapter
LAN
10GbE
Link

HBA
HBA HBA

CNA
10GbEE 10GbEE

BRKSAN-2821 14572_05_2008_c1

10GbE

Link

Fibre Channel Ethernet

PCIe

Ethernet Drivers

Ethernet

PCIe

Fibre Channel Drivers

Ethernet Drivers

Fibre Channel Drivers

Operating System
80

View from Operating System
Standard drivers Same management Operating system sees:
Dual-port, 10 Gigabit Ethernet adapter Dual-port, 4 Gbps Fibre Channel HBAs

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Open-FCoE Software
HBA HBA Mgmt Plane Linux Kernel File System layers SCSI Layer Linux Kernel File System layers SCSI Layer OpenFC Layer HBA Driver FCoE Layer Net Device HBA Ethernet Driver Ethernet FCoE FCoE Mgmt Plane

Server
Cisco Public

Ethernet

Server
82

Open-FCoE Software: How to Get It
Open source project Open-FCoE.org—source Git trees Or Open-FCoE source package—TBD Install a Linux Red Hat EL5, Fedora Core 7, or SuSE 10 distribution Update kernel to 2.6.23 or later Install: see Quick Start Guide at open-fcoe.org Use switch, soft-target, or gateway

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Wireshark
Once known as Ethereal Captures and displays network traffic Available from: http://wireshark.org/ Sample trace file
/common/openfc/traces/fcoe-t11.cap

Use tcpdump to capture
tcpdump –i eth0 –s 0 –w /tmp/fcoe.cap

Screenshots/demo

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Wireshark Screenshot

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Section 4 Case Studies

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Current Data Center Environment
LAN Core SAN-A SAN-B

Distribution
4 MDS 9500 Cisco Catalyst® 6509

4 x 4G FC

1 x 10 GE

STP BLK

POD 1

POD N

Access
MDS 9500 Cisco Catalyst 6509 NIC Teaming Active/Standby Discrete 1 GE NICs and FC HBA

Server Cabinet 1

Server Cabinet N

Server Cabinet 1

Server Cabinet N

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Top-of-Rack Consolidated I/O
I/O Consolidation at Access
LAN Core SAN-A 8 8 SAN-B

Distribution
MDS 9500 Cisco Catalyst 6509

STP BLK

POD 1

POD N

Access
Nexus 5000

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Server Cabinet Pair 1
BRKSAN-2821 14572_05_2008_c1

Server Cabinet Pair N
Cisco Public

Server Cabinet Pair 1

Server Cabinet Pair N
88

Top-of-Rack Consolidated I/O
Ethernet Host Virtualizer
LAN Core SAN-A 8 8 SAN-B

Distribution
MDS 9500 Cisco Catalyst 6509

Ethernet Host Virtualizer Active/Active POD 1 POD N

Access
Nexus 5000

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Server Cabinet Pair 1
BRKSAN-2821 14572_05_2008_c1

Server Cabinet Pair N
Cisco Public

Server Cabinet Pair 1

Server Cabinet Pair N
89

Top-of-Rack Consolidated I/O
VSS Support at Aggregation
LAN Core SAN-A 8 SAN-B

VSS
8

Distribution
MDS 9500 Cisco Catalyst 6509

VSS Support at Aggregation POD 1 POD N

Access
Nexus 5000

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Server Cabinet Pair 1
BRKSAN-2821 14572_05_2008_c1

Server Cabinet Pair N
Cisco Public

Server Cabinet Pair 1

Server Cabinet Pair N
90

Physical Topology—40 Servers, ToR
LAN SAN-A SAN-B

4x10 GE Ports

6x4 GFC

Nexus 5000 40x10 GE Ports per Switch

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Physical Topology—200 Servers
40 4GFC
LAN SAN-A SAN-B

4x10 GE Ports

6x4 GFC

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Blade Servers with Copper Pass-Through
DC Core MDS SAN-Core 8 MDS SAN-Core

Distribution
8 MDS 9500 Cisco Catalyst 6509

POD 1

POD N

Access
Nexus Family

Blade Server
Copper Pass-Through

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Blade Server 1
BRKSAN-2821 14572_05_2008_c1

Blade Server N
Cisco Public

Blade Server 1

Blade Server N
93

Blade Servers with Ethernet Switches
DC Core MDS SAN-Core 8 MDS SAN-Core

Distribution
8 MDS 9500 Cisco Catalyst 6509

POD 1

POD N

Access
Nexus Family

Blade Server
Ethernet-Only Switch

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Blade Server 1
BRKSAN-2821 14572_05_2008_c1

Blade Server 1
Cisco Public

Blade Server 1

Blade Server 1
94

Physical Topology— 20 Blade Server, ToR
LAN SAN-A SAN-B

2x10 GE Ports

4x4 GFC

Nexus 5000

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Topology—100 Blade Servers, ToR
LAN SAN-A SAN-B

4x10 GE Ports 8x4 GFC 4x10 GE Ports 8x4 GFC

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Access Nexus 5000 Distribution Nexus 7000
LAN Core SAN-A 8 SAN-B

Distribution
8 MDS 9500 Nexus 7000

Layer 2 Multipath POD 1 POD N

Access
Nexus 5000

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Server Cabinet Pair 1
BRKSAN-2821 14572_05_2008_c1

Server Cabinet Pair N
Cisco Public

Server Cabinet Pair 1

Server Cabinet Pair N
97

Consolidation in the Distribution Layer
LAN Core SAN-A 8 SAN-B

Distribution
8 MDS 9500 Nexus 7000

Layer 2 Multipath POD 1 POD N

Access
Nexus 5000

Adapter: CNA Converged Network Adapter 10 GE/FCoE

Server Cabinet Pair 1
BRKSAN-2821 14572_05_2008_c1

Server Cabinet Pair N
Cisco Public

Server Cabinet Pair 1

Server Cabinet Pair N
98

Conclusions

BRKSAN-2821 14572_05_2008_c1

Cisco Public

Challenges
FCoE redefines consolidated scenarios
Ethernet switch manufacturers will try to enter the FC switching market FC switch manufacturers will try to enter the Ethernet switching market HBA manufacturers will try to enter the NIC market NIC manufacturers will try to enter the HBA market

Deep integration with virtualization will take some time

BRKSAN-2821 14572_05_2008_c1

Cisco Public

100

Web Pointers
PCI Express
http://en.wikipedia.org/wiki/Pci_express

IEEE 802.3
http://standards.ieee.org/getieee802/802.3.html

Improvements to Ethernet
http://www.nuovasystems.com/EthernetEnhancements-Final.pdf

IEEE 802.1 activities
http://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf http://www.ieee802.org/1/files/public/docs2007/new-cm-pelissier-enabling-block-storage-0705-v01.pdf http://www.ieee802.org/1/files/public/docs2007/au-ko-fabric-convergence-0507.pdf http://www.ieee802.org/1/pages/802.1au.html http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf

FCoE
http://www.fcoe.com/ http://www.t11.org/ http://www.open-fcoe.org/ http://www.fibrechannel.org/OVERVIEW/FCIA_SNW_FCoE_WP_Final.pdf

101

Thank You

BRKSAN-2821 14572_05_2008_c1

Cisco Public

102

Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books

103

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.

BRKSAN-2821 14572_05_2008_c1

Cisco Public

104

BRKSAN-2821 14572_05_2008_c1

Cisco Public

105

BRKSAN-2821

Related titles

BRKSAN-2821 14572_05_2008_c1

Related Interests

Documents Similar To BRKSAN-2821

More From jeffgrantinct