14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1
I/O Consolidation
in the Data Center
BRKSAN-2821
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2
© 2006, Cisco Systems, Inc. All rights reserved. 1
Presentation_ID.scr
Agenda
Section 1: What is I/O Consolidation
Section 2: Enabling Technologies
Section 3: FCoE (Fibre Channel over Ethernet)
Section 4: I/O Consolidation Use Cases
Challenges
Closing Remarks
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3
Section 1
What Is I/O
Consolidation
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4
© 2006, Cisco Systems, Inc. All rights reserved. 2
Presentation_ID.scr
What Is I/O Consolidation
IT organizations operate multiple parallel networks
IP and other LAN protocols over an Ethernet network
SAN over a Fibre Channel network
HPC/IPC over an InfiniBand network
I/O consolidation supports all three types of traffic onto
a single network
Servers have a common interface adapter that supports
all three types of traffic
IPC: Inter-Process Communication
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5
I/O Consolidation in the Network
Processor Processor
Memory Memory
I/O I/O I/O I/O Subsystem
Storage
Storage
LAN
IPC
LAN
IPC
IPC: Inter-Process Communication
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6
© 2006, Cisco Systems, Inc. All rights reserved. 3
Presentation_ID.scr
I/O Consolidation in the Host
Fewer CNAs (Converged Network Adapters) instead
of NICs, HBAs, and HCAs
Limited number of interfaces for Blade Servers
FC HBA FC Traffic
FC HBA FC Traffic
CNA All Traffic
NIC Enet Traffic Goes over
10 GE
NIC Enet Traffic CNA
NIC Enet Traffic
HCA IPC Traffic
HCA IPC Traffic
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7
Cabling and I/O Consolidation
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8
© 2006, Cisco Systems, Inc. All rights reserved. 4
Presentation_ID.scr
I/O Consolidation: Benefits to Customers
FC Traffic
FC Traffic FCoE
FCoE SAN A
FCoE
Enet Traffic FCoE SAN B
Enet Traffic
FCoE SAN
Fewer CNAs and Cables Storage Keeps the Same
Management Model as Native FC
Display
FCoE
Adapter
FC Storage FC Switch FCoE Server
Switch
No Storage Gateway Less Power and Cooling
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9
Merging the Requirements
IPC
LAN/IP Storage (Inter-Process
Communication)
Must be Ethernet Must follow the Doesn’t care of
Too much
Fibre Channel the underlying
investment model network, provided
that:
Too many Losing frames is
applications that not an option It is cheap
assume Ethernet It is low latency
It supports APIs
like OFED, RDS,
MPI, sockets
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10
© 2006, Cisco Systems, Inc. All rights reserved. 5
Presentation_ID.scr
Why Has It Not Succeeded Yet?
Previous attempts
Fibre Channel
Never credible
InfiniBand
Not Ethernet
iSCSI
Not Fibre Channel
Before PCI-Express there was not enough I/O
bandwidth in the servers
It needs to be Ethernet, but…
1 GE didn’t have enough bandwidth
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11
PCI-Express
PCI Express (PCI-E or PCIe)
A computer expansion card interface format designed to
replace PCI, PCI-X, and AGP
PCIe 1.1
Serial links at 2.5 Gbps (2 Gbps at the Datalink)
Speeds from 2 Gbps (1x) to 32 Gbps (16x)
8x is required for 10 GE
PCIe 2.0 (aka PCIe Gen 2)
Doubles the bandwidth per serial link from 2 Gbit/s to 4 Gbit/s
Spec available since January 2007
Products are making their way into the market
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12
© 2006, Cisco Systems, Inc. All rights reserved. 6
Presentation_ID.scr
10 GE
2008 will be the year of 10GE
10 GE has enough bandwidth
Merging example
2 x 1 GE Ethernet NIC
1 x 4 Gbps FC (really 3.2 Gbps)
Total 5.2 Gbps over a 10 Gbps link
CNAs will all be dual-ported for HA
20 Gbps usable bandwidth per server with a single CNA
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13
Evolution of Ethernet Physical Media
Role of Transport in Enabling These Technologies
Mid-1980s Mid-1990s Early-2000s Late-2000s
SFP+ to SFP+
10 Mb 100 Mb 1SFP+
Gb Cu 10 Gb
UTP Cat 5 UTP Cat 5
SFP Fiber
Power Transceiver
Technology Cable Distance (Each Side) Latency (Link)
SFP+ CU
Twinax 10 m 0.1W 0.1 μs
Copper
SFP+ USR MM OM2 10 m
1W 0
Ultra Short Reach MM OM3 100 m
SFP+ SR MM 62.5 μm 82 m
Short Reach MM 50 μm 300 m 1W 0
Cat6 55 m 8W 2.5 μs
10GBASE-T Cat6a/7 100 m 8W 2.5 μs
Cat6a/7 30 m 4W 1.5 μs
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14
© 2006, Cisco Systems, Inc. All rights reserved. 7
Presentation_ID.scr
Twin-ax Copper Cable
Low power consumption SAN A LAN SAN B
Low cable cost
Low transceiver latency
Application Server
Low error rate (10–17) Application Server
Application Server
Thinner cable with higher Application Server
16x10 GE Cables
16x10 GE Cables
Application Server
bend radius Application Server
Application Server
Easier to manage cabling Application Server
Application Server
solution reduces Application Server
deployment time Application Server
Application Server
All copper cables are Application Server
Application Server
contained within rack Application Server
Application Server
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15
Drivers for 10GE to the Servers
Multicore CPU Architectures Allowing Bigger and
Multiple Workloads on the Same Machine
Server Virtualization Driving the Need for More
Bandwidth per Server Due to Server Consolidation
Growing Need for Network Storage Driving the
Demand for Higher Network Bandwidth to the Server
Multicore CPUs and Server Virtualization Driving the
Demand for Higher Bandwidth Network Connections
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16
© 2006, Cisco Systems, Inc. All rights reserved. 8
Presentation_ID.scr
Section 1
Enabling Technologies
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17
Three Challenges + One
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18
© 2006, Cisco Systems, Inc. All rights reserved. 9
Presentation_ID.scr
Why Are Frames Lost?
Transmission
Collision Congestion
Error
No longer present Very rare in the Most common
in full duplex data center cause
Ethernet
Congestion is a
switch issue, not a
link issue
A full duplex IEEE
802.3 link does not
lose frames
It must be dealt
with in the
bridge/switch
By IEEE 802.1
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19
Can Ethernet Be Lossless?
Yes, with Ethernet PAUSE Frame
Ethernet Link
STOP PAUSE Queue Full
Switch A Switch B
Defined in IEEE 802.3—Annex 31B
The PAUSE operation is used to inhibit transmission of
data frames for a specified period of time
Ethernet PAUSE transforms Ethernet into a
lossless fabric
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20
© 2006, Cisco Systems, Inc. All rights reserved. 10
Presentation_ID.scr
How PAUSE Works
Threshold
A B
PAUSE
Frame
Start
Stop Sending
Frames
FramesforAgain
This
Interval of Time
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21
Let’s Compare PAUSE with
FC Buffer to Buffer Credit
Eight credits preagreed
A B
R_RDY
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22
© 2006, Cisco Systems, Inc. All rights reserved. 11
Presentation_ID.scr
How PAUSE Propagates
Threshold
PAUSE PAUSE
S1 S2 S3
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23
PAUSE Frame Format
PAUSE Frame
A standard Ethernet frame, not
01:80:C2:00:00:01 tagged
Source Station MAC
EtherType = 0x8808 means MAC
Control Frame
EtherType = 0x8808
Opcode = 0x0001
Pause_Time
Opcode = 0x0101 means PAUSE
Pause_Time is the time the link
needs to remain paused in
Pad
Pause Quanta (512-bits time)
42 Bytes
There is a single Pause_Time for
the whole link
…
CRC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24
© 2006, Cisco Systems, Inc. All rights reserved. 12
Presentation_ID.scr
Why Is PAUSE Not Widely Deployed?
Inconsistent implementations
Standard allows for asymmetric implementations
Easy to fix
PAUSE applies to the whole links
Single mechanism for all traffic classes
This may cause “traffic interference”
e.g., Storage traffic paused due to a congestion on IP traffic
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25
Priority Flow Control (PFC)
a.k.a. PPP (Per Priority Pause)
PFC enables PAUSE functionality per Ethernet priority
IEEE 802.1Q defines eight priorities
Traffic classes are mapped to different priorities:
No traffic interference
IP traffic may be paused while storage traffic is being forwarded
Or, vice versa
Requires independent resources per priority (buffers)
High level of industry support
Cisco distributed proposal
Standard track in IEEE 802.1Qb
16 3 1 12 Bits
EtherType = IEEE 802.1Q Priority CFI VLAN ID
IEEE 802.1Q Tag
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26
© 2006, Cisco Systems, Inc. All rights reserved. 13
Presentation_ID.scr
Priority Flow Control in Action
Transmit Queues Receive Queues
Ethernet Link
One One
Two Two
Three Three
Eight
Four Four Priorities
Five Five
Six STOP PAUSE Six
Seven Seven
Eight Eight
Switch A Switch B
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27
PFC Frame Format
Priority Flow Control
Similar to the PAUSE frame
01:80:C2:00:00:01
Opcode = 0x0101 is used to
Source Station MAC
distinguish PFC from PAUSE
EtherType = 0x8808 Class vector indicates for which
Opcode = 0x0101
Class Enable Vector priorities the frame carries valid
Time (Class 0)
Time (Class 1)
Pause information
Time (Class 2)
Time (Class 3) There are eight Time fields,
Time (Class 4)
Time (Class 5)
one per priority
Time (Class 6)
Time (class 7)
Pad
28 Bytes
…
CRC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28
© 2006, Cisco Systems, Inc. All rights reserved. 14
Presentation_ID.scr
Is Lossless Better?
Frames are TCP relies on Congestion
not dropped losses spreading and
head of line
FC over lossless We can run it on
blocking
Ethernet works a priority where
well we do not enable
Pause
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29
Is Anything Else Required?
In Order to Build a Deployable I/O Consolidation
Solution, the Following Additional Components
Are Required:
Discovery protocol (DCBX)
Bandwidth manager
Congestion management
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30
© 2006, Cisco Systems, Inc. All rights reserved. 15
Presentation_ID.scr
Discovery Protocol
DCBX:
Data Center
Bridging eXchange
Data Center
Ethernet Links
DCBX Data Center Ethernet
Links with Partial
Enhancements
DCBX
Legacy Ethernet
Links
Data Center
Ethernet Cloud
XP
Legacy Ethernet
Network
BC
DC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31
DCBX
Hop-by-hop negotiation for:
Priority Flow Control (PFC)
Bandwidth management
Congestion management (BCN/QCN)
Applications
Logical link-down
Based on LLDP (Link Level Discovery Protocol)
Added reliable transport
Allows either full configuration or configuration checking
Link partners can choose supported features and willingness to
accept configuration from peer
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32
© 2006, Cisco Systems, Inc. All rights reserved. 16
Presentation_ID.scr
Bandwidth Management
IEEE 802.1Q defines priorities, but not a simple,
effective, and consistent scheduling mechanism
Products typically implement some form of Deficit
Weighted Round Robin (DWRR)
Configuration and interworking is problematic
Proposal for HW-efficient, two-level DWRR with strict
priority support
Consistent behavior and configuration across network elements
Standard track in IEEE 802.1Qaz
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33
Priority Groups
Priority Groups
Are Then
Scheduled
LAN
Priorities Are
Assigned to
Individual SAN
Traffic Classes
IPC Final Link
Behavior
Priority
Groups
First Level of Scheduling
Inside Each Group
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34
© 2006, Cisco Systems, Inc. All rights reserved. 17
Presentation_ID.scr
Goals
BW assignment for each Cannot compromise low-
“Priority Group” latency application due
Example: 40% LAN, to convergence
40% SAN, 20% IPC Allow strict, high priority
scheduling of IPC (and
Should allow multiple traffic equivalent) traffic
classes within a “Priority
Group” Should provide
management infrastructure
Allow these traffic classes to
share BW without hard (MIBs)
configuration Defining scheduling algorithms
Example: VoIP and bulk traffic is too restrictive and not
to share 40% LAN BW necessary
Interoperability for
management is important
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35
Example of Link Bandwidth Allocation
Offered Traffic 10 GE Link Realized Traffic Utilization
3 Gbs 3 Gbs 2 Gbs (30%) HPC Traffic (20%)
(30%)
LAN Traffic
(30%) (50%)
3 Gbs 4 Gbs 6 Gbs (40%)
Storage Traffic
(30%) (30%)
3 Gbs 3 Gbs 3 Gbs (30%)
T1 T2 T3 T1 T2 T3
HPC Traffic—Priority Class “High”—20% Guaranteed Bandwidth
LAN Traffic—Priority Class “Medium”—50% Guaranteed Bandwidth
Storage Traffic—Priority Class “Medium-High”—30% Default Bandwidth
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36
© 2006, Cisco Systems, Inc. All rights reserved. 18
Presentation_ID.scr
Congestion Management
Layer 2, end-to-end congestion management
Standards track in IEEE 802.1Qau
a.k.a. BCN (Backward Congestion Notification) or
QCN (Quantized Congestion Notification)
Switch Switch
Switch
Transmit Queue
Thro
ttle Receive Buffer
Switch Switch
e
ottl
Thr
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37
Congestion Management Principles
Move congestion to network edges to avoid congestion
spreading
Use rate-limiters at the edge to shape flows causing
congestion
Tune rate-limiter parameters based on feedback coming from
congestion points
Inspired by:
TCP
AIMD (Additive Increase, Multiplicative Decrease) rate control
TCP window increases linearly in absence of congestion
Decreases exponentially (gets halved) at every congestion
indication (either implicit or explicit)
FCC (Fibre Channel Congestion Control)
A feature on Cisco MDS switches
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38
© 2006, Cisco Systems, Inc. All rights reserved. 19
Presentation_ID.scr
Congestion Point and Reaction Point
Roles and responsibilities
Reaction Points (RP) shape
traffic entering the network
RP
Congestion Points (CP)
indicate congested state
of queuing points
CP
RP
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39
DCB: Data Center Bridging
Industry consensus term to indicate
Priority flow control
Bandwidth management
Congestion management
Discovery (DCBX)
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40
© 2006, Cisco Systems, Inc. All rights reserved. 20
Presentation_ID.scr
DCE: Data Center Ethernet
Cisco term used to indicate Cisco switches that
implement the DCB features, plus
Layer 2 multipathing
Fibre Channel over Ethernet
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41
Layer 2 Multipathing
Increase bandwidth
of L2 networks via
multiple active links
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42
© 2006, Cisco Systems, Inc. All rights reserved. 21
Presentation_ID.scr
L2 Multipathing
Multiple paths are used, reclaiming network bandwidth
L3 multipathing is common in IP networks
Important when there is limited or no differentiation in
speed between access links and backbone links
Reduces latency
L2 multipathing
Eliminates Spanning Tree from the backbone
No packet flooding
Small forwarding tables
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43
Layer 2 Multipathing
Cisco DCE is:
A precursor of TRILL, an IETF project for Layer 2 multipath
Inspired to FSPF (Fibre Channel Shortest Path First)
Cisco DCE
9 Computes topology and forwarding via IS-IS
9 Provides optimal pair-wise unicast forwarding
9 Provides multipathing for unicast and multicast frames
9 Provides seamless interoperability with existing devices
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44
© 2006, Cisco Systems, Inc. All rights reserved. 22
Presentation_ID.scr
FCoE: Fibre Channel over Ethernet
FCoE is the protocol
used to carry Fibre
SAN LAN SAN
Channel over CEE/DCE
Allows storage
I/O consolidation
FCoE
It’s in an advanced state
of definition in INCITS
T11 FC-BB-5 WG
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45
Delayed Drop
Delayed Drop is a mechanism that:
Allows a switch buffer to virtually extend to previous hop
This reduces packet drop for transient congestions
Is enabled per priority
It is implemented by asserting PFC on the priority for
a short time
After that time, traffic can flow again or can be dropped
Delayed Drop Is a Means of Using PFC to Mitigate
the Effects of Short-Term Traffic Bursts While
Maintaining Packet Drop for Long-Term Congestion
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46
© 2006, Cisco Systems, Inc. All rights reserved. 23
Presentation_ID.scr
Delay Drop and Proxy Queue
Actual Queue
Traffic
Flow
PAUSE
Proxy Queue
UNPAUSE
During short-term congestion, both queues drain fast enough that
the actual queue releases the PAUSE on its own
During long-term congestion, the proxy queue fills to its high-water
mark, and it releases the PAUSE; the actual queue begins
dropping packets, and the congestion is managed through higher-
level protocols
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47
Section 3
FCoE: Fibre Channel
over Ethernet
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48
© 2006, Cisco Systems, Inc. All rights reserved. 24
Presentation_ID.scr
FCoE: FC over Ethernet
FCoE is I/O consolidation of FC storage traffic
over Ethernet
FC traffic shares Ethernet links with other traffics
Requires a lossless Ethernet fabric
Ethernet
Fibre Channel
Traffic
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49
FCoE Protocol Stack
From a Fibre Channel
standpoint, its FC connectivity
over a new type of cable
called an Ethernet cloud
From an Ethernet standpoint,
it’s yet another ULP (Upper
Layer Protocol) to be
transported
1, 10…Gbps
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50
© 2006, Cisco Systems, Inc. All rights reserved. 25
Presentation_ID.scr
FCoE Benefits
FCoE benefits are the same of any I/O consolidation
solution
Fewer cables
Both block I/O and Ethernet traffic coexist on same cable
Fewer adapters needed
Overall less power
Plus additional advantages of being FC
Seamless integration with existing FC SANs
No gateway
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51
FCoE Benefits
LAN SAN-A SAN-B LAN SAN-A SAN-B
4 4
2
2
Nearly Half
the Cables
16 Servers Enet FC Total 16 Servers Enet FC Total
Adapters 16 16 32 Adapters 16 0 16
Switches 2 2 4 Switches 2 0 2
Cables 36 36 72 Cables 36 4 40
Mgmt Pts 2 2 4 Mgmt Pts 2 0 2
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52
© 2006, Cisco Systems, Inc. All rights reserved. 26
Presentation_ID.scr
FCoE Is Fibre Channel
FCoE Is Fibre Channel at the Host and Switch Level
Easy to Completely Based
Understand on the FC Model
Same Host-to-Switch and
Same
Operational Model Switch-to-Switch Behavior
of FC
Same Techniques of e.g., in Order Delivery or
Traffic Management FSPF Load Balancing
Same Management WWNs, FC-IDs, Hard/Soft
and Security Models Zoning, DNS, RSCN
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53
Protocol Organization
FCoE Itself FIP
(FCoE Initialization Protocol)
Is the data plane protocol It is the control plane protocol
It is used to carry most of the It is used to discover the FC
FC frames and all the SCSI entities connected to an
traffic Ethernet cloud
It is also used to login to and
logout from the FC fabric
The Two Protocols Have:
Two different EtherTypes
Two different frame formats
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54
© 2006, Cisco Systems, Inc. All rights reserved. 27
Presentation_ID.scr
FCoE Frame Size
Ethernet
Header 12 Bytes (MAC Addresses) +
4 Bytes (802.1Q Tag)
FCoE
Header
16 Bytes
FC
Header
24 Bytes
Tota
l: 2180
FC Payload
Byte
Up to 2112 Bytes s
4 Bytes
1 Byte (EOF) + 3 Bytes (Padding)
CRC
EOF
FCS
4 Bytes
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55
FCoE Frame Format
Destination MAC Address
Source MAC Address
IEEE 802.1Q Tag
ET = FCoE Ver Reserved
Reserved
Reserved
Reserved SOF
Encapsulated FC Frame
(Including FC-CRC)
EOF Reserved
FCS
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56
© 2006, Cisco Systems, Inc. All rights reserved. 28
Presentation_ID.scr
ENode: Simplified Model
ENode (FCoE Node): a Fibre Channel HBA
implemented within an Ethernet NIC
a.k.a. CNA (Converged Network Adapter)
FC Node
FCoE FCoE
Enet Enet
Port Port
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57
FCoE Switch: Simplified Model
FCF (Fibre Channel Forwarder), the forwarding entity
inside an FCoE switch
FC
Port
FCoE Switch
FC
FCF Port
FCoE
FC
Port
Ethernet Bridge FC
Port
Eth Eth Eth Eth Eth Eth Eth Eth
Port Port Port Port Port Port Port Port
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58
© 2006, Cisco Systems, Inc. All rights reserved. 29
Presentation_ID.scr
FC-BB-5 Terminology
Unchanged from previous FC standard
VN_Port: Virtual N_Port
VF_Port: Virtual F_Port
VE_Port: Virtual E_Port
Added to support FCoE
FCoE_LEP (FCoE link endpoint): The data forwarding
component that handles FC frame encapsulation/decapsulation,
and transmission/reception of FCoE frames
FCoE Controller: the entity that implement the FIP protocol
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59
ENode: Complete Model
FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s
FC FC FC FC
VN_Port VN_Port VN_Port VN_Port
Entity Entity Entity Entity
FCoE FCoE FCoE FCoE
FCoE_LEP FCoE_LEP FCoE_LEP FCoE_LEP
Entity Entity Entity Entity
FCoE FCoE
Controller Controller
Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port
DISC
F FLOGI
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60
© 2006, Cisco Systems, Inc. All rights reserved. 30
Presentation_ID.scr
Means
FCoE Switch: Complete Model Optional
FC Fabric Interface E_Port E_Port E_Port F_Port F_Port F_Port
FC Switching Element
FC FC FC FC
VE_Port Entity VF_Port Entity VE_Port Entity VF_Port Entity
FCoE FCoE_LEP FCoE FCoE FCoE_LEP FCoE
FCoE_LEP Entity FCoE_LEP Entity FCoE_LEP Entity FCoE_LEP Entity
FCoE_LEP FCoE_LEP
FCoE FCoE
Controller Controller
Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port
Ethernet_Port Ethernet_Port
Lossless Ethernet Lossless Ethernet
Bridging Element Bridging Element
Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61
FCoE: Initial Deployment
SAN A SAN B
10 GE
Backbone
VF_Ports
VN_Ports
10 GE
4/8 Gbps FC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62
© 2006, Cisco Systems, Inc. All rights reserved. 31
Presentation_ID.scr
FCoE: Adding Blade Servers
SAN A SAN B
10 GE
Backbone
VF_Ports
VN_Ports
10 GE
4/8 Gbps FC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63
FCoE: Adding Native FCoE Storage
SAN A SAN B
10 GE
Backbone
VN_Ports
VF_Ports
VN_Ports
10 GE
4/8 Gbps FC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64
© 2006, Cisco Systems, Inc. All rights reserved. 32
Presentation_ID.scr
FCoE: Adding VE_Ports
SAN A SAN B
10 GE
Backbone
VE_Ports
VF_Ports
VN_Ports
10 GE
4/8 Gbps FC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65
FCoE Addressing and Forwarding
FCoE frames have:
MAC addresses (hop-by-hop)
FC addresses (end-to-end)
FC Storage FCID 1.1.1
FC Domain 3 FC Domain 1 MAC C
FCID 7.1.1 FC Domain 7 MAC A MAC B
Ethernet Ethernet
Fabric Fabric
FC Fabric FC Fabric
Dest. = MAC B Dest. = MAC C
D_ID = FC-ID (1.1.1) D_ID = FC-ID (1.1.1) Srce. = MAC A Srce. = MAC B
S_ID = FC-ID (7.1.1) S_ID = FC-ID (7.1.1) FCoE
D_ID = FC-ID (1.1.1) D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1) S_ID = FC-ID (7.1.1) Frame
FC Frame FC Frame
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66
© 2006, Cisco Systems, Inc. All rights reserved. 33
Presentation_ID.scr
FCoE MAC Addresses
VE_Ports and VF_Ports always use MAC addresses
derived from the switch pool
VN_Ports may use two types of MAC addresses
SPMA (Server Provided MAC Addresses)
FPMA (Fabric Provided MAC Addresses)
MAC Addresses are negotiated in FIP
Initial deployment will use FPMA only
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67
The Mapped MAC Addresses
A dedicated MAC address for each FC-ID
Assigned by the FC fabrics
Consistent with the FC model
OUIs with U/L = 1 (Local addressing), called FC-MAPs
Multiple FC-MAPs may be supported (one per FC fabric)
24 Bits 24 Bits
FC-MAP FC-ID
(ex 02-12-34) 7.8.9
MAC FC-MAP FC-ID
Address (ex 02-12-34) 7.8.9
48 Bits
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68
© 2006, Cisco Systems, Inc. All rights reserved. 34
Presentation_ID.scr
Initial Login Flow Ladder
ENode FCoE Switch
Solicitati
on
Discovery Discovery
Advertisement
FIP:
FCoE
Initialization
Protocol
FLOGI/FDISC FLOGI/FDISC Accept
FC Command FCOE
FC Command Responses Protocol
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69
FIP Frame: Contains FIP Operation
Destination MAC Address
Source MAC Address
IEEE 802.1Q Tag
ET = FIP Ver Reserved
Encapsulated FIP Operation
(Self-Describing Length)
PAD to Minimum Length or Mini-Jumbo Length
Ethernet FCS
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70
© 2006, Cisco Systems, Inc. All rights reserved. 35
Presentation_ID.scr
FIP Descriptors (1)
Type = 1 Len = 4 Reserved Priority
Type = 2 Len = 8
MAC Address
Type = 3 Len = 8 Reserved
FC-MAP
Type = 4 Len = 12 Reserved
Switch_Name
Type = 5 Len = 12 Reserved
Fabric_Name
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71
FIP Descriptors (2)
Type = 6 Len = 12 Reserved
Port_Name
Type = 7 Len = XX Reserved
FLOGI Request, FLOGI LS_ACC/LS_RJT
NPIV FDISC Request, FDISC LS_ACC/LS_RJT
Fabric LOGO Request, LOGO LS_ACC/LS_RJT
(No SOF/EOF / FC-CRC?)
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72
© 2006, Cisco Systems, Inc. All rights reserved. 36
Presentation_ID.scr
FIP
Multicast Solicitation from H2
MAC (H1) FCF-MAC (A)
H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A
MAC (H2) FCF-MAC (B)
All-MACs
Solicitation identifies VF_Port capable FCF-MACs
MAC(H2)
with compatible addressing capabilities
Solicitation (FIP)
Other parameters may include ENode’s
[F=0, S=0, MAC(H2),
Capability, Other]
Port_Name for optional duplicate MAC
address detection
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73
FIP
Unicast Advertisements from A and B
MAC (H1) FCF-MAC (A)
H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A
MAC (H2) FCF-MAC (B)
MAC(H2) MAC(H2)
FCF-MAC(A) FCF-MAC(B)
H2’s FCF list: FCF not meeting
Mini-jumbo Mini-jumbo
Advertisement (FIP) Advertisement (FIP) FCF-MAC(A) [J] capability of ENode
[S=1, F=1, Priority, FC-MAP, [S=1, F=1, Priority, FC-MAP, FCF-MAC(B) [J] does not reply
FCF-MAC(A), Switch_Name, FCF-MAC(B), Switch_Name,
Fabric_Name, Capability, Fabric_Name, Capability,
Other] Other]
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74
© 2006, Cisco Systems, Inc. All rights reserved. 37
Presentation_ID.scr
FIP
FLOGI Request
MAC (H1) FCF-MAC (A)
H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A
MAC (H2) FCF-MAC (B)
FCF-MAC(A) FCF-MAC(B)
MAC (H2) MAC(H2)
Capability agreed
FLOGI Request (FIP) FLOGI Request (FIP)
during discovery
[FC Header, FLOGI data, [FC Header, FLOGI data,
Proposed MAC’(H2)] Proposed MAC’’(H2)]
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75
FIP
FLOGI LS_ACC
MAC (H1) FCF-MAC (A)
H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A
MAC (H2) FCF-MAC (B)
MAC (H2) MAC(H2)
FCF-MAC(A) FCF-MAC(B)
ENode uses MAC address
in FIP FLOGI LS_ACC as the
FLOGI LS-ACC (FIP) FLOGI LS-ACC (FIP) VN_Port MAC address for the
[FC Header, LS_ACC data, [FC Header, LS_ACC data, FC-ID contained in the FLOGI data
Approved MAC(H2)’] Approved MAC(H2)’’] for subsequent FCoE frames
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76
© 2006, Cisco Systems, Inc. All rights reserved. 38
Presentation_ID.scr
FCoE
Data Transfer
MAC (H1) FCF-MAC (A)
H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A
MAC (H2)
MAC (H2)’ FCF-MAC (B)
MAC (H2)’’
FCF-MAC(A) FCF-MAC(B)
All subsequent FCoE frames use
MAC(H2)’ MAC(H2)’’ granted MAC address and assigned
Fibre Channel Frame Fibre Channel Frame FC-ID
(FCoE) (FCoE) FIP frames continue to use MAC(H2)
For SPMA, MAC(H2)’ = MAC(H2)’’ = MAC(H2)
[FC SOF, FC Header, FC [FC SOF, FC Header, FC For FPMA, MAC(H2)’ and MAC(H2)’’ use
Data, FC CRC FC EOF] Data, FC CRC, FC EOF] FC-IDs as low order 24 bits
and FC-MAP for upper 24 bits
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77
The Most Asked Question:
Is FCoE Routable?
FCoE
1, 2, 4, (8), 10 Gbps 1, 10 . . . Gbps 10, 20 Gbps
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78
© 2006, Cisco Systems, Inc. All rights reserved. 39
Presentation_ID.scr
Is FCoE Routable?
Most folks mean “Is FCoE IP-routable”
The answer is NO, there is no IP layer in FCoE
This was a design goal to keep FCoE simple
FC-BB-5 contains FCIP that is “IP-routable”
FCoE is FC-routable
FCoE switches may forward FC frames across different
Ethernet clouds
FCoE switches may forward FC frames over the Internet using FCIP
FCoE FCIP FCIP FCoE
IP Cloud
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79
CNA: Converged Network Adapter
LAN HBA CNA
10GbEE
10GbEE
10GbE
10GbE
HBA
HBA
Link Link Link
Fibre Channel
Fibre Channel
Ethernet
Ethernet
Ethernet
PCIe PCIe PCIe
Ethernet Fibre Channel Ethernet Fibre Channel
Drivers Drivers Drivers Drivers
Operating System Operating System
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80
© 2006, Cisco Systems, Inc. All rights reserved. 40
Presentation_ID.scr
View from Operating System
Standard drivers
Same management
Operating system sees:
Dual-port, 10 Gigabit
Ethernet adapter
Dual-port, 4 Gbps Fibre
Channel HBAs
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81
Open-FCoE Software
HBA FCoE
HBA Mgmt Plane FCoE Mgmt Plane
Linux Kernel Linux Kernel
File System layers File System layers
SCSI Layer
SCSI Layer
OpenFC Layer
HBA Driver FCoE Layer
Net Device
HBA Ethernet Driver
Ethernet
Fibre Server Ethernet Server
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82
© 2006, Cisco Systems, Inc. All rights reserved. 41
Presentation_ID.scr
Open-FCoE Software: How to Get It
Open source project
Open-FCoE.org—source Git trees
Or Open-FCoE source package—TBD
Install a Linux Red Hat EL5, Fedora Core 7,
or SuSE 10 distribution
Update kernel to 2.6.23 or later
Install: see Quick Start Guide at open-fcoe.org
Use switch, soft-target, or gateway
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83
Wireshark
Once known as Ethereal
Captures and displays network traffic
Available from: http://wireshark.org/
Sample trace file
/common/openfc/traces/fcoe-t11.cap
Use tcpdump to capture
tcpdump –i eth0 –s 0 –w /tmp/fcoe.cap
Screenshots/demo
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84
© 2006, Cisco Systems, Inc. All rights reserved. 42
Presentation_ID.scr
Wireshark Screenshot
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85
Section 4
Case Studies
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86
© 2006, Cisco Systems, Inc. All rights reserved. 43
Presentation_ID.scr
Current Data Center Environment
LAN Core
SAN-A SAN-B
Distribution
MDS 9500
4
Cisco Catalyst® 6509
4 x 4G FC 1 x 10 GE STP BLK
POD 1 POD N Access
MDS 9500
Cisco Catalyst 6509
NIC Teaming
Active/Standby
Discrete
1 GE NICs
and
FC HBA
Server Cabinet 1 Server Cabinet N Server Cabinet 1 Server Cabinet N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87
Top-of-Rack Consolidated I/O
I/O Consolidation at Access
LAN Core
SAN-A SAN-B
8
Distribution
8 MDS 9500
Cisco Catalyst 6509
STP BLK
POD 1 POD N Access
Nexus 5000
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88
© 2006, Cisco Systems, Inc. All rights reserved. 44
Presentation_ID.scr
Top-of-Rack Consolidated I/O
Ethernet Host Virtualizer
LAN Core
SAN-A SAN-B
8
Distribution
8 MDS 9500
Cisco Catalyst 6509
Ethernet Host Virtualizer
Active/Active
POD 1 POD N Access
Nexus 5000
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89
Top-of-Rack Consolidated I/O
VSS Support at Aggregation
LAN Core
SAN-A SAN-B
8
VSS Distribution
MDS 9500
8
Cisco Catalyst 6509
VSS Support
at Aggregation
POD 1 POD N Access
Nexus 5000
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90
© 2006, Cisco Systems, Inc. All rights reserved. 45
Presentation_ID.scr
Physical Topology—40 Servers, ToR
LAN SAN-A SAN-B
4x10 GE
Ports
4x10 GE 6x4 GFC
Ports 6x4 GFC
Nexus 5000
40x10 GE Ports
per Switch
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91
Physical Topology—200 Servers
40 4GFC
SAN-A SAN-B
LAN
4x10 GE
Ports 6x4 GFC
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92
© 2006, Cisco Systems, Inc. All rights reserved. 46
Presentation_ID.scr
Blade Servers with
Copper Pass-Through
DC Core
MDS MDS
SAN-Core SAN-Core
8
Distribution
MDS 9500
8 Cisco Catalyst 6509
POD 1 POD N Access
Nexus Family
Blade Server
Copper
Pass-Through
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Blade Server 1 Blade Server N Blade Server 1 Blade Server N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93
Blade Servers with
Ethernet Switches
DC Core
MDS MDS
SAN-Core SAN-Core
8
Distribution
MDS 9500
8 Cisco Catalyst 6509
POD 1 POD N Access
Nexus Family
Blade Server
Ethernet-Only
Switch
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Blade Server 1 Blade Server 1 Blade Server 1 Blade Server 1
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 94
© 2006, Cisco Systems, Inc. All rights reserved. 47
Presentation_ID.scr
Physical Topology—
20 Blade Server, ToR
LAN SAN-A SAN-B
2x10 GE
Ports
2x10 GE 4x4 GFC
Ports 4x4 GFC
Nexus 5000
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 95
Topology—100 Blade Servers, ToR
LAN SAN-A SAN-B
4x10 GE
Ports
8x4 GFC
8x4 GFC
4x10 GE
Ports
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 96
© 2006, Cisco Systems, Inc. All rights reserved. 48
Presentation_ID.scr
Access Nexus 5000
Distribution Nexus 7000
LAN Core
SAN-A SAN-B
8
Distribution
MDS 9500
8 Nexus 7000
Layer 2 Multipath
POD 1 POD N Access
Nexus 5000
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 97
Consolidation in the Distribution Layer
LAN Core
SAN-A SAN-B
8
Distribution
MDS 9500
8 Nexus 7000
Layer 2 Multipath
POD 1 POD N Access
Nexus 5000
Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 98
© 2006, Cisco Systems, Inc. All rights reserved. 49
Presentation_ID.scr
Conclusions
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 99
Challenges
FCoE redefines consolidated scenarios
Ethernet switch manufacturers will try to enter the FC
switching market
FC switch manufacturers will try to enter the Ethernet
switching market
HBA manufacturers will try to enter the NIC market
NIC manufacturers will try to enter the HBA market
Deep integration with virtualization will take some time
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 100
© 2006, Cisco Systems, Inc. All rights reserved. 50
Presentation_ID.scr
Web Pointers
PCI Express
http://en.wikipedia.org/wiki/Pci_express
IEEE 802.3
http://standards.ieee.org/getieee802/802.3.html
Improvements to Ethernet
http://www.nuovasystems.com/EthernetEnhancements-Final.pdf
IEEE 802.1 activities
http://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf
http://www.ieee802.org/1/files/public/docs2007/new-cm-pelissier-enabling-block-storage-0705-v01.pdf
http://www.ieee802.org/1/files/public/docs2007/au-ko-fabric-convergence-0507.pdf
http://www.ieee802.org/1/pages/802.1au.html
http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf
FCoE
http://www.fcoe.com/
http://www.t11.org/
http://www.open-fcoe.org/
http://www.fibrechannel.org/OVERVIEW/FCIA_SNW_FCoE_WP_Final.pdf
TRILL
http://www.ietf.org/html.charters/trill-charter.html
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 101
Thank You
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 102
© 2006, Cisco Systems, Inc. All rights reserved. 51
Presentation_ID.scr
Recommended Reading
Continue your Cisco Live
learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books
Available Onsite at the Cisco Company Store
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 103
Complete Your Online
Session Evaluation
Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 104
© 2006, Cisco Systems, Inc. All rights reserved. 52
Presentation_ID.scr
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 105
© 2006, Cisco Systems, Inc. All rights reserved. 53
Presentation_ID.scr