You are on page 1of 46

Session_ID

Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1

Data Center Facilities
Consideration in Designing
and Building Networks

BRKDCT-2867

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1
Presentation_ID.scr
Agenda
ƒ DC Standard
ƒ DC Challenges
ƒ Cooling Issues and Solutions
ƒ Cabling Issues and Solutions
ƒ Power Issues and Solutions
ƒ DC Physical Considerations
ƒ Modular Access (example)
ƒ Consolidated I\O Architecture with Virtualization
ƒ Summary
ƒ Q&A

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Data Center Standard

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2
Presentation_ID.scr
DC Standard

ANSI/TIA-942
Telecommunications Infrastructure
Standard for Data Centers

TIA/EIA TIA/EIA TIA/EIA TIA/EIA
568 569 606 607
Copper & Fiber Pathways Administration Grounding
Cabling & Spaces & Bonding

ASHRAE IEEE 1100
Uptime ITE Grounding
Cooling/HVAC
Institute

TIA: Telecommunications Industry Association http://www.tiaonline.org/
Uptime Institite: http://uptimeinstitute.org/
Government work on server and DC Energy Efficiency:
http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency
Session_ID
© 2008 Cisco Systems, Inc. All rights reserved.
Presentation_ID Cisco Public 5

TIA-942 Logical Layout

Offices, Operations Access Access
Providers Entrance Room
Center, Support (Carrier Equip & Providers
Rooms Demarcation)

Horizontal Cabling
Backbone Cabling
Computer
Main Dist Area
Telecom Room Backbone Cabling (Routers/Backbone
Room
(Office & Operations LAN/SAN Switches,
Center LAN Switches) PBX, M13 Muxes) Backbone Cabling

Horiz Dist Area
(LAN/SAN/KVM
Switches)

Horizontal Cabling Horiz Dist Area Horiz Dist Area Horiz Dist Area
(LAN/SAN/KVM (LAN/SAN/KVM (LAN/SAN/KVM
Switches) Switches) Switches)
Zone Dist Area
Horizontal Cabling Horizontal Cabling Horizontal Cabling Horizontal Cabling

Equipment Dist Equipment Dist Equipment Dist Equipment Dist
Area Area Area Area
(Rack / Cabinet) (Rack / Cabinet) (Rack / Cabinet) (Rack / Cabinet)

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3
Presentation_ID.scr
Example Data Center Layout

Main Distribution Area

Horizontal Distribution Area

Equipment Distribution Area

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Zone Distribution Area (ZDA)

LEGEND:
EDA
(server cabinet)
Horizontal Cabling
(in hot aisles)
Patch Cord
(to server)
ZDA
(Zone Outlet or
Consolidation Point)
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4
Presentation_ID.scr
Data Center Tiering
Excerpt from TIA-942 Standard

Tier II Tier III
Tier I Tier IV
Redundant Concurrently
Basic Fault Tolerant
Components Maintainable
Site Availability 99.671% 99.749% 99.982% 99.995%
Downtown(Hours/Year) 28.8 22.0 1.6 0.4
Operations Center Not Required Not Required Required Required
Redundant Access
Not Required Not Required Required Required
Provider Services
Redundant Backbone
No No Yes Yes
Pathways
Redundant Horizontal
No No No Optional
Cabling
UPS Redundancy N N+1 N+1 2N
Gaseous Suppression Clean Agents Clean Agents
No No
System FM200/Intergen FM200/Intergen
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

Data Center
Challenges

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5
Presentation_ID.scr
Problem Statement
Systems Entering Data Centers Have Changed;
Most Legacy Server Environments Lack Sufficient
Infrastructure to Gracefully Handle Them
ƒ Blade servers ƒ Storage devices
80-84 severs in cabinet Large footprints
30kW of Power 1500, 2000, 2500+ lbs.
ƒ 1U systems 700, 900, 1100+ kgs.
Greater port densities
Greater heat output
More weight

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

Data Center Environmental Challenges
Typical Shortcomings

ƒ Cooling
ƒ Structured cabling
ƒ Power
ƒ Structural loading

The nature of Data Center
infrastructure makes it
challenging to find solutions
that don’t spawn other
problems

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6
Presentation_ID.scr
Data Center Inefficiencies

ƒ Uptime is the ultimate goal, as a result……
ƒ Data Centers are over-planned
To provide 24X7 availability
Allow for future growth
90% of corporate Data Centers have more cooling capacity
than required (Uptime Institute)
ƒ Inefficient equipment deployment
Server performance, one application per server
72% of cooling bypasses the computing equipment entirely
(Uptime Institute)
ƒ Inherent power inefficiencies
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Cooling Issues and
Solutions

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7
Presentation_ID.scr
Cooling Issues and Solutions

ƒ Today’s products 2000 – 2010 Product Heat Density Trend Chart
are hotter than
yesterday’s
ƒ Tomorrow’s
products will be
hotter than today’s
ƒ Data Center
Managers prefer to
tightly install
equipment to fully
utilize cabinet space

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

Cooling Issues and Solutions
ƒ Ambient temperatures
50, 100, 150, 200 watts/sq. ft.
500, 1000, 1600, 2150 watts/ sq. m.
ƒ Hot spots
ƒ Short-cycling of air handlers
Arrhenius Rate Law
For every 18 °F (10 °C) increase in temperature there is a
50% decrease in reliability of electronics
Activation Energy
Rate Constant
EA
-
Kelvin Temperature
K = AeRT
Frequency Factor
or Pre-Exponential The Gas Constant
Factor Mathematical
Session_ID
Quantity, e
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8
Presentation_ID.scr
Cooling Issues and Solutions
Design Solutions
ƒ Spread out heat sources
Increase aisle spaces
Don’t cluster high-density rows

ƒ Use deep plenum below floor and above ceiling
ƒ Orient air handlers perpendicular to rows

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Catalyst Switches in a Cabinet

ƒ Catalyst 6K Switches in a
Cabinet
ƒ Utilize Ducting
ƒ Improve Air Flow
Characteristics

CFD of 2 – Catalyst 6509 CFD of 2 – Catalyst 6509
Switches with ducting Switches without ducting

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9
Presentation_ID.scr
Airflow
Cold Aisle / Hot Aisle

Air Circulation Components Cold Aisle/Hot Aisle Concept

Session_ID Cable Placement Additional Cooling Capacity
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Cooling Issues and Solutions
Design Solutions
ƒ Hot / cold aisles ƒ Ducted air return
ƒ Chimney design

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10
Presentation_ID.scr
Cooling Issues and Solutions -
Architectures

Central Air Handling Unit Computer Room Air Handler In-Row Air Handler
(CAHU) (CRAH) (IRAH)

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

Cooling Issues and Solutions - A Hybrid
Approach to Cooling
ƒ Most Data Centers will have a mix of heat densities and
therefore cooling solutions
IT Refreshes happen every 2-4 years resulting in a mix of distributed
IT assets
IT management of blades and storage deployments is often in clusters
Concentrated high density loads
Older server assets may be well served by room cooling units
Leverage existing cooling assets to maintain room conditions.

Row-
oriented Rack-
d
oriented

Room-
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. oriented
Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11
Presentation_ID.scr
Cooling Issues and Solutions –
Alternative cooling architectures w/ power considerations

Method Application Density
Traditional room- Low density 1-5kW per rack
oriented raised floor Very flexible
cooling
In-row Medium density 3-15kW per rack
General use

In-row with hot aisle Very high density 10-25kW per rack
containment Targeted zones
Assured redundancy

Rack-coupled Very high density specific 20-45kW per rack
racks
Mix of very high and low
density

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

Cooling Issues and Solutions
Design Solutions

ƒ Deploy redundant infrastructure
ƒ Supplemental cooling solutions
Overhead cooling units
Water-cooled cabinets
Auto-adjusting tile dampers
Cool Door Technology
In row cooling

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12
Presentation_ID.scr
Cooling Issues and Solutions
Operational Solutions

ƒ Distribute hottest systems
ƒ Limit clustering of like systems
ƒ Install air dams
ƒ Streamline cabling
ƒ Maintain static pressure
ƒ Employ temperature monitoring tools
ƒ Install air conservation system on floor

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Cooling Issues and Solutions – In Row
Cooling (iRAH)
ƒ Elimination of mixing enables a
predictable cooling pattern
IT Racks
Capture heat at the source and
Neutralize
ƒ Capable of cooling high densities > Cooling
30 kW per rack
Close Coupled to Heat Load
Thermal Containment Options
ƒ Dynamic fan control matches heat
removal to heat generation
Redundancy – Reduce Power
Virtualization – Ramp-up and Down
to meet thermal demand
ƒ Reduced deployment cycle and
Cost through use of modular
scaleable components
Build out in Zones.
Cool only what you need to
ƒ Increased data center efficiency
Front View IRAH – In row air handler
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13
Presentation_ID.scr
Cooling Issues and Solutions - Hot Aisle
Containment Systems (HACS)
Hot Aisle Ceiling
Tiles/Cable Trough
Seals in hot air, prevents
mixing with room air
ƒ High Density Zones
ƒ Supports InRow products
ƒ Hot air scavenging system
Ducted Return / Free Supply
ƒ Optimize InRow Cooling
Increase Efficiency
Improve Predictability
Use at any density Chamber Doors
InfraStruXure InRow RC
Access to hot aisle, locks for
In-Row Air Conditioner Cools hot security
chamber air

InfraStruXure High Density
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

Data Center Cooling Solutions Summary
1. Conduct a cooling checkup/survey.
2. Route data cabling in the hot aisles
and power cable in the cold aisles.
3. Control air path leaks and manage
cabling system pathways.
4. Remove obstructions below raised
floor and seal cutouts.
5. Separate blade server cabinets.
6. Implement ASHRAE TC9.9 hot
aisle/cold aisle design.
7. Place CRAC units at the ends of the
hot aisles.
8. Manage floor vents.
9. Install air flow assisting devices as
needed.
10. In extreme cases, consider self-
contained cooling units.

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14
Presentation_ID.scr
Cabling Issues and
Solutions

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Cabling Issues and Solutions

ƒ Insufficient ports
Users “borrow” other connections
Piecemeal fixes

ƒ Chaotic cabling
Restricts air flow
Hinders troubleshooting
Creates unplanned dependencies
Prone to accidental downtime
Mess begets mess

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15
Presentation_ID.scr
Cabling Issues and Solutions
Design Solutions

ƒ Right-size port counts
Set capacity to meet 95% of deployments
Prewire cabinet locations

ƒ Plan cable management
Choose graceful high-density solutions
Strategically deploy wire management

ƒ Employ a distributed physical design

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

Cabling Issues and Solutions
Direct-Connect Design
ƒ One main networking row
ƒ Cabling routed directly to
server cabinet locations
ƒ Excellent for the logical
elements of the network

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16
Presentation_ID.scr
Cabling Issues and Solutions
Direct-Connect Design
ƒ Bad for the physical element
of a network
ƒ Scales poorly
ƒ Prone to cable overlap

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Cabling Issues and Solutions
Distributed Design
ƒ Network substations
ƒ Cabling to all server cabinets
ƒ Subset of cables to the main
network row

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17
Presentation_ID.scr
Cabling Issues and Solutions
Design Solutions
ƒ Distributed Design
ƒ Good for the physical
element of a network
ƒ Scales well
ƒ No cable overlap
ƒ Cable runs are shorter
and better organized
Easier to manage
Less expensive
Less restrictive for air flow

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

Cabling Issues and Solutions: Pathways

ƒ Cable Sizes Growing
Cat 5e 0.157”
Cat 6 0.25”
Cat 6a 0.315”
ƒ Multiple Diverse Routes
ƒ Under the Access Floor
ƒ Above the Racks \ Cabinets
ƒ Mixture of Under & Over the Access Floor

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18
Presentation_ID.scr
Cabling Issues and Solutions:
Pathways and Spaces Under Floor
ƒ Benefits
Pedestals create infrastructure pathways
Utilization of real estate
Cabling is hidden

ƒ Concerns
Could restrict cold airflow
Creating segregated pathways
Accessibility to cables

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

Cabling Issues and Solutions:
Pathways and Spaces Overhead
ƒ Benefits
Alleviates congestion beneath access floor
Creation of segregated pathways
Minimizes restrictions to cold air floor

ƒ Concerns
Requires adequate space above the racks
Infrastructure provisions to support the pathways
Cabling may be exposed

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19
Presentation_ID.scr
Cabling Issues and Solutions

Before After

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

Cabling Issues and Solutions
- Zone Cabling in the DC

ƒ Flexibility to Cable Full Rack Equipment Easily
Mainframes

SAN Equipment

ƒ Ability to make MACs Quickly and Easily

ƒ Reduced Network Downtime When Changes are Required

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20
Presentation_ID.scr
Cabling Issues and Solutions
- Cabling a Cisco Modular Switch

Which way do you cable switch?

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Cabling Issues and Solutions
Operational Solutions

ƒ Neatly route cabling
ƒ Don’t use overly long
patch cords
ƒ Use wire management as
designed

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21
Presentation_ID.scr
Evolution of Ethernet Physical Media –
Impact on Facilities
Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s

10Mb 100Mb 1Gb 10Gb

UTP Cat 3 UTP Cat 5 UTP Cat 5 X2
SFP Fiber SFP+ Cu
SFP+ Fiber
Cat 6/7 ??

Power Transceiver
Technology Cable Distance (each side) Latency (link)

SFP+ CU
Twinax 10m ~0.1W ~0.1μs
Copper

SFP+ USR MM OM2 10m
1W ~0
ultra short reach MM OM3 100m

SFP+ SR MM 62.5μm 82m
1W ~0
short reach MM 50μm 300m

Cat6 55m ~8W 2.5μs
10GBASE-T Cat6a/7 100m ~8W 2.5μs
Cat6a/7 30m ~4W 1.5μs

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

Power Issues and
Solutions

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22
Presentation_ID.scr
Power Issues and Solutions
ƒ Growing demand for circuits
From: (2) 120V 20 amp circuits -Two poles
To: (2) 208V 30 amp circuits - Four poles

ƒ Growing demand for capacity
ƒ Today’s electrical loads exceed yesterday’s designs
Eliminates redundancy (N+1)
Exceeds total building capacity

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Power Issues and Solutions
Design Solutions
ƒ Prewire all cabinet locations
Set capacity to meet 95% of deployments

ƒ Provide redundant power
Feed from multiple PDUs
Keep all component loads below 50%

ƒ Provide capacity to accommodate growth
ƒ Employ a distributed physical design

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23
Presentation_ID.scr
Power Issues and Solutions
Direct-Connect Design
ƒ Electrical conduits routed
directly to all cabinet
locations
Restrict airflow
Complicate troubleshooting
Create unintentional physical
dependencies

ƒ Overlapping conduits
ƒ In large Data Centers
conduit lengths can be
excessive

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

Power Issues and Solutions
Distributed Design
ƒ Fewer overlapping
conduits
ƒ Improved airflow
ƒ Reduced costs to modify
shorter conduits
ƒ Reduced risk of multiple
outages from a physical
accident

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24
Presentation_ID.scr
Power Issues and Solutions
Electrical/Network Substation
ƒ Remote Power Panels
Installed back-to-back
Each fed from a different PDU

ƒ Network Patching Fields

Cabinets Floor Tiles Ports per Tile
1 15 24 Fiber, 18 Copper
2 24-30 36 Fiber, 36 Copper
5 50 48 Fiber, 48 Copper

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

Power Issues and Solutions
Operational Solutions
ƒ Use hosts with redundant power supplies
ƒ Limit systems with odd-numbered power cords
ƒ Consider amp-reading power strips

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25
Presentation_ID.scr
Data Center Power Considerations Summary
1. See Cooling top 10 Steps!
2. Standardize on rack SOE
3. Implement scalable UPS systems
4. Increase Voltage
5. Targeted higher UPS loading
6. Investigate power efficiency
7. Load balance
8. Limit branch circuit proliferation
9. Monitor power
10. Manage and target power based
on monitoring benchmark

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

DC Physical
Considerations

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26
Presentation_ID.scr
Data Center Topology
Network Equipment and Zones
Core
Layer

Aggregation
Layer

Access
Layer

DC

Zone
Pod
Pod

Network Rack
HOT AISLE
Server Rack
COLD AISLE
Storage Rack Pod
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Pod Concept
Network Zones and Pods

HOT AISLE

Pod
COLD AISLE

Pod

Sizing
▪Zone: Typically mapped to agg pair size
▪Pod: Typically mapped to access switch pair
▪ Size: determined by distance and density
▪ Cabling distance from server racks to network racks
▪ 100m Copper
▪ 200-500m Fiber
▪ Cabling density: # of servesr per rack and I/Os per server
▪Rack
▪ Server: 6-30 Servers per rack – limited by power
▪ Network: Depends on access model: Modular, ToR or Blade
▪ Storage: special Cabinets
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27
Presentation_ID.scr
Network Equipment Distribution
End of Row End of Row

End of Row
▪Traditionally used
▪Copper from server to access switches Patch panel Patch panel
▪Poses challenges on highly dense server farms
Patch panel Patch panel
▫ Distance from farthest rack to access point X-connect X-connect
server server
▫ Row length may not lend itself well to
switch port density

Network Network
Access Point Access Point
Common Characteristics A-B C-D
▪Typically used for modular access server server
▪Cabling is done at DC build-out
▪Model evolving from EoR to MoR
▪Lower cabling distances (lower cost) Fiber
▪Allows denser access (better flexibility)
▪6-12 multi-RU servers per Rack Copper
▪4-6 kW per server rack, 10Kw-20Kw per network End of Row
rack
▪Subnets and VLANs: one or many per switch.
Subnets tend to be medium and large
Patch panel Patch panel

End of Row (half row) server
Patch panel
X-connect
Patch panel
server
X-connect
▪Use is starting to increase given EoR
challenges
▪Copper from servers to access switches
▪Fiber may be used to aggregate ToR
Network Network
▪It addresses aggregation requirements for Access Point Access Point
ToR access environments A-B C-D
server server
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

Network Equipment Distribution
Top of Rack

ToR Patch panel Patch panel
▪Used in conjunction with dense access Top of Rack Top of Rack
racks(1U servers) server
Patch panel Patch panel
server
X-connect X-connect
▪Typically one access switch per rack
▪Some customers are considering two +
cluster
▪Typically: Network
Aggregation
Network
Aggregation
▪ ~10-15 server per rack (enterprises) Point Point
▪ ~15-30 server per rack (SP) server
A-B A-B
server
▪ Use of either side of rack is gaining traction
▪ Cabling:
▪Within rack: Copper for server to access To network core
switch
▪Outside rack (uplink):
▪Copper (GE): needs a MoR model for fiber
aggregation
Patch panel Patch panel
▪Fiber (GE or 10GE):is more flexible and also
requires aggregation model (MoR) Top of Rack Top of Rack
Patch panel Patch panel
▪Subnets and VLANS: Top of Rack
X-connect X-connect
Top of Rack
▪ one or many subnets per access switch server server
▪ Subnets tent to be small

Network Network
Access Point Access Point
A-B C-D

server server
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28
Presentation_ID.scr
Network Equipment Distribution
Blade Chassis

End of Row (Switch to Switch)
▪Scales well for blade server racks (~3 blade
chassis per rack) Patch panel Patch panel
▪Most current uplinks are copper but the NG
switches will offer fiber Patch panel Patch panel
sw1 sw2 X-connect X-connect sw1 sw2
Blade Chassis Blade Chassis
End of Row (Pass-through)
▪Scales well for pass-through blade racks sw1 sw2 sw1 sw2
▪Copper from servers to access switches Blade Chassis Network Network Blade Chassis
Aggregation Aggregation
Point Point
sw1 sw2 A–B–C-D A–B-C-D sw1 sw2
Blade Chassis Blade Chassis

ToR Patch panel
Patch panel
▪Have not seen it used in conjunction with Top of Rack
blade switches Patch panel Patch panel
▪May be a viable option on pass-through Pass-through X-connect X-connect Pass-through

environments is the access port count is right Blade Chassis Blade Chassis

Pass-through Pass-through
Blade Chassis Network Network Blade Chassis
Aggregation Aggregation
Point Point
Pass-through A–B–C-D A–B-C-D Pass-through
Blade Chassis Blade Chassis

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

Width of Cabinets
24” vs 32” Cabinets?

ƒ For Each Scenario in this example
96in x 42in footprint
Total of 16.8KW of Power
Total of 48 Servers
Air Flow delivered is the same
ƒ 32in 45RU Cabinet Advantages
Reduced Static pressure due to proper cable
management with more than 4-5 2RU servers per
cabinet
Vertical patch panels minimizes patch cord lengths,
number of sizes, & increases usable RU spacing
Power cables and Network cables have good
separation
Additional room for cabling thus reducing accidental
downtime
ƒ Footprint only increases with the number
of servers in a 24” cabinet.
17 – 2RU Servers
34 – 1RU Servers
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29
Presentation_ID.scr
Differences between Cabinets and Racks

• Aesthetics
• Security
• Additional Usable Cable Management
Area & Flexibility
• Vertical Patch Panels
• Footprint
• Additional Cooling Options
• Cool Door
• Convert back and forth from Server to
Switch cabinet

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

Risks to Consider in Capacity Planning
Legacy Server High-Density Server
Power per Server 2-3 kW per rack > 20 kW per rack

Power per Floor Space 30-40 W/ft² 700-800 W/ft²

Cooling Needs—chilled airflow 200-300 cfm 3,000 cfm

Source: Gartner 2006

20,000 ft²

Legacy DC designed to accommodate
800kW 2-3kW per Rack

+33%

Introducing 1/3 high-density infrastructure into
a legacy facility has cost, power, weight, and
100-200 Racks cooling implications *Peripheral DC costs considered
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30
Presentation_ID.scr
Modular Access / End
Row Example

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

End of Row Example (Modular Access)
The Challenge
Data Center
Core
ƒ ~4000 Servers
Aggregation
ƒ Server LAN A&B Connectivity Aggregation
Module
Layer
ƒ Servers 1Gb Connectivity
ƒ Uplinks 10Gb Connectivity
ƒ Utilize Catalyst 6509 Switches 2
2 2 2 2
2 2 2

ƒ Core, Agg and Access Design 48 Switches

ƒ SAN Cabinets
4000 Servers
ƒ Mainframe\Midrange Cabinets
ƒ 450 Watts per Server Access Layer

ƒ 5.5 kW per Switch Cabinet
ƒ 4 kW per SAN\Midrange Cabinet 10 GbE
GE
ƒ“POD” Concept Design

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31
Presentation_ID.scr
Data Center Sizing
Enterprise Modular Access: 6509

Enterprise
Servers 1500 < x < 4000 Core:
8000 6509: 16 10GE ÷ 4 ports per LC = 4 LCs
Ports*
2 LCs for cross connectivity
Switch Types 6509
Uplinks: based on DC outbound requirements
Core 2: 6509
Aggregation:
Switches LCs: 6 4x10GE
6509: 96x10GE ÷ 4 ports per LC = 24 LCs
4: 6509s
2 LCs for cross connectivity
Aggregation LCs: 26 4x10GE

Switches Outbound Capacity: 80GE 26 LCs ÷ 8 Slots = 3.25 ~ 4 chassis
Oversubscription: 96:16 4 chassis x 4 uplinks = 16 10GE ports
Access 24: 6509 Access: 8000 ÷ 336 = 23.8 ≈ 24 chassis
Switches Oversubscription: 8.4:1 4 10GE uplinks per chassis = 96 10GE

Maximum Capacity Oversubscription: 336:40 ~ 8.4:1

Access: 336 x 24 = 8064 ports ~ 4032 Servers
4 10GE uplinks per chassis = 8 10GE
Oversubscription: 336:40 ~ 8.4:1

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

End of Row Example (Modular Access)
The Layout and Sample Solution
12 Server “PODs” Servers: 4032
Consists of the following: 6509 Switches: 30
4 Switch Cabinets for LAN & SAN Server\Switch Cabinets: 399
32 Server Cabinets Midrange\SAN Cabinets Allotted For: 124
12 Servers per Server Cabinet
Core 1 Core 2

Agg1 Agg2 Agg3 Agg4

Acc1 Acc2 Acc11 Acc12 Acc13 Acc14 Acc23 Acc24

6 Pair 6 Pair
Switches Switches

336 Servers 336 Servers 336 Servers 336 Servers
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32
Presentation_ID.scr
End of Row Example (Modular Access)
The Data Center Layout Total White Space:
14,400 sqFt

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

End of Row Example (Modular Access)
The “POD” with EDA Breakout

Equipment Distribution
Area (EDA)
Single “POD”

Acc1 Acc2

336 Servers

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33
Presentation_ID.scr
End of Row Example (Modular Access)
EDA Application Photos

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

End of Row Example (Modular Access)
The “POD” with HDA Breakout

Horizontal Distribution
Area (HDA)
Single “POD”

Acc1 Acc2

336 Servers

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34
Presentation_ID.scr
End of Row Example (Modular Access)
The “POD” with HDA Breakout

Horizontal Distribution
Area (HDA)
Single “POD”

Acc1 Acc2

336 Servers

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

End of Row Example (Modular Access)
HDA Application Photos

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35
Presentation_ID.scr
End of Row Example (Modular Access)
Main Distribution Area (MDA)
Core 1 Core 2
Additional Equipment:
Core Routing\Firewalls

Agg1 Agg2 Agg3 Agg4
LAN Appliances
SAN Directors

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

End of Row Example (Modular Access)
Main Distribution Area (MDA)
Additional Equipment
Core 1 Core 2
Core Routing\Firewalls
LAN Appliances
SAN Directors

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36
Presentation_ID.scr
End of Row Example (Modular Access)
CFD Analysis
Designed in a Hot –
Cold Architecture

12 - 20 Ton CRAC Units
Outside

12 - 30 To CRAC Units
Inside

Utilizing Ceiling plenum
for return air

All Perforated Tiles at
25% Open

Peak Temp was 114°

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

Consolidated I/O
Architecture with
Virtualization

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37
Presentation_ID.scr
Increased Efficiency, Simpler Operations

Mgmt
Network

Front-End
Network
Backup
Network

Unified
Fabric
Storage Back-End
Network Network

Unified Fabric and I/O

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

Fibre Channel over Ethernet (FCoE)

ƒ A method for a direct mapping of FC frames over
Ethernet
Seamlessly connects to FC networks Æ Extends FC across the datacenter
over the Ethernet
FCoE appears as FC to the host and the
network
Preserves current FC infrastructure 10Gb Etherne
t
and management Pipe
FC
FC frame is unchanged
Can operate over standard switches
(with jumbo frames)
Priority Flow Control guarantees no-drops
Mimics FC credit-buffer system, avoids TCP
Does not require expensive off-loads

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38
Presentation_ID.scr
VM-Optimized Services
ƒ Enables convergence of SAN A SAN B
LAN LAN
multiple traffic types
Virtual Machines LAN
Virtual Machines SAN
Hypervisor Mgmt LAN
Virtual Infrastructure Services LAN

ƒ Scales VM LAN performance
Increase I/O bandwidth
Increase VM density VMotion

ƒ Accelerates Virtual
Infrastructure Services
Live VM migrations via VMotion and 10 GbE
DRS features 10 GbE DCE

Enable additional services Fibre Channel

10 GbE
FCoE/DCE

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

Virtualized Server Environment – Unified Fabric
ToR Deployment
40 Servers per Cabinet Pair VMs per Server: 10
2 Nexus 5020 per Cabinet Pair Servers: 400
10 Cabinet Pairs Nexus 7000 Switches: 2
Nexus 5000 Switches: 10
MDS Switches: 2

Agg1 Agg2

Acc1 Acc2 Acc1 Acc2

10 CP

40 Servers 40 Servers

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39
Presentation_ID.scr
Virtualized Server Environment – Unified Fabric
A Single “POD” Using Virtualized Model

ƒ 24 Cabinets Total
• 10 Server Cabinet Pairs (20 Total)
• 4 Switch Cabinets

ƒ Over Head Fiber Optic Cabling Only
ƒ 400 Servers with 10 Virtualized Each (4000
Total Virtual Servers)

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

Virtualized Server Environment – Unified Fabric
The Cabinet Pair Breakout

ƒ Cross Patching
between Cabinets
ƒ 20 Servers per Cabinet
X 2 Cabinets (40 Total)
ƒ 40 Connections per
Switch
ƒ Each Switch Uplinks
• 4 – LAN A \ 4 – LAN B
• 4 – SAN A \ 4 – SAN B

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved. 40
Presentation_ID.scr
Unified Fabric - Consolidated I/O Extends
Benefits of ToR
Consolidate LAN I/O Consolidate SAN I/O
• Consolidate multiple LAN adapters • No need for SAN-specific adapters
into two CNAs • Reduce SAN switches and cables
• Consolidate multiple cables into two • Unified I/O switches connect to
paths existing SAN infrastructure

Unconsolidated I/O Consolidated I/O

2x SAN
Consolidated I/O Benefits 2x Consolidated
2x LAN
• CapEx Savings up to 30%
• Cable reduction of 50% or
more for ToR and EoR
designs
• Potential power savings

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

Virtualization Impact on Critical Facilities.

ƒ Power Needed per Server has increased
ƒ Could be up to 16KW for this cabinet
ƒ Need for Supplemental Cooling required
ƒ Weight of Equipment on Raised Floor
ƒ All Copper Cabling is Contained with the Two Cabinets
ƒ Cable reduction
ƒ Fiber Optic Cabling for the Uplinks
The need for quality Fiber Cabling has increased

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved. 41
Presentation_ID.scr
Summary

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

Data Center Environmental Challenges
Typical Shortcomings
ƒ Cooling
ƒ Structured cabling
ƒ Power
ƒ Structural loading

The nature of Data Center
infrastructure makes it
challenging to find solutions
that don’t spawn other
problems

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84

© 2006, Cisco Systems, Inc. All rights reserved. 42
Presentation_ID.scr
Data Center Environmental Challenges
Universal Solutions
ƒ Physically Distribute
Equipment
Power & Cooling
ƒ Right-size infrastructure
(modularity)
ƒ Virtualize
ƒ Network Architecture and
Facilities Dependencies

Use pools of servers and
storage, controlled by the
network, to provide Data
Center resources.
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85

Universal Solutions
Service Oriented Data Center (SODC)
ƒ More efficient use of computing resources
ƒ Shared computing among multiple machines
Built-in redundancy
Individual servers are expendable

ƒ Devices can be managed on a by-cabinet basis

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86

© 2006, Cisco Systems, Inc. All rights reserved. 43
Presentation_ID.scr
Q and A

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87

Recommended Reading
Topics
ƒ Site selection
ƒ Data Center sizing and layout
ƒ Physical design
ƒ Large-scale server moves
ƒ Remote monitoring
ƒ Change management

Includes downloadable
design template

Available Onsite at the Cisco Company Store
Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88

© 2006, Cisco Systems, Inc. All rights reserved. 44
Presentation_ID.scr
Recommended Reading
ƒ Continue your Networkers at Cisco
Live learning experience with further
reading from Cisco Press
ƒ Check the Recommended Reading
flyer for suggested books

ƒ Cisco Press
Data Center Fundamentals

Session_ID
Available Onsite at the Cisco Company Store
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89

Additional Resources

ƒ URLs
6500 Cabinet Information
http://wwwin.cisco.com/dss/isbu/6500/enviro/index.shtml
Panduit
http://www.panduit.com/default.asp
TIA – Telecommunications Industry Association
http://www.tiaonline.org/
ASHRAE – American Society of Heating, Refrigerating and Air-Conditioning Engineers
http://www.ashrae.org/
Uptime Institute
http://uptimeinstitute.org/
Government work on server and DC Energy Efficiency:
http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90

© 2006, Cisco Systems, Inc. All rights reserved. 45
Presentation_ID.scr
Complete Your Online
Session Evaluation

ƒ Win fabulous prizes; give us
your feedback
ƒ Receive ten Passport Points
for each session evaluation
you complete
ƒ Go to the Internet stations
located throughout the
Convention Center to complete
your session evaluation
ƒ Winners will be announced
daily at the Internet stations

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91

Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92

© 2006, Cisco Systems, Inc. All rights reserved. 46
Presentation_ID.scr