You are on page 1of 46

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

1

Data Center Facilities Consideration in Designing and Building Networks

BRKDCT-2867

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

1

Agenda
DC Standard DC Challenges Cooling Issues and Solutions Cabling Issues and Solutions Power Issues and Solutions DC Physical Considerations Modular Access (example) Consolidated I\O Architecture with Virtualization Summary Q&A

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

Data Center Standard

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

4

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

2

DC Standard

ANSI/TIA-942
Telecommunications Infrastructure Standard for Data Centers

TIA/EIA 568
Copper & Fiber Cabling

TIA/EIA 569
Pathways & Spaces

TIA/EIA 606
Administration

TIA/EIA 607
Grounding & Bonding

ASHRAE
Cooling/HVAC

Uptime Institute

IEEE 1100
ITE Grounding

TIA: Telecommunications Industry Association http://www.tiaonline.org/ Uptime Institite: http://uptimeinstitute.org/ Government work on server and DC Energy Efficiency:
Session_ID Presentation_ID

http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

5

TIA-942 Logical Layout
Offices, Operations Center, Support Rooms
Horizontal Cabling

Access Providers

Entrance Room
(Carrier Equip & Demarcation)

Access Providers

Backbone Cabling

(Office & Operations Center LAN Switches)

Telecom Room

Backbone Cabling

(Routers/Backbone LAN/SAN Switches, PBX, M13 Muxes)

Main Dist Area

Computer Room
Backbone Cabling

Horiz Dist Area
(LAN/SAN/KVM Switches)

Horizontal Cabling

Horiz Dist Area
(LAN/SAN/KVM Switches)

Horiz Dist Area
(LAN/SAN/KVM Switches)

Horiz Dist Area
(LAN/SAN/KVM Switches)

Zone Dist Area
Horizontal Cabling

Horizontal

Cabling

Horizontal

Cabling

Horizontal

Cabling

Equipment Dist Area
(Rack / Cabinet)

Equipment Dist Area
(Rack / Cabinet)

Equipment Dist Area
(Rack / Cabinet)

Equipment Dist Area
(Rack / Cabinet)

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

6

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

3

Example Data Center Layout

Main Distribution Area

Horizontal Distribution Area

Equipment Distribution Area

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

7

Zone Distribution Area (ZDA)

LEGEND:
EDA (server cabinet) Horizontal Cabling (in hot aisles) Patch Cord (to server) ZDA (Zone Outlet or Consolidation Point)
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Session_ID Presentation_ID

8

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

4

Data Center Tiering
Excerpt from TIA-942 Standard

Tier I Basic

Tier II Redundant Components

Tier III Tier IV Concurrently Fault Tolerant Maintainable

99.671% 99.749% 99.982% 99.995% Site Availability 28.8 22.0 1.6 0.4 Downtown(Hours/Year) Not Required Not Required Required Required Operations Center Redundant Access Not Required Not Required Required Required Provider Services Redundant Backbone No No Yes Yes Pathways Redundant Horizontal No No No Optional Cabling N N+1 N+1 2N UPS Redundancy Clean Agents Clean Agents Gaseous Suppression No No FM200/Intergen FM200/Intergen System
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

9

Data Center Challenges

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

5

Problem Statement
Systems Entering Data Centers Have Changed; Most Legacy Server Environments Lack Sufficient Infrastructure to Gracefully Handle Them
Blade servers
80-84 severs in cabinet 30kW of Power

Storage devices
Large footprints 1500, 2000, 2500+ lbs. 700, 900, 1100+ kgs.

1U systems
Greater port densities Greater heat output More weight

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

Data Center Environmental Challenges
Typical Shortcomings
Cooling Structured cabling Power Structural loading

The nature of Data Center infrastructure makes it challenging to find solutions that don’t spawn other problems
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

12

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

6

Data Center Inefficiencies
Uptime is the ultimate goal, as a result…… Data Centers are over-planned
To provide 24X7 availability Allow for future growth 90% of corporate Data Centers have more cooling capacity than required (Uptime Institute)

Inefficient equipment deployment
Server performance, one application per server 72% of cooling bypasses the computing equipment entirely (Uptime Institute)

Inherent power inefficiencies
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

13

Cooling Issues and Solutions

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

7

Cooling Issues and Solutions
Today’s products are hotter than yesterday’s Tomorrow’s products will be hotter than today’s Data Center Managers prefer to tightly install equipment to fully utilize cabinet space
2000 – 2010 Product Heat Density Trend Chart

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

15

Cooling Issues and Solutions
Ambient temperatures
50, 100, 150, 200 watts/sq. ft. 500, 1000, 1600, 2150 watts/ sq. m.

Hot spots Short-cycling of air handlers Arrhenius Rate Law For every 18 °F (10 °C) increase in temperature there is a 50% decrease in reliability of electronics
Activation Energy Rate Constant
RT K = Ae EA

Kelvin Temperature

Frequency Factor or Pre-Exponential Factor
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved.

The Gas Constant Mathematical Quantity, e
Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

8

Cooling Issues and Solutions
Design Solutions Spread out heat sources
Increase aisle spaces Don’t cluster high-density rows

Use deep plenum below floor and above ceiling Orient air handlers perpendicular to rows

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

17

Catalyst Switches in a Cabinet

Catalyst 6K Switches in a Cabinet Utilize Ducting Improve Air Flow Characteristics

CFD of 2 – Catalyst 6509 Switches with ducting

CFD of 2 – Catalyst 6509 Switches without ducting

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

9

Airflow
Cold Aisle / Hot Aisle

Air Circulation Components

Cold Aisle/Hot Aisle Concept

Session_ID Presentation_ID

Cable Placement
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Additional Cooling Capacity

19

Cooling Issues and Solutions
Design Solutions
Hot / cold aisles Chimney design Ducted air return

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

10

Cooling Issues and Solutions Architectures
Central Air Handling Unit

(CAHU)

Computer Room Air Handler

In-Row Air Handler

(CRAH)

(IRAH)

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

Cooling Issues and Solutions - A Hybrid Approach to Cooling
Most Data Centers will have a mix of heat densities and therefore cooling solutions
IT Refreshes happen every 2-4 years resulting in a mix of distributed IT assets IT management of blades and storage deployments is often in clusters Concentrated high density loads Older server assets may be well served by room cooling units Leverage existing cooling assets to maintain room conditions.

Roworiented d

Rackoriented

Session_ID Presentation_ID

Room© 2008 Cisco Systems, Inc. All rights reserved.

oriented Cisco Public

22

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

11

Cooling Issues and Solutions –
Alternative cooling architectures w/ power considerations Method
Traditional roomoriented raised floor cooling In-row

Application
Low density Very flexible Medium density General use Very high density Targeted zones Assured redundancy Very high density specific racks Mix of very high and low density

Density
1-5kW per rack

3-15kW per rack

In-row with hot aisle containment

10-25kW per rack

Rack-coupled

20-45kW per rack

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

Cooling Issues and Solutions
Design Solutions Deploy redundant infrastructure Supplemental cooling solutions
Overhead cooling units Water-cooled cabinets Auto-adjusting tile dampers Cool Door Technology In row cooling

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

24

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

12

Cooling Issues and Solutions
Operational Solutions Distribute hottest systems Limit clustering of like systems Install air dams Streamline cabling Maintain static pressure Employ temperature monitoring tools Install air conservation system on floor

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

Cooling Issues and Solutions – In Row Cooling (iRAH)
Elimination of mixing enables a predictable cooling pattern
Capture heat at the source and Neutralize
IT Racks

Capable of cooling high densities > 30 kW per rack
Close Coupled to Heat Load Thermal Containment Options

Cooling

Dynamic fan control matches heat removal to heat generation
Redundancy – Reduce Power Virtualization – Ramp-up and Down to meet thermal demand

Reduced deployment cycle and Cost through use of modular scaleable components
Build out in Zones. Cool only what you need to
Session_ID Presentation_ID

Increased data center efficiency
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Front View

IRAH – In row air handler
26

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

13

Cooling Issues and Solutions - Hot Aisle Containment Systems (HACS)
Hot Aisle Ceiling Tiles/Cable Trough
Seals in hot air, prevents mixing with room air

High Density Zones Supports InRow products Hot air scavenging system
Ducted Return / Free Supply

Optimize InRow Cooling
Increase Efficiency Improve Predictability Use at any density
Chamber Doors InfraStruXure InRow RC
In-Row Air Conditioner Cools hot chamber air
InfraStruXure High Density
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Access to hot aisle, locks for security

27

Data Center Cooling Solutions Summary
1. 2. 3. 4. 5. 6. 7. 8. 9. Conduct a cooling checkup/survey. Route data cabling in the hot aisles and power cable in the cold aisles. Control air path leaks and manage cabling system pathways. Remove obstructions below raised floor and seal cutouts. Separate blade server cabinets. Implement ASHRAE TC9.9 hot aisle/cold aisle design. Place CRAC units at the ends of the hot aisles. Manage floor vents. Install air flow assisting devices as needed.

10. In extreme cases, consider selfcontained cooling units.
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

28

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

14

Cabling Issues and Solutions

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

Cabling Issues and Solutions
Insufficient ports
Users “borrow” other connections Piecemeal fixes

Chaotic cabling
Restricts air flow Hinders troubleshooting Creates unplanned dependencies Prone to accidental downtime Mess begets mess

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

15

Cabling Issues and Solutions
Design Solutions Right-size port counts
Set capacity to meet 95% of deployments Prewire cabinet locations

Plan cable management
Choose graceful high-density solutions Strategically deploy wire management

Employ a distributed physical design

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

Cabling Issues and Solutions
Direct-Connect Design One main networking row Cabling routed directly to server cabinet locations Excellent for the logical elements of the network

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

32

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

16

Cabling Issues and Solutions
Direct-Connect Design Bad for the physical element of a network Scales poorly Prone to cable overlap

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

33

Cabling Issues and Solutions
Distributed Design Network substations Cabling to all server cabinets Subset of cables to the main network row

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

34

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

17

Cabling Issues and Solutions
Design Solutions Distributed Design Good for the physical element of a network Scales well No cable overlap Cable runs are shorter and better organized
Easier to manage Less expensive Less restrictive for air flow

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

35

Cabling Issues and Solutions: Pathways
Cable Sizes Growing Cat 5e 0.157” Cat 6 0.25” Cat 6a 0.315” Multiple Diverse Routes Under the Access Floor Above the Racks \ Cabinets Mixture of Under & Over the Access Floor

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

36

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

18

Cabling Issues and Solutions:
Pathways and Spaces Under Floor
Benefits Pedestals create infrastructure pathways Utilization of real estate Cabling is hidden Concerns Could restrict cold airflow Creating segregated pathways Accessibility to cables

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

37

Cabling Issues and Solutions:
Pathways and Spaces Overhead
Benefits Alleviates congestion beneath access floor Creation of segregated pathways Minimizes restrictions to cold air floor Concerns Requires adequate space above the racks Infrastructure provisions to support the pathways Cabling may be exposed

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

38

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

19

Cabling Issues and Solutions
Before After

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

39

Cabling Issues and Solutions
- Zone Cabling in the DC
Flexibility to Cable Full Rack Equipment Easily
Mainframes SAN Equipment

Ability to make MACs Quickly and Easily Reduced Network Downtime When Changes are Required

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

40

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

20

Cabling Issues and Solutions - Cabling a Cisco Modular Switch Which way do you cable switch?

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Cabling Issues and Solutions
Operational Solutions Neatly route cabling Don’t use overly long patch cords Use wire management as designed

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

21

Evolution of Ethernet Physical Media – Impact on Facilities
Mid 1980’s 10Mb
UTP Cat 3

Mid 1990’s 100Mb
UTP Cat 5

Early 2000’s 1Gb
UTP Cat 5 SFP Fiber

Late 2000’s 10Gb
X2 SFP+ Cu SFP+ Fiber Cat 6/7 ??
Transceiver Latency (link)

Technology SFP+ CU Copper SFP+ USR
ultra short reach

Cable

Distance

Power (each side)

Twinax

10m

~0.1W

~0.1μs

MM OM2 MM OM3 MM 62.5μm MM 50μm
Cat6 Cat6a/7 Cat6a/7

10m 100m 82m 300m
55m 100m 30m

1W

~0

SFP+ SR
short reach

1W

~0

10GBASE-T

~8W ~8W ~4W

2.5μs 2.5μs 1.5μs

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

Power Issues and Solutions

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

22

Power Issues and Solutions
Growing demand for circuits
From: (2) 120V 20 amp circuits -Two poles To: (2) 208V 30 amp circuits - Four poles

Growing demand for capacity Today’s electrical loads exceed yesterday’s designs
Eliminates redundancy (N+1) Exceeds total building capacity

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

Power Issues and Solutions
Design Solutions Prewire all cabinet locations
Set capacity to meet 95% of deployments

Provide redundant power
Feed from multiple PDUs Keep all component loads below 50%

Provide capacity to accommodate growth Employ a distributed physical design

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

46

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

23

Power Issues and Solutions
Direct-Connect Design Electrical conduits routed directly to all cabinet locations
Restrict airflow Complicate troubleshooting Create unintentional physical dependencies

Overlapping conduits In large Data Centers conduit lengths can be excessive
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

47

Power Issues and Solutions
Distributed Design Fewer overlapping conduits Improved airflow Reduced costs to modify shorter conduits Reduced risk of multiple outages from a physical accident

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

48

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

24

Power Issues and Solutions
Electrical/Network Substation Remote Power Panels
Installed back-to-back Each fed from a different PDU

Network Patching Fields

Cabinets 1 2 5
Session_ID Presentation_ID

Floor Tiles 15 24-30 50

Ports per Tile 24 Fiber, 18 Copper 36 Fiber, 36 Copper 48 Fiber, 48 Copper

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

Power Issues and Solutions
Operational Solutions Use hosts with redundant power supplies Limit systems with odd-numbered power cords Consider amp-reading power strips

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

25

Data Center Power Considerations Summary
1. See Cooling top 10 Steps! 2. Standardize on rack SOE 3. Implement scalable UPS systems 4. Increase Voltage 5. Targeted higher UPS loading 6. Investigate power efficiency 7. Load balance 8. Limit branch circuit proliferation 9. Monitor power 10. Manage and target power based on monitoring benchmark

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

DC Physical Considerations

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

52

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

26

Data Center Topology
Network Equipment and Zones
Core Layer Aggregation Layer Access Layer

DC Zone Pod Network Rack
HOT AISLE

Pod

Server Rack
COLD AISLE

Storage Rack
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Pod
53

Pod Concept
Network Zones and Pods

HOT AISLE

COLD AISLE

Pod

Pod

Sizing

▪Zone: Typically mapped to agg pair size ▪Pod: Typically mapped to access switch pair

▪ Server: 6-30 Servers per rack – limited by power ▪ Network: Depends on access model: Modular, ToR or Blade ▪ Storage: special Cabinets Session_ID
Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

▪Rack

▪ Size: determined by distance and density ▪ Cabling distance from server racks to network racks ▪ 100m Copper ▪ 200-500m Fiber ▪ Cabling density: # of servesr per rack and I/Os per server

54

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

27

Network Equipment Distribution
End of Row
End of Row
▪Traditionally used ▪Copper from server to access switches ▪Poses challenges on highly dense server farms ▫ Distance from farthest rack to access point ▫ Row length may not lend itself well to switch port density

End of Row

Patch panel Patch panel X-connect Patch panel X-connect server

Patch panel server

Common Characteristics

▪Typically used for modular access ▪Cabling is done at DC build-out ▪Model evolving from EoR to MoR ▪Lower cabling distances (lower cost) ▪Allows denser access (better flexibility) ▪6-12 multi-RU servers per Rack ▪4-6 kW per server rack, 10Kw-20Kw per network rack ▪Subnets and VLANs: one or many per switch. Subnets tend to be medium and large

Network Access Point A-B

Network Access Point C-D server server

Fiber Copper

End of Row

Patch panel

Patch panel Patch panel X-connect Patch panel X-connect server

End of Row (half row)

▪Use is starting to increase given EoR challenges ▪Copper from servers to access switches ▪Fiber may be used to aggregate ToR ▪It addresses aggregation requirements for ToR access environments

server

Network Access Point A-B server

Network Access Point C-D server

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

55

Network Equipment Distribution
Top of Rack

ToR

▪Used in conjunction with dense access racks(1U servers) ▪Typically one access switch per rack ▪Some customers are considering two + cluster ▪Typically: ▪ ~10-15 server per rack (enterprises) ▪ ~15-30 server per rack (SP) ▪ Use of either side of rack is gaining traction ▪ Cabling: ▪Within rack: Copper for server to access switch ▪Outside rack (uplink):
▪Copper (GE): needs a MoR model for fiber aggregation ▪Fiber (GE or 10GE):is more flexible and also requires aggregation model (MoR) ▪Subnets and VLANS: ▪ one or many subnets per access switch ▪ Subnets tent to be small

Patch panel Top of Rack server Patch panel X-connect Patch panel X-connect

Patch panel Top of Rack server

server

Network Aggregation Point A-B

Network Aggregation Point A-B server

To network core

Patch panel Top of Rack Top of Rack server Patch panel X-connect Patch panel X-connect

Patch panel Top of Rack Top of Rack server

Network Access Point A-B server
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Network Access Point C-D server

56

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

28

Network Equipment Distribution
Blade Chassis
End of Row (Switch to Switch)
▪Scales well for blade server racks (~3 blade chassis per rack) ▪Most current uplinks are copper but the NG switches will offer fiber ▪Scales well for pass-through blade racks ▪Copper from servers to access switches

Patch panel Patch panel X-connect Patch panel X-connect

Patch panel

sw1

sw2

sw1

sw2

End of Row (Pass-through)

Blade Chassis sw1 sw2 Network Aggregation Point A–B–C-D

Blade Chassis sw1 Network Aggregation Point A–B-C-D sw2

Blade Chassis sw1 sw2

Blade Chassis sw1 sw2

Blade Chassis

Blade Chassis

ToR

▪Have not seen it used in conjunction with blade switches ▪May be a viable option on pass-through environments is the access port count is right

Patch panel Pass-through Blade Chassis Pass-through Blade Chassis Pass-through Blade Chassis Network Aggregation Point A–B–C-D Network Aggregation Point A–B-C-D Patch panel X-connect Patch panel X-connect

Patch panel Top of Rack Pass-through Blade Chassis Pass-through Blade Chassis Pass-through Blade Chassis

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Width of Cabinets
24” vs 32” Cabinets?
For Each Scenario in this example
96in x 42in footprint Total of 16.8KW of Power Total of 48 Servers Air Flow delivered is the same

32in 45RU Cabinet Advantages
Reduced Static pressure due to proper cable management with more than 4-5 2RU servers per cabinet Vertical patch panels minimizes patch cord lengths, number of sizes, & increases usable RU spacing Power cables and Network cables have good separation Additional room for cabling thus reducing accidental downtime

Footprint only increases with the number of servers in a 24” cabinet.
17 – 2RU Servers 34 – 1RU Servers
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

58

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

29

Differences between Cabinets and Racks

• Aesthetics • Security • Additional Usable Cable Management Area & Flexibility • Vertical Patch Panels • Footprint • Additional Cooling Options • Cool Door • Convert back and forth from Server to Switch cabinet

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

Risks to Consider in Capacity Planning
Legacy Server Power per Server Power per Floor Space Cooling Needs—chilled airflow 2-3 kW per rack 30-40 W/ft² 200-300 cfm High-Density Server > 20 kW per rack 700-800 W/ft² 3,000 cfm
Source: Gartner 2006

20,000 ft²

800kW
+33%

Legacy DC designed to accommodate 2-3kW per Rack

100-200 Racks
Session_ID Presentation_ID

Introducing 1/3 high-density infrastructure into a legacy facility has cost, power, weight, and cooling implications *Peripheral DC costs considered
Cisco Public

© 2008 Cisco Systems, Inc. All rights reserved.

60

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

30

Modular Access / End Row Example

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

End of Row Example (Modular Access)
The Challenge
~4000 Servers Server LAN A&B Connectivity Servers 1Gb Connectivity Uplinks 10Gb Connectivity Utilize Catalyst 6509 Switches Core, Agg and Access Design SAN Cabinets Mainframe\Midrange Cabinets 450 Watts per Server 5.5 kW per Switch Cabinet 4 kW per SAN\Midrange Cabinet “POD” Concept Design
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Data Center Core
Aggregation Module

Aggregation Layer

2

2

2

2

2

2

2

2

48 Switches

4000 Servers

Access Layer

10 GbE GE

62

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

31

Data Center Sizing
Enterprise Modular Access: 6509
Enterprise Servers Ports* Switch Types Core Switches Aggregation Switches Access Switches
1500 < x < 4000 8000 6509
2: 6509 LCs: 6 4x10GE 4: 6509s LCs: 26 4x10GE Outbound Capacity: 80GE Oversubscription: 96:16 24: 6509 Oversubscription: 8.4:1

Core:
6509: 16 10GE ÷ 4 ports per LC = 4 LCs 2 LCs for cross connectivity Uplinks: based on DC outbound requirements

Aggregation:
6509: 96x10GE ÷ 4 ports per LC = 24 LCs 2 LCs for cross connectivity 26 LCs ÷ 8 Slots = 3.25 ~ 4 chassis 4 chassis x 4 uplinks = 16 10GE ports

Access: 8000 ÷ 336 = 23.8 ≈ 24 chassis
4 10GE uplinks per chassis = 96 10GE Oversubscription: 336:40 ~ 8.4:1

Maximum Capacity Access: 336 x 24 = 8064 ports ~ 4032 Servers
4 10GE uplinks per chassis = 8 10GE Oversubscription: 336:40 ~ 8.4:1

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

End of Row Example (Modular Access)
The Layout and Sample Solution
12 Server “PODs” Consists of the following: 4 Switch Cabinets for LAN & SAN 32 Server Cabinets 12 Servers per Server Cabinet
Core 1

Servers: 4032 6509 Switches: 30 Server\Switch Cabinets: 399 Midrange\SAN Cabinets Allotted For: 124
Core 2

Agg1

Agg2

Agg3

Agg4

Acc1

Acc2
6 Pair Switches

Acc11

Acc12

Acc13

Acc14
6 Pair Switches

Acc23

Acc24

336 Servers
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved.

336 Servers
Cisco Public

336 Servers

336 Servers 64

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

32

End of Row Example (Modular Access) The Data Center Layout Total White Space:
14,400 sqFt

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

65

End of Row Example (Modular Access) The “POD” with EDA Breakout

Equipment Distribution Area (EDA) Single “POD”
Acc1 Acc2

336 Servers

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

66

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

33

End of Row Example (Modular Access) EDA Application Photos

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

67

End of Row Example (Modular Access) The “POD” with HDA Breakout

Horizontal Distribution Area (HDA)

Single “POD”
Acc1 Acc2

336 Servers

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

68

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

34

End of Row Example (Modular Access) The “POD” with HDA Breakout

Horizontal Distribution Area (HDA)

Single “POD”
Acc1 Acc2

336 Servers

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

End of Row Example (Modular Access) HDA Application Photos

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

70

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

35

End of Row Example (Modular Access) Main Distribution Area (MDA)
Core 1 Core 2

Additional Equipment: Core Routing\Firewalls LAN Appliances SAN Directors

Agg1

Agg2

Agg3

Agg4

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

71

End of Row Example (Modular Access) Main Distribution Area (MDA)
Core 1 Core 2

Additional Equipment Core Routing\Firewalls LAN Appliances SAN Directors

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

72

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

36

End of Row Example (Modular Access) CFD Analysis
Designed in a Hot – Cold Architecture 12 - 20 Ton CRAC Units Outside 12 - 30 To CRAC Units Inside Utilizing Ceiling plenum for return air All Perforated Tiles at 25% Open Peak Temp was 114°

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

73

Consolidated I/O Architecture with Virtualization

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

74

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

37

Increased Efficiency, Simpler Operations
Mgmt Network Front-End Network

Backup Network Unified Fabric

Storage Network

Back-End Network

Unified Fabric and I/O

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

75

Fibre Channel over Ethernet (FCoE)
A method for a direct mapping of FC frames over Ethernet
Seamlessly connects to FC networks over the Ethernet
FCoE appears as FC to the host and the network Preserves current FC infrastructure and management FC frame is unchanged

Extends FC across the datacenter

10Gb Etherne t Pipe

FC

Can operate over standard switches (with jumbo frames) Priority Flow Control guarantees no-drops Mimics FC credit-buffer system, avoids TCP Does not require expensive off-loads

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

38

VM-Optimized Services
Enables convergence of multiple traffic types
Virtual Machines LAN Virtual Machines SAN Hypervisor Mgmt LAN Virtual Infrastructure Services LAN

LAN

LAN

SAN A

SAN B

Scales VM LAN performance
Increase I/O bandwidth Increase VM density VMotion

Accelerates Virtual Infrastructure Services
Live VM migrations via VMotion and DRS features Enable additional services
Session_ID Presentation_ID

10 GbE 10 GbE DCE Fibre Channel 10 GbE FCoE/DCE

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

77

Virtualized Server Environment – Unified Fabric ToR Deployment
40 Servers per Cabinet Pair 2 Nexus 5020 per Cabinet Pair 10 Cabinet Pairs VMs per Server: 10 Servers: 400 Nexus 7000 Switches: 2 Nexus 5000 Switches: 10 MDS Switches: 2
Agg1 Agg2

Acc1

Acc2
10 CP

Acc1

Acc2

40 Servers

40 Servers

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

39

Virtualized Server Environment – Unified Fabric A Single “POD” Using Virtualized Model

24 Cabinets Total
• 10 Server Cabinet Pairs (20 Total) • 4 Switch Cabinets

Over Head Fiber Optic Cabling Only 400 Servers with 10 Virtualized Each (4000 Total Virtual Servers)

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

79

Virtualized Server Environment – Unified Fabric The Cabinet Pair Breakout

Cross Patching between Cabinets 20 Servers per Cabinet X 2 Cabinets (40 Total) 40 Connections per Switch Each Switch Uplinks
• 4 – LAN A \ 4 – LAN B • 4 – SAN A \ 4 – SAN B
Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

80

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

40

Unified Fabric - Consolidated I/O Extends Benefits of ToR
Consolidate LAN I/O • Consolidate multiple LAN adapters into two CNAs • Consolidate multiple cables into two paths
Unconsolidated I/O
2x SAN 2x LAN

Consolidate SAN I/O • No need for SAN-specific adapters • Reduce SAN switches and cables • Unified I/O switches connect to existing SAN infrastructure
Consolidated I/O
2x Consolidated

Consolidated I/O Benefits • CapEx Savings up to 30% • Cable reduction of 50% or more for ToR and EoR designs • Potential power savings

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

81

Virtualization Impact on Critical Facilities.
Power Needed per Server has increased Could be up to 16KW for this cabinet Need for Supplemental Cooling required Weight of Equipment on Raised Floor All Copper Cabling is Contained with the Two Cabinets Cable reduction Fiber Optic Cabling for the Uplinks
The need for quality Fiber Cabling has increased

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

82

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

41

Summary

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

83

Data Center Environmental Challenges
Typical Shortcomings Cooling Structured cabling Power Structural loading

The nature of Data Center infrastructure makes it challenging to find solutions that don’t spawn other problems
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

84

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

42

Data Center Environmental Challenges
Universal Solutions
Physically Distribute
Equipment Power & Cooling

Right-size infrastructure (modularity) Virtualize Network Architecture and Facilities Dependencies

Use pools of servers and storage, controlled by the network, to provide Data Center resources.
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

85

Universal Solutions
Service Oriented Data Center (SODC) More efficient use of computing resources Shared computing among multiple machines
Built-in redundancy Individual servers are expendable

Devices can be managed on a by-cabinet basis

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

86

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

43

Q and A

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

87

Recommended Reading
Topics
Site selection Data Center sizing and layout Physical design Large-scale server moves Remote monitoring Change management
Includes downloadable design template

Available Onsite at the Cisco Company Store
Session_ID Presentation_ID © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

88

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

44

Recommended Reading
Continue your Networkers at Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books Cisco Press Data Center Fundamentals

Session_ID Presentation_ID

Available Onsite at the Cisco Company Store
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

89

Additional Resources
URLs
6500 Cabinet Information http://wwwin.cisco.com/dss/isbu/6500/enviro/index.shtml Panduit http://www.panduit.com/default.asp TIA – Telecommunications Industry Association http://www.tiaonline.org/ ASHRAE – American Society of Heating, Refrigerating and Air-Conditioning Engineers http://www.ashrae.org/ Uptime Institute
http://uptimeinstitute.org/

Government work on server and DC Energy Efficiency:
http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

90

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

45

Complete Your Online Session Evaluation
Win fabulous prizes; give us your feedback Receive ten Passport Points for each session evaluation you complete Go to the Internet stations located throughout the Convention Center to complete your session evaluation Winners will be announced daily at the Internet stations
Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

91

Session_ID Presentation_ID

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

92

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

46