You are on page 1of 74

Troubleshooting Nexus 7000
BRKRST-3066

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 2

Session Goal

To provide you with an understanding of the Cisco Nexus™
7000 built-in troubleshooting tools and troubleshooting
technique of NX-OS operating system
Increase familiarity with Cisco Nexus™ 7000 logging and
information capture
Ensure you get the right information at the right time

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Public .Agenda  Introduce NXOS software architecture and logging capability  Define built-in troubleshooting tools that will assist with problem analysis and resolution  Demonstrate troubleshooting tools used when investigating common network functions BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public 5 . All rights reserved.NXOS Architecture Introduction Traditional IOS Versus NX-OS – Software Modularity BRKRST-3066 © 2013 Cisco and/or its affiliates.

2MB file to review Directory of bootflash:/ 9 -rwx 1194813 Feb 10 2012 06:42:30 +00:00 6k. Please Wait . Cisco Public .tech N7K1# show tech > bootflash:giant.tech Cat6K#dir bootflash: 1..NXOS Architecture Introduction IOS vs NX-OS – Show Tech  Logging output greatly increased.tech Show tech brief will take 4-6 minutes to complete. All rights reserved.tech BRKRST-3066 © 2013 Cisco and/or its affiliates. assists stateful information capture  More data requires more effective filtering  Capture feature related information for later analysis Cat6K#show tech | redirect bootflash:6k. 120MB file to review N7K1# dir bootflash: 120077767 Feb 10 16:18:40 2012 giant..

Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

04:30:29.1.Troubleshooting Tools CLI Capture and Filter – eigrp. [0/0].1.11.100. N7K1-VPC# show ip route | egrep "local$" *via 1.5- 9. local BRKRST-3066 © 2013 Cisco and/or its affiliates. start-time. Lo0. start-time  Use the filtering options in real-time to increase your effectiveness N7K1-VPC# show logging logfile start-time 2012 Jan 11 16:00:00 2012 Jan 23 16:00:08 Campus_N7K1-VPC %ETHPORT-5-IF_DOWN_NONE: Interface port-channel40 is down (None) N7K1-VPC# show logging logfile | egrep "MAC|VLAN|ETHPM" 2012 Jan 23 11:42:49 Campus_N7K1-VPC %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1. wc. include. last  Powerful CLI based on Linux Kernel  Built-in CLI filter techniques increases speed and relevance  grep.1002-1005.203. sed.200. egrep. All rights reserved. Cisco Public . last.1.1111 on Interface port-channel10 are being suspended.

254/Vlan64 2012 Mar 26 23:33:25.253.0.0.253.0 crc:0xfdd2 aut:0 aukid:0 from 192. should not be seen on Vlan64 2012 Mar 26 23:33:25.254. All rights reserved.64.253.992586 ospf: 6467 [3981] (default) rcvd: prty:7 ver:2 t:HELLO len:44 rid:0.0.992780 ospf: 6467 [3981] Invalid src address 192.992966 ospf: 6467 [3981] Invalid src address 192.254. should not be seen on Vlan64 BRKRST-3066 © 2013 Cisco and/or its affiliates. debug-filter  Debugging per feature logged to bootflash  Set a filter file and log to a new file N7K1-VPC# debug-filter ip ospf interface vlan 64 N7K1-VPC# debug logfile offending_traffic N7K1-VPC# debug ip ospf packets N7K1-VPC# undebug all N7K1-VPC# show debug logfile offending_traffic 2012 Mar 26 23:33:25.0. Cisco Public .64.64.0 area:0.Troubleshooting Tools Debugging Log and Filter – debug logfile.

(Reason: Vlan is not allowed on Peer-link) N7K1-VPC# show system internal ethpm event-history interface port-channel 20 29) FSM:<port-channel20> Transition at 500252 usecs after Mon Jan 23 17:20:12 2012 Previous state: [ETH_PORT_FSM_ST_TRUNK_UP] Triggered event: [ETH_PORT_FSM_EV_LOGICAL_CHG] Next state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_CHANGE_TRUNK] BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting Tools Logging and Feature History – show logging logfile. All rights reserved. Cisco Public . show <feature> internal  Syslogs and feature event history per Virtual Device Context (VDC)  Feature interaction tracked through ‘event-history’  Use syslogs with feature logging to compare feature behaviour N7K1-VPC# show logging logfile 2012 Jan 23 17:20:12 Campus_N7K1-VPC %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 50 on Interface port- channel20 are being suspended.

ERSPAN  ACL Log to flash N7K1-VPC(config)# monitor session 1 type ? acl-capture Create an acl-capture session erspan-destination Create an erspan destination session erspan-source Create an erspan source session local Create a local session N7K1-VPC(config)# monitor session 1 type local N7K1-VPC(config-monitor)# source interface e2/1 N7K1-VPC(config-monitor)# destination interface e2/2 N7K1-VPC(config-monitor)# no shut BRKRST-3066 © 2013 Cisco and/or its affiliates.SPAN  Data Plane SPAN in hardware  VACL. Local SPAN.Troubleshooting Tools Data Plan Traffic Capture . All rights reserved. Cisco Public .

Troubleshooting Tools Onboard Logging & Diagnostic Monitoring – show logging enabled. Cisco Public . per module logging  Review event history for failure detection  Always check diagnostics before troubleshooting! N7K1# show logging onboard module 2 exception-log ---------------------------- Module: 2 ---------------------------- Exception Log Record : Wed Nov 16 09:36:28 2011 (608385 us) N7K1# show diagnostic events 1) Event:E_DEBUG. All rights reserved. at 475956 usecs after Mon Jan 23 16:17:07 2012 [104] Event_INFO: TestName->ASICRegisterCheck TestingType->health monitoring module->5 Result->pass Reason->Success BRKRST-3066 © 2013 Cisco and/or its affiliates. show diagnostic  Persistent. length:115.

697182 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0. Cisco Public .0 (Request) 2012-01-26 10:13:18.Ethanalyzer  Ethanalyzer built in Wireshark capture utility  Filter capture based on granular ‘tcpdump’ syntax  Display capture natively from NXOS console using Wireshark filters N7K1# ethanalyzer local interface inband capture-filter 'arp' limit-captured-frames 100 write bootflash:arp.0 (Request) 2012-01-26 10:13:18.697098 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.0.0. N7K1# ethanalyzer local read bootflash:arp.0 (Request) BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting Tools CPU Traffic Capture .697430 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.pcap limit-captured-frames 100 2012-01-26 10:13:18.0.697680 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.0.0.0.pcap Capturing on inband 100 Program exited with status 0.0.0.0 (Request) 2012-01-26 10:13:18. All rights reserved.

Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public .

show proc cpu  N7K Sup1 DualCore CPU with robust preemptive scheduling  High CPU process is not always an issue  Per VDC output. 1 running CPU states : 0. identify the process using CPU resources N7K1# show system resources Load average: 1 minute: 0. 5096040K free N7K1# show proc cpu sort | grep -v 0. 95. Cisco Public .36 15 minutes: 0. 3164848K used. ----------- 3530 259509 119851 2165 1.5% user. ----------.28 Processes : 1091 total.0% kernel.40 5 minutes: 0. 4.5% idle Memory usage: 8260888K total.9% platform Please note that only processes from the requested vdc are shown above BRKRST-3066 © 2013 Cisco and/or its affiliates. -----. -------.0 PID Runtime(ms) Invoked uSecs 1Sec Process ----. All rights reserved. ----.Troubleshooting CPU Identify Process and Timeline – show system resources.

............7..5....4.4.....1....2.......5..6....1.3...5.6.... 0 5 0 5 0 5 0 5 0 5 0 5 0 CPU% per hour (last 72 hours) * = maximum CPU% # = average CPU% BRKRST-3066 © 2013 Cisco and/or its affiliates.....3.2. Cisco Public .Troubleshooting CPU Identify Process and Timeline – show process cpu history N7K1# show proc cpu history  History shows combined utilisation across VDC <snip>  Compare problem timeline with CPU spikes 1 555554435555544354657844550 and average 466546476420229956261360720  Identify the start of utilisation issues 100 * 90 *  Snapshot to bootflash file for historical trending 80 * * 70 ** * 60 *** * * **** * * 50 ****** ***** * ******* *** 40 *************************** Boot up 26 hours earlier 30 *************************** 20 *************************** 10 ########################### 0. All rights reserved.

9 1534:01 naxos 1917 root -2 0 35656 6600 4524 S 2. no network traffic module-1# show system resources Load average: 1 minute: 0.6%si.34 Processes : 69 total.8%id.0%wa. 443784k used.1%sy. 0. 1 user.6 377:50.34 Tasks: 71 total.Troubleshooting CPU Module CPU Health Check – show system resources  CPU per module. 0. 0. 78. Cisco Public . 1. load average: 0. 0k free. 0.14:06:36 up 21 days.24 5 minutes: 0.0% kernel.6%ni. 3 running. manages module processes. 443632K used. 68 sleeping. 0.27.0 0.40 15 minutes: 0.8%us. All rights reserved. 0k buffers Swap: 0k total.1%hi.48 eureka BRKRST-3066 © 2013 Cisco and/or its affiliates. 2. 0 zombie Cpu(s): 9.1 0.8 0. 0. 592144K free module-1# show system internal processes cpu top .0% user. 0k used.8 3240:42 stats_client 1921 root -2 0 37296 9476 4116 S 15.0% idle Memory usage: 1035776K total. 2 running CPU states : 3.42. 591992k free. 15:34. 7. 0 stopped. 97.0%st Mem: 1035776k total. 93352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1935 root 25 5 39304 8352 3628 R 81.

0 higher than usual. ----.0 Per VDC instance of netstack. -------. Cisco Public .5% kernel. traffic hitting CPU 5861 360 126 2861 11.0% idle Please note that only processes from the requested vdc are shown above N7K1# switchto vdc VPC Netstack and ARP are running N7K1-VPC# show proc cpu sort | ex 0. PID Runtime(ms) Invoked uSecs 1Sec Process each processes utilisation ----. 21. 4.Troubleshooting CPU Receiving Traffic Sent to CPU – show proc cpu sort N7K1# show proc cpu | ex 0. ----------.9% R2D2_usd CPU util : 12. we have PID Runtime(ms) Invoked uSecs 1Sec Process ----.3% netstack 5840 162 38 4273 4.0% kernel. ----. -----.0% user. 73.7% arp CPU util : 5. ----------.0% user.5% idle Please note that only processes from the requested vdc are shown above BRKRST-3066 © 2013 Cisco and/or its affiliates. ----------- separate from other 3904 204 74 2764 0.9% netstack 3939 477316 717760 665 0. All rights reserved. ----------. 84. -------. -----.

Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . All rights reserved.

All rights reserved.Troubleshooting CoPP Control Plane Policing – Protect CPU from your network Fabric ASIC Dedicated Arbitration Central Path VOQ Arbiter 1GE Inband System Controller 1.66GHz Dual-Core 2GB 4GB Internal CF DRAM Main slot0: 2MB CPU NVRAM log-flash: BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

All rights reserved. nd (ipv6)  DHCP traffic  Glean traffic (no arp or nd)  Two stage policing  Rate-limit in hardware per-module before sending to CPU  Policy based policing traffic that have passed rate-limiters  May require tweaking to match your network conditions BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .Troubleshooting CoPP Control Plane Policing – Protect CPU from your network  Protect CPU from network traffic  arp.

CPU issues no match exception ip icmp unreachable longer created by ICMP exception match exception ipv6 option generation match exception ipv6 icmp unreachable set cos 1 police cir 360 kbps. check module 2 : for violated traffic conformed 0 bytes. action: transmit violated 0 bytes. Cisco Public .Troubleshooting CoPP Control Plane Policing – show policy-map interface control-plane N7K1# show policy-map int control | grep –v Control Plane service-policy input: copp-system-policy class-map copp-system-class-exception (match-any) match exception ip option Exception traffic. action: transmit Granular class based matching. action: drop IP and MAC ACL in one class class-map copp-system-class-critical (match-any) match access-group name copp-system-acl-bgp BRKRST-3066 © 2013 Cisco and/or its affiliates. violated 0 bytes. action: drop module 10 : conformed 273344 bytes. action: transmit violated 0 bytes. bc 250 ms Implemented per-module. action: drop module 3 : conformed 0 bytes. All rights reserved.

action: drop violated 0 bytes. action: drop 100Kbps default dropped traffic BRKRST-3066 © 2013 Cisco and/or its affiliates. action: drop class-map copp-system-class-management (match-any) violated 0 bytes. All rights reserved. no drops violated 0 bytes. action: drop violated 0 bytes. action: drop violated 0 bytes. action: drop Routing protocols class. action: drop TFTP.Troubleshooting CoPP Control Plane Policing – show policy-map interface control-plane  Filter output to confirm affected classes  Check class-map definitions if violated traffic does not match your expectation N7K1# show policy-map int control | egrep "class-map|violated" class-map copp-system-class-critical (match-any) violated 0 bytes. SNMP. Cisco Public . violated 274930048 bytes. action: drop violated 0 bytes. FTP protocols class-map class-default (match-any) violated 0 bytes. action: drop Low rate policer on default class.

Cisco Public . limited before CoPP Dropped : 829800  Limit based on packet definitions Total : 4764070 receive Traffic destined to Sup Config : 30000 Allowed : 6875754 Dropped : 0 Total : 6875754 BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting CoPP Control Plane Policing – show hardware rate-limit  Per module rate-limiter in addition to control-plane policy N7K1# show hardware rate-limiter | grep –v Units for Config: packets per second Rate Limiter Class Parameters ------------------------------------------------------------ layer-3 control Config : 10000 Allowed : 0 Review layer-3 control if Dropped : 0 Total : 0 control-plane instablility <snip>  Hardware Rate-Limiter performed by forwarding engine hardware copy Copy to sup used for Config : 30000  Global system wide hardware Allowed : 3934270 feature ARP. All rights reserved.

......... 32000 pps Tick counter ... 1203243 Traffic tail dropped..... All rights reserved.Troubleshooting CoPP Review Traffic Sent to CPU – show hardware internal inband cpu-mac stats N7K-1# show hardware internal cpu-mac inband stats RMON counters Rx Tx Total number of total packets 779905245 1421785114 good packets 779905245 1421650279 frames received and total octets (hi) 0 0 sent by CPU total octets (low) 172302724342 192974265660 Error counters --------------------------------+-- Rx no buffers .... Cisco Public ............ Throttle statistics after CoPP and RL Throttle interval ............... 2 * 100ms Packet rate limit ....... 12414130 CPU bound traffic Rx packet rate (current/max) 4993 / 20296 pps current pps Tx packet rate (current/max) 60 / 3474 pps MAC counters MAC0 (R2D2) MAC1 (CPU) /maximum pps Rx Tx Rx Tx reached XOFF packets auto-generated 5447 XOFF packets 7590855 6731953 XON packets 0 18561642 How many times did throttling kick in BRKRST-3066 © 2013 Cisco and/or its affiliates....

Cisco Public .pcap  Local review or export to wireshark BRKRST-3066 © 2013 Cisco and/or its affiliates. define filters based on first capture  Capture to text file or .Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local interface  Start by capturing all traffic. All rights reserved.

html BRKRST-3066 © 2013 Cisco and/or its affiliates.txt Filter syntax: http://www. Cisco Public .Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local interface N7K1# ethanalyzer local int inband limit-captured-frame 100 write bootflash:cpu. Provides information from internal system headers.pcap Creates pcap file which can later be analysed by Capturing on inband GUI wireshark 100 Program exited with status 0.wireshark. All rights reserved. not allowed with ‘write’ keyword N7K1# ethanalyzer local int inband decode-internal limit-captured-frames 100 > cpu.org/docs/wsug_html_chunked/ChCapCaptureFilterSection.

387538 172.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.388066 172. All rights reserved.123 -> 172.1 UDP Source port: 63 Destination port: 63 Local analysis completed from CLI prompt BRKRST-3066 © 2013 Cisco and/or its affiliates.pcap 2012-01-26 10:02:20.60.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.16.16.50.123 -> 172.388940 172.16.388566 172.60.60.50.50.60.123 -> 172.16.16.123 -> 172.16.50.60.Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local read N7K1# ethanalyzer local read bootflash:cpu.387691 172.50.123 -> 172. Cisco Public .16.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.16.16.16.

All rights reserved..... = IG bit: Individual address (unicast) .000000000 seconds] Frame Number: 1 Frame Length: 92 bytes Use filter in next capture to Capture Length: 60 bytes increase the relevance [Frame is marked: False] [Protocols in frame: eth:ip:udp:data] Ethernet II.pcap detail Wireshark output....387538000 help to find relevant packets [Time delta from previous captured frame: 0. 2012 10:02:20....... . . .Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local read N7K1# ethanalyzer local read bootflash:cpu. .. Src: 00:24:97:36:81:3f (00:24:97:36:81:3f). ..... . Cisco Public .. .000000000 seconds] [Time since reference or first frame: 0.0 ....... = LG bit: Globally unique address (factory default) BRKRST-3066 © 2013 Cisco and/or its affiliates.. display filters Frame 1 (92 bytes on wire..0.. . 60 bytes captured) Arrival Time: Jan 26. Dst: 04:c5:a4:e9:ac:44 (04:c5:a4:e9:ac:44) Destination: 04:c5:a4:e9:ac:44 (04:c5:a4:e9:ac:44) Address: 04:c5:a4:e9:ac:44 (04:c5:a4:e9:ac:44) . ..000000000 seconds] [Time delta from previous displayed frame: 0.....

Cisco Public . All rights reserved.Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates.

only reseat will resolve physical issue N7K1# reload module 3 This command will reload module 3.Troubleshooting Hardware Module – show module. All rights reserved. ----------------------------------. ---------- 2 32 10 Gbps Ethernet XL Module N7K-M132XP-12L ok 3 48 1000 Mbps Optical Ethernet Module N7K-M148GS-11 testing 5 0 Supervisor module-1X N7K-SUP1 active * 6 0 Supervisor module-1X N7K-SUP1 ha-standby 10 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok <snip> Mod Online Diag Status --.-----------------. Module booting during troubleshooting 2 Pass 3 Untested 5 Pass Reload from CLI denies module access 6 Pass 10 Pass to power shelf.. Proceed[y/n]? [n] y reloading module 3 . -----------------. reload module <x> N7K1# show module Mod Ports Module-Type Model Status --.. BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . ----.

3) SecondaryBootROM--------------> . A = Abort. All rights reserved. Per port diagnostic test. 5) OBFL--------------------------> . at 362751 usecs after Wed Jan 25 03:26:21 2012 [103] Event_ERROR: TestName->SpineControlBus TestingType->health monitoring module->5 Result->fail Reason-> XBar BRKRST-3066 © 2013 Cisco and/or its affiliates. . . . . . . . . Initiated from module CPU. . . . show diagnostic event N7K1# show diagnostic result mod 3 | grep -v ^$ Current bootup diagnostic level: complete Eliminates empty lines Module 3: 1000 Mbps Optical Ethernet Module Test results: (. .Troubleshooting Hardware Online Diagnostics Review – show diagnostic result. . . . . . . N7K1# show diagnostic events error 1) Event:E_DEBUG. . . F = Fail. . length:217. . . . F could be transient or 6) PortLoopback: survive a module reset Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ----------------------------------------------------- . . . . . . I = Incomplete. . U = Untested. . . . . Port 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ----------------------------------------------------- . . 2) PrimaryBootROM----------------> . . E = Error disabled) 1) ASICRegisterCheck-------------> . . . . . . = Pass. Cisco Public . . . . 4) EOBCPortLoopback--------------> . . Port 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------------------------------------- .

check B -. Cisco Public . ---------- 1 0 Fabric Module 1 N7K-C7010-FAB-1 ok 2 0 Fabric Module 1 N7K-C7010-FAB-1 ok 3 0 Fabric Module 1 N7K-C7010-FAB-1 ok N7K1# show hardware capacity fabric-utilization ------------------------------------------------------------------------ Fabric Planes: A -. ----------------------------------. ----.Unicast fabric interface Xbar utilization max. -----------------. show hardware capacity N7K1# show module xbar Xbar Ports Module-Type Model Status --.Troubleshooting Hardware Fabric – show module.Multicast/Multidestination fabric interface previous history for trending info -------------------------PEAK FABRIC UTILIZATION------------------------ I/O |-----FABRIC----| Ingress | Egress Slot |Mod Inst Plane| Util Time | Util Time ------------------------------------------------------------------------ 2 1 1 A 0% 01-23@16:41:46 0% 01-23@16:41:46 2 1 1 B 0% 01-23@16:41:46 0% 01-23@16:41:46 N7K1(config)# poweroff xbar 1 Power off xbar before trying hardware swap BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

Agenda

 NXOS Architecture Introduction
 Built-in Troubleshooting Tools
 Troubleshooting Nexus 7000
 CPU
 Control-Plane – CoPP
 Hardware
 vPC
 Unicast Layer 2 and Layer 3 Forwarding and ARP
 Multicast Layer 2 and Layer 3 Forwarding
 QoS

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Troubleshooting vPC
Distributed Layer 2 Forwarding

 Designed as an alternative to 172.16.100.1 172.16.100.2
Po10
STP E2/1 E2/1

N7K-1 E2/2 E2/2 N7K-2
 Appear as one L2 device to the
vPC PKL (L3)
network E10/25 E2/17

 Peers must maintain consistent E2/17 E10/25

network view Po20/V100 Po40/V50

 Recovery mechanisms protect Sw3 Sw4
forwarding 172.16.100.3 172.16.50.40

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Troubleshooting vPC
Global Consistency Check – show vpc brief
N7K1-VPC# show vpc brief
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive Secondary role, vPCs
Configuration consistency status : success shut if peer-link is down
Per-vlan consistency status : success
Type-2 consistency status : failed
Type-2 inconsistency reason : SVI type-2 configuration incompatible
vPC role : secondary
Number of vPCs configured : 2
Peer Gateway : Enabled Peer gateway enabled;
<snip> routing for peer
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po10 up 1,5-9,11,100,200,203,1111 Type-2 Inconsistency
failed, but vlans still
vPC status
---------------------------------------------------------------------- passing on vPC and
id Port Status Consistency Reason Active vlans peer-link
-- ---- ------ ----------- ------ ------------
20 Po20 up success success 1,5-9,11,203
40 Po40 up success success 1,5-9,11,100,200,203,1111

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

configured locally VTP password 2 - VTP pruning status 2 Disabled - Interface-vlan admin up 2 5-6. ---------------------.9 93-994.508. Disabled.156.200-201. Edge 1 Normal. 1999. Edge BPDUGuard Disabled Disabled STP MST Simulate PVST 1 Enabled Enabled VTP domain 2 TEST - VTP version 2 1 . BPDUFilter.Troubleshooting vPC Global Consistency Check – show vpc consistency parameters global N7K1-VPC# show vpc consistency-parameters global Type 1 : vPC will be suspended in case of mismatch Name Type Local Value Peer Value ------------. SVI 200 not VTP mode 2 Server .20-23.200 Mismatched allowed Interface-vlan routing 2 1.921.100 1. ----------------------- STP Mode 1 Rapid-PVST Rapid-PVST STP Disabled 1 None None VTP enabled locally.32.5-6.200.220 -222.203.2901 Local suspended VLANs .5-9.555. 1.1111. Cisco Public .1221.5-12. <snip> disabled on remote STP Port Type.100.16. 1002-1005 - BRKRST-3066 © 2013 Cisco and/or its affiliates.1111 00. Normal.230.100.2211-2213. Disabled.11.203.1000. ---.2233-22 34.100. All rights reserved.5-6.200 VLANs capability Allowed VLANs .1 002-1005.1 1-3.100 5-6.

d87c. F F sup-eth1(R) * 100 0021.Overlay MAC age .primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.ac01 static .aac4 static . G .LID ---------+-----------------+--------+---------+------+----+------------------ G 100 0000.a4e9. All rights reserved.Routed MAC. Cisco Public .seconds since last seen.a4e9.ac44 static . route traffic destined to peer peer-gateway BRKRST-3066 © 2013 Cisco and/or its affiliates.primary entry.SSID.Troubleshooting vPC MAC Address Synchronised – show mac address-table  MAC addresses synchronised via CFS  First Hop Redundancy Protocol addresses forwarded by both peers  Peer-gateway allows routing for peer SVI MAC N7K1-VPC# show mac add vlan 100 * . F F vPC Peer-Link(R) G 100 04c5. (R) .Gateway MAC. O .0c07.2740 dynamic 0 F F Po20 Peer MAC address with G-bit N7K1-VPC# show run vpc | egrep "peer-gateway" set. F F vPC Peer-Link(R) G 100 04c5.+ .

All rights reserved.Troubleshooting vPC Data Collection – show tech vpc. show tech pixm N7K1-VPC# show tech-support vpc | grep "`show " `show version` `show module` `show vpc brief` `show vpc role` `show running-config vpc` `show system internal vpcm event-history global` `show system internal vpcm event-history errors` `show system internal vpcm event-history msgs` `show system internal vpcm event-history interactions` `show system internal vpcm mem-stats detail` `show system internal vpcm info all` `show system internal vpcm info global` `show cfs internal ethernet-peer database` `show spanning-tree` N7K1-VPC# show tech-support stp N7K1-VPC# show tech-support vtp N7K1-VPC# show tech-support pixm N7K1-VPC# show tech-support forwarding l2 unicast BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . show tech stp.

All rights reserved. Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates.

100.16. Cisco Public .16.16. All rights reserved.2 Po10 forwarding engine per E2/1 E2/1 module N7K-1 E2/2 E2/2 N7K-2 vPC PKL (L3)  L3 entries learned by CPU E10/25 E2/17 and pushed to hardware E2/17 E10/25 engine Po20/V100 Po40/V50  Verify software entries Sw3 Sw4 against topology diagrams 172.100.Troubleshooting Unicast Forwarding Forwarding L2 and L3 in Hardware  L2 learning in hardware by 172.40  Verify hardware entries against software entries BRKRST-3066 © 2013 Cisco and/or its affiliates.100.3 172.16.1 172.50.

All rights reserved.Troubleshooting Unicast Forwarding Forwarding L2 and L3 in Hardware OSPF OSPF RIP IS-IS EIGRP Static BGP ARP v2 v3 AM u4RIB u6RIB Unicast Routing Information Base (uRIB) mRIB uFDM Supervisor FIB Manager I/O Module Forwarding Hardware BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

Cisco Public . G . [250/0]. Vlan50.16. attached *via 172.SSID.16.9736. show mac address-table N7K1-VPC# show ip route 172.seconds since last seen.50.Troubleshooting Unicast Forwarding Software Tables – show ip route.50.primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID. ubest/mbest: 1/0.40 IP Route Table for VRF "default" '*' denotes best ucast next-hop 172.primary entry.LID ---------+-----------------+--------+---------+------+----+------------------ * 50 0024.Overlay MAC age .50.9736.16.813f 50 arp Vlan50 Campus_N7K1-VPC# show mac address-table address 0024.16. All rights reserved.50.Routed MAC.40. O .40 0024.Adjacencies Throttled for Glean IP Adjacency Table for VRF default Total number of entries: 1 Address MAC Address Pref Source Interface 172.+ .813f dynamic 150 F F Po40 BRKRST-3066 © 2013 Cisco and/or its affiliates.16.9736. show ip adjacency. 00:02:09.Gateway MAC.40/32.50. am N7K1-VPC# show ip adjacency 172.813f Legend: * . (R) .40 Flags: # .

0000.0128 0x07ff1 1 0x001 0 29 0 0 0 0 0 0 0 1 0 0 N7K1-VPC# show vlan internal bd-info vlan-to-bd 50 Allocated bridge-domain VDC Id Vlan Id BD Id matches in hardware table -----.ac44 0x00400 1 0x000 0 130 1 1 0 0 0 0 0 0 0 0 0 1 0 59 4180. Cisco Public .0127 0x07ff1 1 0x001 0 170 0 0 0 0 0 0 0 1 0 0 0 1 0 59 4180. show system internal pixm N7K1-VPC# show hardware mac add 2 vlan 50 FE | Valid| PI| BD | MAC | Index| Stat| SW | Modi| Age| Tmr| GM| Sec| TR| NT| RM| RMA| Cap| Fld|Always | | | | | | ic | | fied|Byte| Sel| | ure| AP| FY| | |TURE| | Learn ---+------+---+------+---------------+-------+-----+-----+-----+----+----+---+----+---+---+---+----+----+----+------ 0 1 0 59 0024. ------- 4 50 59 DMAC sent to LTL index for PO40 N7K1-VPC# show system internal pixm info ltl 0xa48 PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT ------------------------------------------------------------------------------ Normal Po40 0x0a48 0x16000027 0x00000000 0x00000002 1 BRKRST-3066 © 2013 Cisco and/or its affiliates.0c07.a4e9.0000.9736.0000.ac01 0x00400 1 0x000 0 156 1 1 0 0 0 0 0 0 0 0 0 1 1 59 04c5.Troubleshooting Unicast Forwarding L2 Hardware Entries – show hardware mac-add.813f 0x00a48 0 0x003 0 9 1 0 0 0 0 0 0 0 0 0 0 1 0 59 0000.0123 0x00a48 0 0x003 0 156 1 0 0 0 0 0 0 0 0 0 0 1 1 59 0000. ------. All rights reserved.

a4e9.16. Cisco Public .50.40 Vlan50 N7K1-VPC# show system internal forwarding ip route 172.50.50.16.ac44 LIF: 0x3b (Vlan50) DI: 0x0 ccc: 4 L2_FWD: NO RDT: NO packets: 0 bytes: 0 zone enforce: 0 BRKRST-3066 © 2013 Cisco and/or its affiliates.40 module 2 ----+---------------------+----------+----------+------+----------- Dev | Prefix | PfxIndex | AdjIndex | LIFB | LIF ----+---------------------+----------+----------+------+----------- 1 172.Troubleshooting Unicast Forwarding L3 Hardware Entries – show ip fib route <dest>. All rights reserved.40/32 172.40/32 0x2033 0x43015 0 0x3b Campus_N7K1-VPC# show system internal forwarding adjacency mod 2 entry 0x43015 det Device: 1 Index: 0x43015 DMAC: 0024.16.9736. show system internal forward N7K1-VPC# show ip fib route 172.813f SMAC: 00c5.16.50.40 module 2 IPv4 routes for table default/base ------------------+------------------+----------------------+----------------- Prefix | Next-hop | Interface | Labels ------------------+------------------+----------------------+----------------- 172.50.16.

Troubleshooting Unicast Forwarding Scale and Utilsation – show ip route summary. All rights reserved. Cisco Public . show hardware internal N7K1-VPC# show ip route summary IP Route Table for VRF "default" Total number of routes: 45 Total number of paths: 47 Best paths per protocol: Backup paths per protocol: am : 4 None local : 9 <snip> Number of routes per mask-length: /8 : 1 /24: 6 /30: 1 /32: 37 N7K1-VPC# show hardware internal forwarding table utilization module 2 Note: Utilization may not reach the maximum. Module 2 usage: Route Type Used %Used Free %Free Total (Log/Phys) (Log/Phys) (Log/Phys) ------------------------------------------------------------------------- IPv4 Unicast: 67/67 0 L2VPN Peer: 0/0 0 MPLS: 0/0 0 ------------------------------------- 904967/904967 99 905120/905120 ------------------------------------- BRKRST-3066 © 2013 Cisco and/or its affiliates.

All rights reserved. Cisco Public . show tech netstack. show tech arp N7K1-VPC# show tech-support forwarding L3 unicast | grep "`show " `show forwarding route summary vrf all` `show forwarding route max-display-count 100000 vrf all` `show forwarding vrf all adjacency` `show forwarding ipv6 route summary vrf all` `show forwarding ipv6 route max-display-count 100000 vrf all` `show forwarding vrf all ipv6 adjacency` `show forwarding trace` `show forwarding internal errors` `show forwarding internal error counts` `show forwarding internal unicast counts vdc all` `show forwarding internal message counts` N7K1-VPC# show tech-support netstack | grep "`show " | wc -l 212 N7K1-VPC# show tech-support arp | grep "`show " `show running-config arp` `show ip arp internal event-history cli` `show ip arp vrf all` `show ip arp static vrf all` `show ip arp summary vrf all` `show ip arp tunnel-statistics` BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting Unicast Forwarding Data Collection – show tech forwarding.

Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public .

All rights reserved. Cisco Public .Troubleshooting Multicast Forwarding Distributing Streams in Hardware IIF Local OIF  Software learning and Module 1 Replication hardware entries mirror IPv4 Engine MET distribution model Fabric ASIC MD  Conservation of L2 replication Fabric Copy Fabric ASIC based on L3 address Module 2 3 4  Egress replication forwarding Fabric ASIC Fabric ASIC Fabric ASIC conserves hardware Replication Replication Replication resources Engine MET Engine MET Engine MET  Single copy sent across fabric conserves bandwidth BRKRST-3066 © 2013 Cisco and/or its affiliates.

Cisco Public .Troubleshooting Multicast Forwarding Data Collection – show tech ip pim. All rights reserved. show tech forwarding N7K1-VPC# show tech-support ip pim | grep "`show " `show running-config pim` `show ip pim group-range vrf all` `show ip pim interface vrf all` `show ip pim neighbor vrf all` `show ip pim route vrf all` [snip] N7K1-VPC# show tech-support forwarding l3 multicast | grep "`show " `show forwarding multicast outgoing-interface-list` `show forwarding ip multicast route summary vrf all ` `show system internal forwarding ip multicast route summary` `show forwarding ipv6 multicast route summary vrf all` `show system internal forwarding adjacency multicast` [snip] N7K1-VPC# show tech-support ip multicast | grep "`show " `show tech-support ip igmp` `show running-config igmp` `show ip igmp route vrf all` [snip] `show tech-support ip msdp` BRKRST-3066 © 2013 Cisco and/or its affiliates.

Cisco Public . All rights reserved.Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates.

added VoQ Fabric and MQC implementation BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting QoS Utilising Hardware Resources Effectively  Nexus 7000 QoS is on by default and cannot be disabled  System defined classification consistent across all VDCs  Trust on all ports. Cisco Public . marking can be modified with policy  Similar queuing to prior platforms. All rights reserved.

2MB egress buffer Output Queue bandwidth DSCP 32 Bandwidth Port Asic Port Asic Q1 Q1 10G 1G Fabric DSCP 48 65MB 6. All rights reserved.2MB Queue-limit Q2 Buffer Buffer Q2 Output Queue limit (size)  Hardware drops due to buffer oversubscription  Must ask: how much traffic is appropriate for this network? BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . Troubleshooting QoS How Drops Occur – Big Pipe to little pipe  10 Gig Module 65MB ingress buffer  1Gig module 6.

All rights reserved. based on port-type and queue  Configurable only in default VDC  Changes apply to ALL ports of specified type in ALL VDCs  Changes are traffic disruptive for ports of specified type BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .Troubleshooting QoS System wide Class-map configuration  Queuing class-map names are static.

Cisco Public . All rights reserved.Troubleshooting QoS System wide Class-map configuration N7K1(Config)#class-map type queuing match-any ? 1G Egress Port Type 10G Ingress Port Type 10G Egress Port Type 1p3q4t-out-pq1 1p3q4t-out-q-default 8q2t-in-q-default 1p7q4t-out-pq1 1p3q4t-out-q2 8q2t-in-q1 1p7q4t-out-q-default 1p3q4t-out-q3 8q2t-in-q2 1p7q4t-out-q2 8q2t-in-q3 1p7q4t-out-q3 8q2t-in-q4 1p7q4t-out-q4 8q2t-in-q5 1p7q4t-out-q5 1G Ingress Port Type 8q2t-in-q6 1p7q4t-out-q6 8q2t-in-q7 1p7q4t-out-q7 2q4t-in-q-default 2q4t-in-q1 BRKRST-3066 © 2013 Cisco and/or its affiliates.

All rights reserved. Cisco Public .Troubleshooting QoS Classification and Queuing – show policy-mpa interface N7K1-VPC# show policy-map int e2/17 Service-policy (queuing) output: default-out-policy SNMP Policy Index: 301990010 Priority queueing on by default Class-map (queuing): out-pq1 (match-any) priority level 1 queue-limit percent 16 queue dropped pkts : 0 Class-map (queuing): out-q2 (match-any) queue-limit percent 1 queue dropped pkts : 0 Class-map (queuing): out-q3 (match-any) Output drops will increment if queue-limit percent 1 queue limit exceeded queue dropped pkts : 0 Class-map (queuing): out-q-default (match-any) queue-limit percent 82 bandwidth remaining percent 25 queue dropped pkts : 0 BRKRST-3066 © 2013 Cisco and/or its affiliates.

Cisco Public . All rights reserved.Troubleshooting QoS Classification and Queuing – show queuing interface N7K1-VPC# show queuing int e2/17 Interface Ethernet2/17 TX Queuing strategy: Weighted Round-Robin Port QoS is enabled Queuing Mode in TX direction: mode-cos Transmit queues [type = 1p7q4t] Queue Id Scheduling Num of thresholds 1p7q4t-out-q-default WRR 04 1p7q4t-out-q2 WRR 04 1p7q4t-out-q3 WRR 04 1p7q4t-out-q4 WRR 04 1p7q4t-out-q5 WRR 04 1p7q4t-out-q6 WRR 04 Scheduling and queue-limit read 1p7q4t-out-q7 WRR 04 from hardware should match your 1p7q4t-out-pq1 Priority 04 configuration WRR configuration read from HW WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 11[1p7q4t-out-q3] 11[1p7q4t- out-q4] 11[1p7q4t-out-q5] 11[1p7q4t-out-q6] 11[1p7q4t-out-q7] queue-limit ratios configuration read from HW queue-limit ratios: 78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3] *1[1p7q4t-out- q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1] * means unused queue with mandatory minimum queue-limit BRKRST-3066 © 2013 Cisco and/or its affiliates.

All rights reserved. Cisco Public .21.23 - 37936 r2d2_tx_taildrop_drop_ctr_q3 0000000000010650 17. ---. ----.23 - |------------------------------------------------------------------------| | Device:Naxos Role:MAC SECURITY Mod: 2 | Per port output read | Last cleared @ Mon Jan 23 11:42:12 2012 from hardware device |------------------------------------------------------------------------| “Naxos” Instance:10 ID Name Value Ports -.19.Troubleshooting QoS Hardware Drop – show hardware internal statistics N7K1-VPC# show hardware internal error mod 2 |------------------------------------------------------------------------| | Device:R2D2 Role:MAC Mod: 2 | Tail drop only typically | Last cleared @ Mon Jan 23 11:42:12 2012 |------------------------------------------------------------------------| applicable Instance:5 ID Name Value Ports -.19. ----- 8194 mac_egress_taildrop_ctr 0000000000010680 21 - 8195 mac_egress_taildrop_ctr 0000000000010680 23 - BRKRST-3066 © 2013 Cisco and/or its affiliates. ---.21. ----. ----- 37920 r2d2_tx_taildrop_drop_ctr_q2 0000000000000030 17.

All rights reserved.Summary  NXOS offers a huge increase in available information over traditional data centre platforms  Familiarising yourself with the tools available gives you the best chance to understand your own network  Start capturing the right information at the right time BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

cisco.com/wiki/Cisco_Nexus_7000_Series_NX-OS_Troubleshooting_Guide  NXOS YouTube Intro http://www. All rights reserved.org/docs/wsug_html_chunked/ChCapCaptureFilterSection.com/wiki/Cisco_Nexus_7000_NX-OS/IOS_Comparison_Tech_Notes  Ethanalyzer Capture Filters http://www.cisco.References  Nexus 7000 Troubleshooting Guide http://docwiki.com/user/nxs7000  NXOS vs IOS Comparison Guide http://docwiki.html BRKRST-3066 © 2013 Cisco and/or its affiliates.wireshark.youtube. Cisco Public .

with NFS mount lifecycle. Cisco Public 61 .  Global system wide hardware feature checksum Enabled 0  Some IDS checks are disabled by protocol Enabled 0 default fragment Disabled -- length minimum Enabled 0 length consistent Enabled 0 length maximum max-frag Enabled 0 length maximum udp Disabled -- length maximum max-tcp Enabled 0 Fragmentation check can cause issues tcp flags Disabled -. disabled by tcp tiny-frag Enabled 0 default version Enabled 0 BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Troubleshooting CoPP IDS Protection – show hardware forwarding ip verify N7K-1# show hardware forwarding ip verify module 1 IPv4 and v6 IDS Checks Status Packets Failed -----------------------------+---------+------------------ address source broadcast Enabled 0 address source multicast Enabled 0  Intrusion Detection System (IDS) address destination zero Enabled 0 performed by forwarding engine address identical Disabled -- hardware address reserved Disabled -- address class-e Disabled -.

2012 Jan 25 11:42:10 Campus_N7K1 %PLATFORM-5-MOD_STATUS: Module 3 current-status is MOD_STATUS_ONLINE/OK 2012 Jan 25 11:42:10 Campus_N7K1 %MODULE-5-MOD_OK: Module 3 is online (serial: JAF1527BSJQ) 2012 Jan 25 11:42:09 Campus_N7K1 %SYSMGR-SLOT3-5-MODULE_ONLINE: System Manager has received notification of local module becoming online. at 683684 usecs after Wed Jan 25 11:42:10 2012 Instance:770. Troubleshooting Hardware Onboard Module Logging – show logging. Ret:SUCCESS [E_MTS_RX] Src:MTS_SAP_PLTFM_CONFIG(424). Cisco Public 62 . Opc:MTS_OPC_LC_INSERTED(1081) RRtoken:0x008067DE <snip> 87) FSM:<ID(770): Slot 3. show module internal event-history Campus_N7K1# show logging start-time 2012 Jan 25 11:42:00 2012 Jan 25 11:42:10 Campus_N7K1 %PLTFM_CONFIG-4-XL_LICENSE_MIX_NOTIFY: Mixed use of non-XL with XL modules in the same VDC may limit common resources to non-XL capacity. All rights reserved. Campus_N7K1# show module internal event-history module 3 85) Event:ESQ_RSP length:38. Seq Id:0x1. node 0x0302> Transition at 683727 usecs Wed Jan 25 11:42:10 2012 Previous state: [LCM_LC_ST_CHECK_INSERT_SEQUENCE] Triggered event: [LCM_EV_LC_ONLINE] Next state: [LCM_LC_ST_ONLINE] Curr state: [LCM_LC_ST_ONLINE] BRKRST-3066 © 2013 Cisco and/or its affiliates.

html BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 63 .com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/b_C onfiguring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter_010. Campus_N7K2-VPC(config)# feature-set fex FEX fabric port must be a port-channel Campus_N7K2-VPC(config)# interface e2/15 Campus_N7K2-VPC(config-if)# switchport Campus_N7K2-VPC(config-if)# switchport mode fex-fabric Campus_N7K2-VPC(config-if)# channel-group 5 mode on Campus_N7K2-VPC(config-if)# interface port-channel 5 Campus_N7K2-VPC(config-if)# fex associate 101 Campus_N7K2-VPC(config-if)# no shutdown FEX 101 downloading image from Campus_N7K2-VPC(config-if)# end Supervisor Campus_N7K2-VPC# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ 101 FEX0101 Image Download N2K-C2248TP-1GE JAF1449CPMR http://www...cisco. Troubleshooting Hardware Fabric Extender– install and configure FEX Campus_N7K2(config)# install feature-set fex SWITCH TO VDC VPC.

State: Active Image upgraded and ports available Eth2/15 .810122: Image Version Mismatch 01/25/2012 14:18:43.810957: Requesting satellite to download image Campus_N7K2-VPC# show fex 101 detail FEX: 101 Description: FEX0101 state: Online FEX version: 5. All rights reserved.Interface Up. State: Active Fex Port State Fabric Port Eth101/1/1 Down Po5 BRKRST-3066 © 2013 Cisco and/or its affiliates.Interface Up.2(1.1(3)N1(1) [Switch version: 5.2(1)] FEX Interim version: 5.2(1)] FEX Interim version: 5.810786: Registration response sent 01/25/2012 14:18:43.2(1) [Switch version: 5.1(3)N1(1) Switch Interim version: 5.809266: Module register received 01/25/2012 14:18:43. Troubleshooting Hardware Fabric Extender– show fex detail Campus_N7K2-VPC# show fex 101 detail FEX: 101 Description: FEX0101 state: Image Download FEX version: 5. Cisco Public 64 .13) Switch Interim version: 5.2(1) Fabric interface state: Po5 .2(1) <snip> Logs: Different image from previous install 01/25/2012 14:18:43.

156.5- software feature 12.1111.993-994.921.220- 222. All rights reserved.1999.2233- 2234. Cisco Public 65 .if_index: 0x16000013 the trunk ---------------------------------------------------------------- ---------------- if_index: 0x16000013 is_mcec: TRUE mcec_num : 20 Interface parameters synchronised Number of allowed vlans(cfg_vlans): 43.16.50.230.508. Troubleshooting vPC Per interface consistency check .2211-2213.200-201.555.100.show system internal vpcm Campus_N7K1-VPC# show system internal vpcm info interface port-channel 20 Configured vlans on port-channel20 .203.2901 <snip> Application database Information: Local Parameters:: ============================================== BRKRST-3066 © 2013 Cisco and/or its affiliates.1221. Bitset: per 1-3.20-23.60.32.1000.

200 Sw3 Sw4 172.40 L3 Stream from VLAN50 L3 Stream: igmp join-group 239.0.50.16.3 172.16.50.0.16.200.200.2 Po10 E2/1 E2/1 N7K-1 E2/2 E2/2 N7K-2 vPC PKL (L3) E10/25 E2/17 E2/17 E10/25 L2 Stream: Po20/V100 Po40/V50 L2 Stream from VLAN200 igmp join-group 239.16.2 172. Cisco Public 66 .70 BRKRST-3066 © 2013 Cisco and/or its affiliates.1 172.50.0. Troubleshooting Multicast Forwarding Example Topology 172.200.0.16.1 172. All rights reserved.16.

F .1 v2 00:03:33 Vlan200 (internal) BRKRST-3066 © 2013 Cisco and/or its affiliates.Fabricpath core port U .200 v2 D Po20 Campus_N7K1-VPC# show ip igmp snooping mrouter vlan 200 Type: S .Static.Internal. typically points towards mrouter Campus_N7K1-VPC# show ip igmp snooping group 239.Dynamic.0. D . Cisco Public 67 . Troubleshooting Multicast Forwarding L2 Software Entries– show ip igmp snooping Querier must be present. V . All rights reserved.User Configured Vlan Router-port Type Uptime Expires 200 Po20 D 6d03h 00:04:59 200 Vlan200 ID 6d03h 00:04:20 Campus_N7K1-VPC# show ip igmp snooping querier vlan 200 Vlan IP Address Version Expires Port 200 172.0.0. R .0.200.vPC Peer Link I .Dynamic. D .Static.200 vlan 200 Type: S .16. F .Router port.Fabricpath core port Vlan Group Address Ver Type Port list 200 239.

Troubleshooting Multicast Forwarding L2 MFDM Software Entries– show forwarding distribution multicast Campus_N7K1-VPC# show forwarding distribution ip igmp snooping vlan 200 group 239.0.200 Vlan: 200.0. Group: 239.200.0.0. Source: 0.0.0. Cisco Public 68 . All rights reserved.0 Outgoing Interface List Index: 5 Reference Count: 3 Platform Index: 0x7fed Vpc peer link exclude flag set Number of Outgoing Interfaces: 2 port-channel10 port-channel20 Campus_N7K1-VPC# show forwarding distribution multicast outgoing-interface-list L2 5 Outgoing Interface List Index: 5 Reference Count: 3 Platform Index: 0x7fed Vpc peer link exclude flag set Number of Outgoing Interfaces: 2 port-channel10 port-channel20 BRKRST-3066 © 2013 Cisco and/or its affiliates.

200 1 5 0x7fed 0x1e60f Campus_N7K1-VPC# show system internal ip igmp snooping vlan 200 group 239. All rights reserved.200 module 10 VDC: 4 Vlan Group Source Epoch RID DTL hwptr 200 239. Cisco Public 69 .0.200 1 12 0x7fed 0x1e60f Campus_N7K1-VPC# show system internal pixm info ltl 0x7fed MCAST LTLs allocated for VDC:4 ============================================ LTL IFIDX LTL_FLAG CB_FLAG MI[0] 0x7fed 0x0000000c 0x00 0x0002 0x002 ------------------ IFIDX LTL --------------------------------- Po20 0x0a42 Po10 0x0a40 BRKRST-3066 © 2013 Cisco and/or its affiliates.0. Troubleshooting Multicast Forwarding L2 Hardware Entries– show system internal ip igmp snooping Campus_N7K1-VPC# show system internal ip igmp snooping vlan 200 group 239.0.0.0.200 module 2 VDC: 4 Vlan Group Source Epoch RID DTL hwptr 200 239.0.0.0.

50.1 1.1 Outgoing interface list: (count: 1) Vlan200.70/32).50.123.1. pim Campus_N7K1-VPC# show ip pim interface brief PIM Interface Status for VRF "default" Interface IP Address PIM DR Address Neighbor Border Count Interface Vlan200 172.0.123/32.1. ip pim mrib Incoming interface: Vlan50.1 2 no loopback0 1. RPF nbr: 1.1.200. RPF nbr: 172. Troubleshooting Multicast Forwarding L3 Software Entries – show ip pim.1 172.0.1 0 no BRKRST-3066 © 2013 Cisco and/or its affiliates.16.50. show ip mroute Campus_N7K1-VPC# show ip mroute IP Multicast Routing Table for VRF "default" (*. uptime: 5d21h.1.0. All rights reserved.1. uptime: 5d21h.50. 239.200.16.1 2 no Vlan50 172. uptime: 5d21h. uptime: 5d21h.1 172.1. Cisco Public 70 .16.16.16. pim ip Incoming interface: loopback0.70/32).16.0. 239. pim (172. internal Outgoing interface list: (count: 1) Vlan200.

0.0. RPF Interface: Vlan50. flags: Received Packets: 422488 Bytes: 334427892 Number of Outgoing Interfaces: 1 Outgoing Interface List Index: 4 Vlan200 Outgoing Packets:370782 Bytes:293426795 Campus_N7K1-VPC# show forwarding multicast outgoing-interface-list 4 slot 2 Outgoing Interface List Index: 4 Reference Count: 4 Vlan200 BRKRST-3066 © 2013 Cisco and/or its affiliates.50.50.0. Troubleshooting Multicast Forwarding L3 Hardware Entries– show forwarding multicast route Campus_N7K1-VPC# show forwarding multicast route group 239.70 source 172.0. 239. RPF Interface: Vlan50.16.16.70/32). All rights reserved. 239.123/32. Cisco Public 71 .0.70/32).50.123/32. flags: Received Packets: 0 Bytes: 0 Number of Outgoing Interfaces: 1 Outgoing Interface List Index: 4 Vlan200 Outgoing Packets:1100739 Bytes:870975864 slot 10 ========== (172.16.123 slot 2 ========== (172.0.

Q&A .

All rights reserved. Solutions on Friday 8 March 12:00pm-2:00pm communities.com/mobile  Visit any Cisco Live Internet Station located Don’t forget to activate your throughout the venue Cisco Live 365 account for Polo Shirts can be collected in the World of access to all session material.com/portal/login. and on-demand and live activities throughout the year.ww BRKRST-3066 © 2013 Cisco and/or its affiliates. Log into your Cisco Live portal and click the "Enter Cisco Live 365" button. www.ciscoliveaustralia. Cisco Public 7 .Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2013 Polo Shirt! Complete your Overall Event Survey and 5 Session Evaluations.ciscoliveaustralia.  Directly from your mobile device on the Cisco Live Mobile App  By visiting the Cisco Live Mobile Site www.

All rights reserved. Cisco Public .BRKRST-3066 © 2013 Cisco and/or its affiliates.