Troubleshooting Nexus 7000
BRKRST-3066

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 2

Session Goal

To provide you with an understanding of the Cisco Nexus™
7000 built-in troubleshooting tools and troubleshooting
technique of NX-OS operating system
Increase familiarity with Cisco Nexus™ 7000 logging and
information capture
Ensure you get the right information at the right time

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Public . All rights reserved.Agenda  Introduce NXOS software architecture and logging capability  Define built-in troubleshooting tools that will assist with problem analysis and resolution  Demonstrate troubleshooting tools used when investigating common network functions BRKRST-3066 © 2013 Cisco and/or its affiliates.

NXOS Architecture Introduction Traditional IOS Versus NX-OS – Software Modularity BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public 5 . All rights reserved.

assists stateful information capture  More data requires more effective filtering  Capture feature related information for later analysis Cat6K#show tech | redirect bootflash:6k. Please Wait ..tech Cat6K#dir bootflash: 1. Cisco Public . 120MB file to review N7K1# dir bootflash: 120077767 Feb 10 16:18:40 2012 giant.2MB file to review Directory of bootflash:/ 9 -rwx 1194813 Feb 10 2012 06:42:30 +00:00 6k.NXOS Architecture Introduction IOS vs NX-OS – Show Tech  Logging output greatly increased..tech N7K1# show tech > bootflash:giant.tech Show tech brief will take 4-6 minutes to complete. All rights reserved.tech BRKRST-3066 © 2013 Cisco and/or its affiliates.

All rights reserved. Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates.

1002-1005. egrep. local BRKRST-3066 © 2013 Cisco and/or its affiliates.11. last.5- 9.1. Lo0.100. wc. last  Powerful CLI based on Linux Kernel  Built-in CLI filter techniques increases speed and relevance  grep. All rights reserved. start-time.Troubleshooting Tools CLI Capture and Filter – eigrp.1. sed. N7K1-VPC# show ip route | egrep "local$" *via 1. 04:30:29.1111 on Interface port-channel10 are being suspended. [0/0]. Cisco Public .203.200. start-time  Use the filtering options in real-time to increase your effectiveness N7K1-VPC# show logging logfile start-time 2012 Jan 11 16:00:00 2012 Jan 23 16:00:08 Campus_N7K1-VPC %ETHPORT-5-IF_DOWN_NONE: Interface port-channel40 is down (None) N7K1-VPC# show logging logfile | egrep "MAC|VLAN|ETHPM" 2012 Jan 23 11:42:49 Campus_N7K1-VPC %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1. include.1.

All rights reserved.254.992966 ospf: 6467 [3981] Invalid src address 192.0. Cisco Public .253.0.0 area:0.992780 ospf: 6467 [3981] Invalid src address 192. should not be seen on Vlan64 BRKRST-3066 © 2013 Cisco and/or its affiliates.254/Vlan64 2012 Mar 26 23:33:25.0.64.64.0 crc:0xfdd2 aut:0 aukid:0 from 192.253. should not be seen on Vlan64 2012 Mar 26 23:33:25.992586 ospf: 6467 [3981] (default) rcvd: prty:7 ver:2 t:HELLO len:44 rid:0.254. debug-filter  Debugging per feature logged to bootflash  Set a filter file and log to a new file N7K1-VPC# debug-filter ip ospf interface vlan 64 N7K1-VPC# debug logfile offending_traffic N7K1-VPC# debug ip ospf packets N7K1-VPC# undebug all N7K1-VPC# show debug logfile offending_traffic 2012 Mar 26 23:33:25.253.64.0.Troubleshooting Tools Debugging Log and Filter – debug logfile.

show <feature> internal  Syslogs and feature event history per Virtual Device Context (VDC)  Feature interaction tracked through ‘event-history’  Use syslogs with feature logging to compare feature behaviour N7K1-VPC# show logging logfile 2012 Jan 23 17:20:12 Campus_N7K1-VPC %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 50 on Interface port- channel20 are being suspended. Cisco Public . (Reason: Vlan is not allowed on Peer-link) N7K1-VPC# show system internal ethpm event-history interface port-channel 20 29) FSM:<port-channel20> Transition at 500252 usecs after Mon Jan 23 17:20:12 2012 Previous state: [ETH_PORT_FSM_ST_TRUNK_UP] Triggered event: [ETH_PORT_FSM_EV_LOGICAL_CHG] Next state: [ETH_PORT_FSM_ST_WAIT_LOGICAL_CHANGE_TRUNK] BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting Tools Logging and Feature History – show logging logfile. All rights reserved.

ERSPAN  ACL Log to flash N7K1-VPC(config)# monitor session 1 type ? acl-capture Create an acl-capture session erspan-destination Create an erspan destination session erspan-source Create an erspan source session local Create a local session N7K1-VPC(config)# monitor session 1 type local N7K1-VPC(config-monitor)# source interface e2/1 N7K1-VPC(config-monitor)# destination interface e2/2 N7K1-VPC(config-monitor)# no shut BRKRST-3066 © 2013 Cisco and/or its affiliates.SPAN  Data Plane SPAN in hardware  VACL.Troubleshooting Tools Data Plan Traffic Capture . Local SPAN. All rights reserved. Cisco Public .

per module logging  Review event history for failure detection  Always check diagnostics before troubleshooting! N7K1# show logging onboard module 2 exception-log ---------------------------- Module: 2 ---------------------------- Exception Log Record : Wed Nov 16 09:36:28 2011 (608385 us) N7K1# show diagnostic events 1) Event:E_DEBUG.Troubleshooting Tools Onboard Logging & Diagnostic Monitoring – show logging enabled. All rights reserved. show diagnostic  Persistent. length:115. at 475956 usecs after Mon Jan 23 16:17:07 2012 [104] Event_INFO: TestName->ASICRegisterCheck TestingType->health monitoring module->5 Result->pass Reason->Success BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

697098 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.0 (Request) BRKRST-3066 © 2013 Cisco and/or its affiliates.0.pcap Capturing on inband 100 Program exited with status 0.0.0.0.0. Cisco Public . N7K1# ethanalyzer local read bootflash:arp. All rights reserved.pcap limit-captured-frames 100 2012-01-26 10:13:18.Troubleshooting Tools CPU Traffic Capture .0.0 (Request) 2012-01-26 10:13:18.697182 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.697680 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.0 (Request) 2012-01-26 10:13:18.0.697430 00:00:00:00:01:23 -> ff:ff:ff:ff:ff:ff ARP Gratuitous ARP for 0.Ethanalyzer  Ethanalyzer built in Wireshark capture utility  Filter capture based on granular ‘tcpdump’ syntax  Display capture natively from NXOS console using Wireshark filters N7K1# ethanalyzer local interface inband capture-filter 'arp' limit-captured-frames 100 write bootflash:arp.0 (Request) 2012-01-26 10:13:18.0.

All rights reserved.Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

-----. 3164848K used. 4. ----------.Troubleshooting CPU Identify Process and Timeline – show system resources. 1 running CPU states : 0. show proc cpu  N7K Sup1 DualCore CPU with robust preemptive scheduling  High CPU process is not always an issue  Per VDC output. ----. 95. ----------- 3530 259509 119851 2165 1. 5096040K free N7K1# show proc cpu sort | grep -v 0.40 5 minutes: 0.0% kernel. -------.36 15 minutes: 0. Cisco Public .0 PID Runtime(ms) Invoked uSecs 1Sec Process ----.5% idle Memory usage: 8260888K total. identify the process using CPU resources N7K1# show system resources Load average: 1 minute: 0.9% platform Please note that only processes from the requested vdc are shown above BRKRST-3066 © 2013 Cisco and/or its affiliates.5% user. All rights reserved.28 Processes : 1091 total.

.3..6........4.3.....5. Cisco Public ......7.. All rights reserved....5.2..6.Troubleshooting CPU Identify Process and Timeline – show process cpu history N7K1# show proc cpu history  History shows combined utilisation across VDC <snip>  Compare problem timeline with CPU spikes 1 555554435555544354657844550 and average 466546476420229956261360720  Identify the start of utilisation issues 100 * 90 *  Snapshot to bootflash file for historical trending 80 * * 70 ** * 60 *** * * **** * * 50 ****** ***** * ******* *** 40 *************************** Boot up 26 hours earlier 30 *************************** 20 *************************** 10 ########################### 0.....1. 0 5 0 5 0 5 0 5 0 5 0 5 0 CPU% per hour (last 72 hours) * = maximum CPU% # = average CPU% BRKRST-3066 © 2013 Cisco and/or its affiliates........1.....5.4.....2.

0.8 0. 0 zombie Cpu(s): 9. 3 running. 0k buffers Swap: 0k total. 0.27. 0k used. 2. 591992k free.Troubleshooting CPU Module CPU Health Check – show system resources  CPU per module. 0.24 5 minutes: 0.0%wa.0% kernel.1%hi. 1.0% user. 1 user. 0 stopped.1%sy.0 0.6%ni. 15:34. 0k free.1 0.34 Tasks: 71 total. 68 sleeping. All rights reserved.8 3240:42 stats_client 1921 root -2 0 37296 9476 4116 S 15. 78.0%st Mem: 1035776k total.9 1534:01 naxos 1917 root -2 0 35656 6600 4524 S 2. 97. 2 running CPU states : 3.6 377:50. 0.8%id.40 15 minutes: 0.48 eureka BRKRST-3066 © 2013 Cisco and/or its affiliates. manages module processes.6%si. load average: 0. no network traffic module-1# show system resources Load average: 1 minute: 0. 592144K free module-1# show system internal processes cpu top .8%us.0% idle Memory usage: 1035776K total.34 Processes : 69 total. 443632K used. 7. 0. 443784k used. 93352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1935 root 25 5 39304 8352 3628 R 81. 0.42. Cisco Public .14:06:36 up 21 days.

84. ----. 21. -----.3% netstack 5840 162 38 4273 4.5% idle Please note that only processes from the requested vdc are shown above BRKRST-3066 © 2013 Cisco and/or its affiliates.9% netstack 3939 477316 717760 665 0. -------.0% user. we have PID Runtime(ms) Invoked uSecs 1Sec Process ----.0% idle Please note that only processes from the requested vdc are shown above N7K1# switchto vdc VPC Netstack and ARP are running N7K1-VPC# show proc cpu sort | ex 0. PID Runtime(ms) Invoked uSecs 1Sec Process each processes utilisation ----.0 higher than usual. -----. ----------.Troubleshooting CPU Receiving Traffic Sent to CPU – show proc cpu sort N7K1# show proc cpu | ex 0. -------. ----.0% user.0 Per VDC instance of netstack. ----------- separate from other 3904 204 74 2764 0. Cisco Public .5% kernel. All rights reserved. ----------.0% kernel.9% R2D2_usd CPU util : 12. ----------. 73. 4.7% arp CPU util : 5. traffic hitting CPU 5861 360 126 2861 11.

Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public .

66GHz Dual-Core 2GB 4GB Internal CF DRAM Main slot0: 2MB CPU NVRAM log-flash: BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .Troubleshooting CoPP Control Plane Policing – Protect CPU from your network Fabric ASIC Dedicated Arbitration Central Path VOQ Arbiter 1GE Inband System Controller 1. All rights reserved.

Cisco Public . All rights reserved. nd (ipv6)  DHCP traffic  Glean traffic (no arp or nd)  Two stage policing  Rate-limit in hardware per-module before sending to CPU  Policy based policing traffic that have passed rate-limiters  May require tweaking to match your network conditions BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting CoPP Control Plane Policing – Protect CPU from your network  Protect CPU from network traffic  arp.

Troubleshooting CoPP Control Plane Policing – show policy-map interface control-plane N7K1# show policy-map int control | grep –v Control Plane service-policy input: copp-system-policy class-map copp-system-class-exception (match-any) match exception ip option Exception traffic. bc 250 ms Implemented per-module. check module 2 : for violated traffic conformed 0 bytes. action: transmit Granular class based matching. action: drop module 10 : conformed 273344 bytes. CPU issues no match exception ip icmp unreachable longer created by ICMP exception match exception ipv6 option generation match exception ipv6 icmp unreachable set cos 1 police cir 360 kbps. violated 0 bytes. All rights reserved. action: drop module 3 : conformed 0 bytes. action: transmit violated 0 bytes. action: transmit violated 0 bytes. action: drop IP and MAC ACL in one class class-map copp-system-class-critical (match-any) match access-group name copp-system-acl-bgp BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

SNMP.Troubleshooting CoPP Control Plane Policing – show policy-map interface control-plane  Filter output to confirm affected classes  Check class-map definitions if violated traffic does not match your expectation N7K1# show policy-map int control | egrep "class-map|violated" class-map copp-system-class-critical (match-any) violated 0 bytes. action: drop 100Kbps default dropped traffic BRKRST-3066 © 2013 Cisco and/or its affiliates. violated 274930048 bytes. no drops violated 0 bytes. action: drop violated 0 bytes. action: drop Low rate policer on default class. FTP protocols class-map class-default (match-any) violated 0 bytes. action: drop violated 0 bytes. All rights reserved. action: drop Routing protocols class. action: drop violated 0 bytes. action: drop violated 0 bytes. action: drop TFTP. Cisco Public . action: drop class-map copp-system-class-management (match-any) violated 0 bytes.

All rights reserved.Troubleshooting CoPP Control Plane Policing – show hardware rate-limit  Per module rate-limiter in addition to control-plane policy N7K1# show hardware rate-limiter | grep –v Units for Config: packets per second Rate Limiter Class Parameters ------------------------------------------------------------ layer-3 control Config : 10000 Allowed : 0 Review layer-3 control if Dropped : 0 Total : 0 control-plane instablility <snip>  Hardware Rate-Limiter performed by forwarding engine hardware copy Copy to sup used for Config : 30000  Global system wide hardware Allowed : 3934270 feature ARP. limited before CoPP Dropped : 829800  Limit based on packet definitions Total : 4764070 receive Traffic destined to Sup Config : 30000 Allowed : 6875754 Dropped : 0 Total : 6875754 BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public .

.... Cisco Public ... 2 * 100ms Packet rate limit ............ 1203243 Traffic tail dropped......Troubleshooting CoPP Review Traffic Sent to CPU – show hardware internal inband cpu-mac stats N7K-1# show hardware internal cpu-mac inband stats RMON counters Rx Tx Total number of total packets 779905245 1421785114 good packets 779905245 1421650279 frames received and total octets (hi) 0 0 sent by CPU total octets (low) 172302724342 192974265660 Error counters --------------------------------+-- Rx no buffers .... 32000 pps Tick counter ............. All rights reserved. Throttle statistics after CoPP and RL Throttle interval ........ 12414130 CPU bound traffic Rx packet rate (current/max) 4993 / 20296 pps current pps Tx packet rate (current/max) 60 / 3474 pps MAC counters MAC0 (R2D2) MAC1 (CPU) /maximum pps Rx Tx Rx Tx reached XOFF packets auto-generated 5447 XOFF packets 7590855 6731953 XON packets 0 18561642 How many times did throttling kick in BRKRST-3066 © 2013 Cisco and/or its affiliates.........

define filters based on first capture  Capture to text file or .Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local interface  Start by capturing all traffic. Cisco Public .pcap  Local review or export to wireshark BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

wireshark.html BRKRST-3066 © 2013 Cisco and/or its affiliates.pcap Creates pcap file which can later be analysed by Capturing on inband GUI wireshark 100 Program exited with status 0. not allowed with ‘write’ keyword N7K1# ethanalyzer local int inband decode-internal limit-captured-frames 100 > cpu. Cisco Public . All rights reserved. Provides information from internal system headers.txt Filter syntax: http://www.Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local interface N7K1# ethanalyzer local int inband limit-captured-frame 100 write bootflash:cpu.org/docs/wsug_html_chunked/ChCapCaptureFilterSection.

388566 172.388940 172.123 -> 172.1 UDP Source port: 63 Destination port: 63 Local analysis completed from CLI prompt BRKRST-3066 © 2013 Cisco and/or its affiliates.50.16.123 -> 172.16.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.16.387691 172.16.123 -> 172.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.388066 172.60.60.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.pcap 2012-01-26 10:02:20.1 UDP Source port: 63 Destination port: 63 2012-01-26 10:02:20.387538 172.50.123 -> 172.60.123 -> 172. Cisco Public .16.60.Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local read N7K1# ethanalyzer local read bootflash:cpu.16.16.60. All rights reserved.16.50.16.50.16.50.

. . .. All rights reserved.. .... .000000000 seconds] [Time delta from previous displayed frame: 0... = LG bit: Globally unique address (factory default) BRKRST-3066 © 2013 Cisco and/or its affiliates... ..Troubleshooting CoPP Review Traffic Sent to CPU – ethanalyzer local read N7K1# ethanalyzer local read bootflash:cpu.. 2012 10:02:20... ..387538000 help to find relevant packets [Time delta from previous captured frame: 0.. . Dst: 04:c5:a4:e9:ac:44 (04:c5:a4:e9:ac:44) Destination: 04:c5:a4:e9:ac:44 (04:c5:a4:e9:ac:44) Address: 04:c5:a4:e9:ac:44 (04:c5:a4:e9:ac:44) . display filters Frame 1 (92 bytes on wire.. 60 bytes captured) Arrival Time: Jan 26.0. Src: 00:24:97:36:81:3f (00:24:97:36:81:3f).000000000 seconds] [Time since reference or first frame: 0..... = IG bit: Individual address (unicast) . Cisco Public .pcap detail Wireshark output....... .. .0 ..000000000 seconds] Frame Number: 1 Frame Length: 92 bytes Use filter in next capture to Capture Length: 60 bytes increase the relevance [Frame is marked: False] [Protocols in frame: eth:ip:udp:data] Ethernet II.....

Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

----. -----------------.-----------------. All rights reserved.Troubleshooting Hardware Module – show module. ---------- 2 32 10 Gbps Ethernet XL Module N7K-M132XP-12L ok 3 48 1000 Mbps Optical Ethernet Module N7K-M148GS-11 testing 5 0 Supervisor module-1X N7K-SUP1 active * 6 0 Supervisor module-1X N7K-SUP1 ha-standby 10 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok <snip> Mod Online Diag Status --.. ----------------------------------.. reload module <x> N7K1# show module Mod Ports Module-Type Model Status --. BRKRST-3066 © 2013 Cisco and/or its affiliates. Module booting during troubleshooting 2 Pass 3 Untested 5 Pass Reload from CLI denies module access 6 Pass 10 Pass to power shelf. only reseat will resolve physical issue N7K1# reload module 3 This command will reload module 3. Proceed[y/n]? [n] y reloading module 3 . Cisco Public .

. show diagnostic event N7K1# show diagnostic result mod 3 | grep -v ^$ Current bootup diagnostic level: complete Eliminates empty lines Module 3: 1000 Mbps Optical Ethernet Module Test results: (. . . . . . F = Fail.Troubleshooting Hardware Online Diagnostics Review – show diagnostic result. . . . . . . . . . . . 2) PrimaryBootROM----------------> . . 3) SecondaryBootROM--------------> . . . . . Port 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------------------------------------- . . . = Pass. A = Abort. . . . E = Error disabled) 1) ASICRegisterCheck-------------> . at 362751 usecs after Wed Jan 25 03:26:21 2012 [103] Event_ERROR: TestName->SpineControlBus TestingType->health monitoring module->5 Result->fail Reason-> XBar BRKRST-3066 © 2013 Cisco and/or its affiliates. I = Incomplete. Cisco Public . Initiated from module CPU. . All rights reserved. . . 5) OBFL--------------------------> . . length:217. . . N7K1# show diagnostic events error 1) Event:E_DEBUG. 4) EOBCPortLoopback--------------> . . . . U = Untested. . . . Port 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ----------------------------------------------------- . . . . . Per port diagnostic test. . . F could be transient or 6) PortLoopback: survive a module reset Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ----------------------------------------------------- .

Troubleshooting Hardware Fabric – show module. ----------------------------------. check B -. Cisco Public . ----. All rights reserved.Multicast/Multidestination fabric interface previous history for trending info -------------------------PEAK FABRIC UTILIZATION------------------------ I/O |-----FABRIC----| Ingress | Egress Slot |Mod Inst Plane| Util Time | Util Time ------------------------------------------------------------------------ 2 1 1 A 0% 01-23@16:41:46 0% 01-23@16:41:46 2 1 1 B 0% 01-23@16:41:46 0% 01-23@16:41:46 N7K1(config)# poweroff xbar 1 Power off xbar before trying hardware swap BRKRST-3066 © 2013 Cisco and/or its affiliates. show hardware capacity N7K1# show module xbar Xbar Ports Module-Type Model Status --. ---------- 1 0 Fabric Module 1 N7K-C7010-FAB-1 ok 2 0 Fabric Module 1 N7K-C7010-FAB-1 ok 3 0 Fabric Module 1 N7K-C7010-FAB-1 ok N7K1# show hardware capacity fabric-utilization ------------------------------------------------------------------------ Fabric Planes: A -.Unicast fabric interface Xbar utilization max. -----------------.

Agenda

 NXOS Architecture Introduction
 Built-in Troubleshooting Tools
 Troubleshooting Nexus 7000
 CPU
 Control-Plane – CoPP
 Hardware
 vPC
 Unicast Layer 2 and Layer 3 Forwarding and ARP
 Multicast Layer 2 and Layer 3 Forwarding
 QoS

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Troubleshooting vPC
Distributed Layer 2 Forwarding

 Designed as an alternative to 172.16.100.1 172.16.100.2
Po10
STP E2/1 E2/1

N7K-1 E2/2 E2/2 N7K-2
 Appear as one L2 device to the
vPC PKL (L3)
network E10/25 E2/17

 Peers must maintain consistent E2/17 E10/25

network view Po20/V100 Po40/V50

 Recovery mechanisms protect Sw3 Sw4
forwarding 172.16.100.3 172.16.50.40

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Troubleshooting vPC
Global Consistency Check – show vpc brief
N7K1-VPC# show vpc brief
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive Secondary role, vPCs
Configuration consistency status : success shut if peer-link is down
Per-vlan consistency status : success
Type-2 consistency status : failed
Type-2 inconsistency reason : SVI type-2 configuration incompatible
vPC role : secondary
Number of vPCs configured : 2
Peer Gateway : Enabled Peer gateway enabled;
<snip> routing for peer
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po10 up 1,5-9,11,100,200,203,1111 Type-2 Inconsistency
failed, but vlans still
vPC status
---------------------------------------------------------------------- passing on vPC and
id Port Status Consistency Reason Active vlans peer-link
-- ---- ------ ----------- ------ ------------
20 Po20 up success success 1,5-9,11,203
40 Po40 up success success 1,5-9,11,100,200,203,1111

BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

1000.1111.1 1-3. All rights reserved. Edge 1 Normal.203. ----------------------- STP Mode 1 Rapid-PVST Rapid-PVST STP Disabled 1 None None VTP enabled locally. BPDUFilter. 1. ---.2901 Local suspended VLANs .5-12. configured locally VTP password 2 - VTP pruning status 2 Disabled - Interface-vlan admin up 2 5-6. Edge BPDUGuard Disabled Disabled STP MST Simulate PVST 1 Enabled Enabled VTP domain 2 TEST - VTP version 2 1 .2233-22 34.156. Disabled. Normal.1111 00.16.5-6. 1999.20-23.100.200-201. ---------------------. 1002-1005 - BRKRST-3066 © 2013 Cisco and/or its affiliates.5-6.921.200 VLANs capability Allowed VLANs . SVI 200 not VTP mode 2 Server .32.1221.508.9 93-994.100 5-6.555.200. <snip> disabled on remote STP Port Type.220 -222.230.100. Cisco Public .203.5-9.2211-2213.100.Troubleshooting vPC Global Consistency Check – show vpc consistency parameters global N7K1-VPC# show vpc consistency-parameters global Type 1 : vPC will be suspended in case of mismatch Name Type Local Value Peer Value ------------.100 1.200 Mismatched allowed Interface-vlan routing 2 1. Disabled.1 002-1005.11.

aac4 static .a4e9. F F vPC Peer-Link(R) G 100 04c5. F F vPC Peer-Link(R) G 100 04c5.2740 dynamic 0 F F Po20 Peer MAC address with G-bit N7K1-VPC# show run vpc | egrep "peer-gateway" set. route traffic destined to peer peer-gateway BRKRST-3066 © 2013 Cisco and/or its affiliates.ac44 static .ac01 static .LID ---------+-----------------+--------+---------+------+----+------------------ G 100 0000.primary entry.seconds since last seen. All rights reserved. O .SSID.Routed MAC. F F sup-eth1(R) * 100 0021. (R) .Troubleshooting vPC MAC Address Synchronised – show mac address-table  MAC addresses synchronised via CFS  First Hop Redundancy Protocol addresses forwarded by both peers  Peer-gateway allows routing for peer SVI MAC N7K1-VPC# show mac add vlan 100 * .primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.a4e9.Overlay MAC age . G .Gateway MAC.+ .d87c.0c07. Cisco Public .

All rights reserved. show tech pixm N7K1-VPC# show tech-support vpc | grep "`show " `show version` `show module` `show vpc brief` `show vpc role` `show running-config vpc` `show system internal vpcm event-history global` `show system internal vpcm event-history errors` `show system internal vpcm event-history msgs` `show system internal vpcm event-history interactions` `show system internal vpcm mem-stats detail` `show system internal vpcm info all` `show system internal vpcm info global` `show cfs internal ethernet-peer database` `show spanning-tree` N7K1-VPC# show tech-support stp N7K1-VPC# show tech-support vtp N7K1-VPC# show tech-support pixm N7K1-VPC# show tech-support forwarding l2 unicast BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . show tech stp.Troubleshooting vPC Data Collection – show tech vpc.

All rights reserved. Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates.

2 Po10 forwarding engine per E2/1 E2/1 module N7K-1 E2/2 E2/2 N7K-2 vPC PKL (L3)  L3 entries learned by CPU E10/25 E2/17 and pushed to hardware E2/17 E10/25 engine Po20/V100 Po40/V50  Verify software entries Sw3 Sw4 against topology diagrams 172.100.16.16.100.50.40  Verify hardware entries against software entries BRKRST-3066 © 2013 Cisco and/or its affiliates.16.16. Cisco Public .100.Troubleshooting Unicast Forwarding Forwarding L2 and L3 in Hardware  L2 learning in hardware by 172. All rights reserved.1 172.3 172.

Cisco Public .Troubleshooting Unicast Forwarding Forwarding L2 and L3 in Hardware OSPF OSPF RIP IS-IS EIGRP Static BGP ARP v2 v3 AM u4RIB u6RIB Unicast Routing Information Base (uRIB) mRIB uFDM Supervisor FIB Manager I/O Module Forwarding Hardware BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

9736.primary entry. ubest/mbest: 1/0.primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID. attached *via 172.Troubleshooting Unicast Forwarding Software Tables – show ip route.seconds since last seen. [250/0].16.40 Flags: # .813f Legend: * . G . (R) . Cisco Public .Routed MAC.40 0024.813f dynamic 150 F F Po40 BRKRST-3066 © 2013 Cisco and/or its affiliates.16. show ip adjacency. Vlan50.Adjacencies Throttled for Glean IP Adjacency Table for VRF default Total number of entries: 1 Address MAC Address Pref Source Interface 172.50.50.LID ---------+-----------------+--------+---------+------+----+------------------ * 50 0024.50. O . show mac address-table N7K1-VPC# show ip route 172.50.+ .Gateway MAC.40/32.16.SSID.16.40. am N7K1-VPC# show ip adjacency 172.40 IP Route Table for VRF "default" '*' denotes best ucast next-hop 172.16.50. 00:02:09.9736.813f 50 arp Vlan50 Campus_N7K1-VPC# show mac address-table address 0024.9736.Overlay MAC age . All rights reserved.

0000.9736. All rights reserved. ------- 4 50 59 DMAC sent to LTL index for PO40 N7K1-VPC# show system internal pixm info ltl 0xa48 PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT ------------------------------------------------------------------------------ Normal Po40 0x0a48 0x16000027 0x00000000 0x00000002 1 BRKRST-3066 © 2013 Cisco and/or its affiliates.0127 0x07ff1 1 0x001 0 170 0 0 0 0 0 0 0 1 0 0 0 1 0 59 4180.0000.Troubleshooting Unicast Forwarding L2 Hardware Entries – show hardware mac-add.0c07.0123 0x00a48 0 0x003 0 156 1 0 0 0 0 0 0 0 0 0 0 1 1 59 0000. Cisco Public . show system internal pixm N7K1-VPC# show hardware mac add 2 vlan 50 FE | Valid| PI| BD | MAC | Index| Stat| SW | Modi| Age| Tmr| GM| Sec| TR| NT| RM| RMA| Cap| Fld|Always | | | | | | ic | | fied|Byte| Sel| | ure| AP| FY| | |TURE| | Learn ---+------+---+------+---------------+-------+-----+-----+-----+----+----+---+----+---+---+---+----+----+----+------ 0 1 0 59 0024.ac44 0x00400 1 0x000 0 130 1 1 0 0 0 0 0 0 0 0 0 1 0 59 4180.0128 0x07ff1 1 0x001 0 29 0 0 0 0 0 0 0 1 0 0 N7K1-VPC# show vlan internal bd-info vlan-to-bd 50 Allocated bridge-domain VDC Id Vlan Id BD Id matches in hardware table -----. ------.0000.813f 0x00a48 0 0x003 0 9 1 0 0 0 0 0 0 0 0 0 0 1 0 59 0000.ac01 0x00400 1 0x000 0 156 1 1 0 0 0 0 0 0 0 0 0 1 1 59 04c5.a4e9.

ac44 LIF: 0x3b (Vlan50) DI: 0x0 ccc: 4 L2_FWD: NO RDT: NO packets: 0 bytes: 0 zone enforce: 0 BRKRST-3066 © 2013 Cisco and/or its affiliates.50. show system internal forward N7K1-VPC# show ip fib route 172.16.16. Cisco Public .Troubleshooting Unicast Forwarding L3 Hardware Entries – show ip fib route <dest>.40 Vlan50 N7K1-VPC# show system internal forwarding ip route 172.50.50.16.40 module 2 IPv4 routes for table default/base ------------------+------------------+----------------------+----------------- Prefix | Next-hop | Interface | Labels ------------------+------------------+----------------------+----------------- 172.40/32 0x2033 0x43015 0 0x3b Campus_N7K1-VPC# show system internal forwarding adjacency mod 2 entry 0x43015 det Device: 1 Index: 0x43015 DMAC: 0024.16.40 module 2 ----+---------------------+----------+----------+------+----------- Dev | Prefix | PfxIndex | AdjIndex | LIFB | LIF ----+---------------------+----------+----------+------+----------- 1 172.16.50. All rights reserved.a4e9.9736.50.813f SMAC: 00c5.40/32 172.

Module 2 usage: Route Type Used %Used Free %Free Total (Log/Phys) (Log/Phys) (Log/Phys) ------------------------------------------------------------------------- IPv4 Unicast: 67/67 0 L2VPN Peer: 0/0 0 MPLS: 0/0 0 ------------------------------------- 904967/904967 99 905120/905120 ------------------------------------- BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.Troubleshooting Unicast Forwarding Scale and Utilsation – show ip route summary. Cisco Public . show hardware internal N7K1-VPC# show ip route summary IP Route Table for VRF "default" Total number of routes: 45 Total number of paths: 47 Best paths per protocol: Backup paths per protocol: am : 4 None local : 9 <snip> Number of routes per mask-length: /8 : 1 /24: 6 /30: 1 /32: 37 N7K1-VPC# show hardware internal forwarding table utilization module 2 Note: Utilization may not reach the maximum.

show tech arp N7K1-VPC# show tech-support forwarding L3 unicast | grep "`show " `show forwarding route summary vrf all` `show forwarding route max-display-count 100000 vrf all` `show forwarding vrf all adjacency` `show forwarding ipv6 route summary vrf all` `show forwarding ipv6 route max-display-count 100000 vrf all` `show forwarding vrf all ipv6 adjacency` `show forwarding trace` `show forwarding internal errors` `show forwarding internal error counts` `show forwarding internal unicast counts vdc all` `show forwarding internal message counts` N7K1-VPC# show tech-support netstack | grep "`show " | wc -l 212 N7K1-VPC# show tech-support arp | grep "`show " `show running-config arp` `show ip arp internal event-history cli` `show ip arp vrf all` `show ip arp static vrf all` `show ip arp summary vrf all` `show ip arp tunnel-statistics` BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. show tech netstack. Cisco Public .Troubleshooting Unicast Forwarding Data Collection – show tech forwarding.

Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public .Troubleshooting Multicast Forwarding Distributing Streams in Hardware IIF Local OIF  Software learning and Module 1 Replication hardware entries mirror IPv4 Engine MET distribution model Fabric ASIC MD  Conservation of L2 replication Fabric Copy Fabric ASIC based on L3 address Module 2 3 4  Egress replication forwarding Fabric ASIC Fabric ASIC Fabric ASIC conserves hardware Replication Replication Replication resources Engine MET Engine MET Engine MET  Single copy sent across fabric conserves bandwidth BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Multicast Forwarding Data Collection – show tech ip pim. Cisco Public . All rights reserved. show tech forwarding N7K1-VPC# show tech-support ip pim | grep "`show " `show running-config pim` `show ip pim group-range vrf all` `show ip pim interface vrf all` `show ip pim neighbor vrf all` `show ip pim route vrf all` [snip] N7K1-VPC# show tech-support forwarding l3 multicast | grep "`show " `show forwarding multicast outgoing-interface-list` `show forwarding ip multicast route summary vrf all ` `show system internal forwarding ip multicast route summary` `show forwarding ipv6 multicast route summary vrf all` `show system internal forwarding adjacency multicast` [snip] N7K1-VPC# show tech-support ip multicast | grep "`show " `show tech-support ip igmp` `show running-config igmp` `show ip igmp route vrf all` [snip] `show tech-support ip msdp` BRKRST-3066 © 2013 Cisco and/or its affiliates.

Cisco Public .Agenda  NXOS Architecture Introduction  Built-in Troubleshooting Tools  Troubleshooting Nexus 7000  CPU  Control-Plane – CoPP  Hardware  vPC  Unicast Layer 2 and Layer 3 Forwarding and ARP  Multicast Layer 2 and Layer 3 Forwarding  QoS BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public . marking can be modified with policy  Similar queuing to prior platforms. All rights reserved. added VoQ Fabric and MQC implementation BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting QoS Utilising Hardware Resources Effectively  Nexus 7000 QoS is on by default and cannot be disabled  System defined classification consistent across all VDCs  Trust on all ports.

All rights reserved.2MB Queue-limit Q2 Buffer Buffer Q2 Output Queue limit (size)  Hardware drops due to buffer oversubscription  Must ask: how much traffic is appropriate for this network? BRKRST-3066 © 2013 Cisco and/or its affiliates.2MB egress buffer Output Queue bandwidth DSCP 32 Bandwidth Port Asic Port Asic Q1 Q1 10G 1G Fabric DSCP 48 65MB 6. Cisco Public . Troubleshooting QoS How Drops Occur – Big Pipe to little pipe  10 Gig Module 65MB ingress buffer  1Gig module 6.

based on port-type and queue  Configurable only in default VDC  Changes apply to ALL ports of specified type in ALL VDCs  Changes are traffic disruptive for ports of specified type BRKRST-3066 © 2013 Cisco and/or its affiliates.Troubleshooting QoS System wide Class-map configuration  Queuing class-map names are static. Cisco Public . All rights reserved.

Troubleshooting QoS System wide Class-map configuration N7K1(Config)#class-map type queuing match-any ? 1G Egress Port Type 10G Ingress Port Type 10G Egress Port Type 1p3q4t-out-pq1 1p3q4t-out-q-default 8q2t-in-q-default 1p7q4t-out-pq1 1p3q4t-out-q2 8q2t-in-q1 1p7q4t-out-q-default 1p3q4t-out-q3 8q2t-in-q2 1p7q4t-out-q2 8q2t-in-q3 1p7q4t-out-q3 8q2t-in-q4 1p7q4t-out-q4 8q2t-in-q5 1p7q4t-out-q5 1G Ingress Port Type 8q2t-in-q6 1p7q4t-out-q6 8q2t-in-q7 1p7q4t-out-q7 2q4t-in-q-default 2q4t-in-q1 BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public .

Troubleshooting QoS Classification and Queuing – show policy-mpa interface N7K1-VPC# show policy-map int e2/17 Service-policy (queuing) output: default-out-policy SNMP Policy Index: 301990010 Priority queueing on by default Class-map (queuing): out-pq1 (match-any) priority level 1 queue-limit percent 16 queue dropped pkts : 0 Class-map (queuing): out-q2 (match-any) queue-limit percent 1 queue dropped pkts : 0 Class-map (queuing): out-q3 (match-any) Output drops will increment if queue-limit percent 1 queue limit exceeded queue dropped pkts : 0 Class-map (queuing): out-q-default (match-any) queue-limit percent 82 bandwidth remaining percent 25 queue dropped pkts : 0 BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . All rights reserved.

Cisco Public .Troubleshooting QoS Classification and Queuing – show queuing interface N7K1-VPC# show queuing int e2/17 Interface Ethernet2/17 TX Queuing strategy: Weighted Round-Robin Port QoS is enabled Queuing Mode in TX direction: mode-cos Transmit queues [type = 1p7q4t] Queue Id Scheduling Num of thresholds 1p7q4t-out-q-default WRR 04 1p7q4t-out-q2 WRR 04 1p7q4t-out-q3 WRR 04 1p7q4t-out-q4 WRR 04 1p7q4t-out-q5 WRR 04 1p7q4t-out-q6 WRR 04 Scheduling and queue-limit read 1p7q4t-out-q7 WRR 04 from hardware should match your 1p7q4t-out-pq1 Priority 04 configuration WRR configuration read from HW WRR bandwidth ratios: 25[1p7q4t-out-q-default] 15[1p7q4t-out-q2] 11[1p7q4t-out-q3] 11[1p7q4t- out-q4] 11[1p7q4t-out-q5] 11[1p7q4t-out-q6] 11[1p7q4t-out-q7] queue-limit ratios configuration read from HW queue-limit ratios: 78[1p7q4t-out-q-default] 1[1p7q4t-out-q2] 1[1p7q4t-out-q3] *1[1p7q4t-out- q4] *1[1p7q4t-out-q5] *1[1p7q4t-out-q6] *1[1p7q4t-out-q7] 16[1p7q4t-out-pq1] * means unused queue with mandatory minimum queue-limit BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public . ----- 37920 r2d2_tx_taildrop_drop_ctr_q2 0000000000000030 17.23 - 37936 r2d2_tx_taildrop_drop_ctr_q3 0000000000010650 17.Troubleshooting QoS Hardware Drop – show hardware internal statistics N7K1-VPC# show hardware internal error mod 2 |------------------------------------------------------------------------| | Device:R2D2 Role:MAC Mod: 2 | Tail drop only typically | Last cleared @ Mon Jan 23 11:42:12 2012 |------------------------------------------------------------------------| applicable Instance:5 ID Name Value Ports -.19. ----. ---.23 - |------------------------------------------------------------------------| | Device:Naxos Role:MAC SECURITY Mod: 2 | Per port output read | Last cleared @ Mon Jan 23 11:42:12 2012 from hardware device |------------------------------------------------------------------------| “Naxos” Instance:10 ID Name Value Ports -. ----.21. ----- 8194 mac_egress_taildrop_ctr 0000000000010680 21 - 8195 mac_egress_taildrop_ctr 0000000000010680 23 - BRKRST-3066 © 2013 Cisco and/or its affiliates.19. ---.21. All rights reserved.

Summary  NXOS offers a huge increase in available information over traditional data centre platforms  Familiarising yourself with the tools available gives you the best chance to understand your own network  Start capturing the right information at the right time BRKRST-3066 © 2013 Cisco and/or its affiliates. Cisco Public . All rights reserved.

All rights reserved. Cisco Public .com/wiki/Cisco_Nexus_7000_Series_NX-OS_Troubleshooting_Guide  NXOS YouTube Intro http://www.References  Nexus 7000 Troubleshooting Guide http://docwiki.com/wiki/Cisco_Nexus_7000_NX-OS/IOS_Comparison_Tech_Notes  Ethanalyzer Capture Filters http://www.wireshark.html BRKRST-3066 © 2013 Cisco and/or its affiliates.org/docs/wsug_html_chunked/ChCapCaptureFilterSection.cisco.com/user/nxs7000  NXOS vs IOS Comparison Guide http://docwiki.cisco.youtube.

with NFS mount lifecycle. Cisco Public 61 . disabled by tcp tiny-frag Enabled 0 default version Enabled 0 BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Troubleshooting CoPP IDS Protection – show hardware forwarding ip verify N7K-1# show hardware forwarding ip verify module 1 IPv4 and v6 IDS Checks Status Packets Failed -----------------------------+---------+------------------ address source broadcast Enabled 0 address source multicast Enabled 0  Intrusion Detection System (IDS) address destination zero Enabled 0 performed by forwarding engine address identical Disabled -- hardware address reserved Disabled -- address class-e Disabled -.  Global system wide hardware feature checksum Enabled 0  Some IDS checks are disabled by protocol Enabled 0 default fragment Disabled -- length minimum Enabled 0 length consistent Enabled 0 length maximum max-frag Enabled 0 length maximum udp Disabled -- length maximum max-tcp Enabled 0 Fragmentation check can cause issues tcp flags Disabled -.

Opc:MTS_OPC_LC_INSERTED(1081) RRtoken:0x008067DE <snip> 87) FSM:<ID(770): Slot 3. Seq Id:0x1. Campus_N7K1# show module internal event-history module 3 85) Event:ESQ_RSP length:38. All rights reserved. 2012 Jan 25 11:42:10 Campus_N7K1 %PLATFORM-5-MOD_STATUS: Module 3 current-status is MOD_STATUS_ONLINE/OK 2012 Jan 25 11:42:10 Campus_N7K1 %MODULE-5-MOD_OK: Module 3 is online (serial: JAF1527BSJQ) 2012 Jan 25 11:42:09 Campus_N7K1 %SYSMGR-SLOT3-5-MODULE_ONLINE: System Manager has received notification of local module becoming online. Troubleshooting Hardware Onboard Module Logging – show logging. Cisco Public 62 . node 0x0302> Transition at 683727 usecs Wed Jan 25 11:42:10 2012 Previous state: [LCM_LC_ST_CHECK_INSERT_SEQUENCE] Triggered event: [LCM_EV_LC_ONLINE] Next state: [LCM_LC_ST_ONLINE] Curr state: [LCM_LC_ST_ONLINE] BRKRST-3066 © 2013 Cisco and/or its affiliates. Ret:SUCCESS [E_MTS_RX] Src:MTS_SAP_PLTFM_CONFIG(424). show module internal event-history Campus_N7K1# show logging start-time 2012 Jan 25 11:42:00 2012 Jan 25 11:42:10 Campus_N7K1 %PLTFM_CONFIG-4-XL_LICENSE_MIX_NOTIFY: Mixed use of non-XL with XL modules in the same VDC may limit common resources to non-XL capacity. at 683684 usecs after Wed Jan 25 11:42:10 2012 Instance:770.

com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/b_C onfiguring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2_chapter_010... Troubleshooting Hardware Fabric Extender– install and configure FEX Campus_N7K2(config)# install feature-set fex SWITCH TO VDC VPC. Campus_N7K2-VPC(config)# feature-set fex FEX fabric port must be a port-channel Campus_N7K2-VPC(config)# interface e2/15 Campus_N7K2-VPC(config-if)# switchport Campus_N7K2-VPC(config-if)# switchport mode fex-fabric Campus_N7K2-VPC(config-if)# channel-group 5 mode on Campus_N7K2-VPC(config-if)# interface port-channel 5 Campus_N7K2-VPC(config-if)# fex associate 101 Campus_N7K2-VPC(config-if)# no shutdown FEX 101 downloading image from Campus_N7K2-VPC(config-if)# end Supervisor Campus_N7K2-VPC# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ 101 FEX0101 Image Download N2K-C2248TP-1GE JAF1449CPMR http://www. All rights reserved. Cisco Public 63 .html BRKRST-3066 © 2013 Cisco and/or its affiliates.cisco.

Cisco Public 64 .2(1. State: Active Fex Port State Fabric Port Eth101/1/1 Down Po5 BRKRST-3066 © 2013 Cisco and/or its affiliates.1(3)N1(1) [Switch version: 5.2(1)] FEX Interim version: 5.809266: Module register received 01/25/2012 14:18:43. Troubleshooting Hardware Fabric Extender– show fex detail Campus_N7K2-VPC# show fex 101 detail FEX: 101 Description: FEX0101 state: Image Download FEX version: 5.810786: Registration response sent 01/25/2012 14:18:43.810122: Image Version Mismatch 01/25/2012 14:18:43.2(1) <snip> Logs: Different image from previous install 01/25/2012 14:18:43.2(1)] FEX Interim version: 5.13) Switch Interim version: 5. All rights reserved. State: Active Image upgraded and ports available Eth2/15 .1(3)N1(1) Switch Interim version: 5.Interface Up.Interface Up.2(1) Fabric interface state: Po5 .2(1) [Switch version: 5.810957: Requesting satellite to download image Campus_N7K2-VPC# show fex 101 detail FEX: 101 Description: FEX0101 state: Online FEX version: 5.

1111.100.203.32.993-994.1000.5- software feature 12.20-23.1221.200-201.555.60.156.1999.16. All rights reserved.230.2901 <snip> Application database Information: Local Parameters:: ============================================== BRKRST-3066 © 2013 Cisco and/or its affiliates.921.2211-2213.508.if_index: 0x16000013 the trunk ---------------------------------------------------------------- ---------------- if_index: 0x16000013 is_mcec: TRUE mcec_num : 20 Interface parameters synchronised Number of allowed vlans(cfg_vlans): 43. Cisco Public 65 . Bitset: per 1-3.show system internal vpcm Campus_N7K1-VPC# show system internal vpcm info interface port-channel 20 Configured vlans on port-channel20 .50. Troubleshooting vPC Per interface consistency check .220- 222.2233- 2234.

40 L3 Stream from VLAN50 L3 Stream: igmp join-group 239.50.0.200 Sw3 Sw4 172. Troubleshooting Multicast Forwarding Example Topology 172.200. All rights reserved.16.16.50.0.0. Cisco Public 66 .3 172.16.70 BRKRST-3066 © 2013 Cisco and/or its affiliates.2 Po10 E2/1 E2/1 N7K-1 E2/2 E2/2 N7K-2 vPC PKL (L3) E10/25 E2/17 E2/17 E10/25 L2 Stream: Po20/V100 Po40/V50 L2 Stream from VLAN200 igmp join-group 239.0.200.1 172.16.16.50.200.1 172.16.2 172.

R .0.vPC Peer Link I .Dynamic. Troubleshooting Multicast Forwarding L2 Software Entries– show ip igmp snooping Querier must be present.User Configured Vlan Router-port Type Uptime Expires 200 Po20 D 6d03h 00:04:59 200 Vlan200 ID 6d03h 00:04:20 Campus_N7K1-VPC# show ip igmp snooping querier vlan 200 Vlan IP Address Version Expires Port 200 172.Dynamic.Internal.Router port.200 vlan 200 Type: S . D .Static. D . typically points towards mrouter Campus_N7K1-VPC# show ip igmp snooping group 239. Cisco Public 67 .0.16.0.Fabricpath core port U .200 v2 D Po20 Campus_N7K1-VPC# show ip igmp snooping mrouter vlan 200 Type: S . V .0. F .200. F .1 v2 00:03:33 Vlan200 (internal) BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.Static.Fabricpath core port Vlan Group Address Ver Type Port list 200 239.

0.0.0 Outgoing Interface List Index: 5 Reference Count: 3 Platform Index: 0x7fed Vpc peer link exclude flag set Number of Outgoing Interfaces: 2 port-channel10 port-channel20 Campus_N7K1-VPC# show forwarding distribution multicast outgoing-interface-list L2 5 Outgoing Interface List Index: 5 Reference Count: 3 Platform Index: 0x7fed Vpc peer link exclude flag set Number of Outgoing Interfaces: 2 port-channel10 port-channel20 BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 68 .0.200.0.0.200 Vlan: 200.0. Source: 0. Troubleshooting Multicast Forwarding L2 MFDM Software Entries– show forwarding distribution multicast Campus_N7K1-VPC# show forwarding distribution ip igmp snooping vlan 200 group 239. Group: 239.

0.200 1 12 0x7fed 0x1e60f Campus_N7K1-VPC# show system internal pixm info ltl 0x7fed MCAST LTLs allocated for VDC:4 ============================================ LTL IFIDX LTL_FLAG CB_FLAG MI[0] 0x7fed 0x0000000c 0x00 0x0002 0x002 ------------------ IFIDX LTL --------------------------------- Po20 0x0a42 Po10 0x0a40 BRKRST-3066 © 2013 Cisco and/or its affiliates.200 1 5 0x7fed 0x1e60f Campus_N7K1-VPC# show system internal ip igmp snooping vlan 200 group 239.0.0.0.200 module 10 VDC: 4 Vlan Group Source Epoch RID DTL hwptr 200 239. Troubleshooting Multicast Forwarding L2 Hardware Entries– show system internal ip igmp snooping Campus_N7K1-VPC# show system internal ip igmp snooping vlan 200 group 239.0.0. Cisco Public 69 . All rights reserved.0.0.200 module 2 VDC: 4 Vlan Group Source Epoch RID DTL hwptr 200 239.

pim ip Incoming interface: loopback0. RPF nbr: 172.1 Outgoing interface list: (count: 1) Vlan200. ip pim mrib Incoming interface: Vlan50. RPF nbr: 1. uptime: 5d21h.1 172. 239. pim (172.50.70/32).1. show ip mroute Campus_N7K1-VPC# show ip mroute IP Multicast Routing Table for VRF "default" (*.123. internal Outgoing interface list: (count: 1) Vlan200.0.16.1 1. pim Campus_N7K1-VPC# show ip pim interface brief PIM Interface Status for VRF "default" Interface IP Address PIM DR Address Neighbor Border Count Interface Vlan200 172.1.200. uptime: 5d21h.16.1 172. All rights reserved.50. Troubleshooting Multicast Forwarding L3 Software Entries – show ip pim. Cisco Public 70 .1.70/32).1 2 no loopback0 1. 239.1 0 no BRKRST-3066 © 2013 Cisco and/or its affiliates.16.16.1 2 no Vlan50 172. uptime: 5d21h.50.200.16. uptime: 5d21h.1.1.50.0.0.16.123/32.0.1.

70/32).70 source 172. 239.16.70/32).0.0. RPF Interface: Vlan50.50. Cisco Public 71 .123/32.0.0.16.16.123/32.123 slot 2 ========== (172.50. RPF Interface: Vlan50. 239.50.0. All rights reserved. flags: Received Packets: 0 Bytes: 0 Number of Outgoing Interfaces: 1 Outgoing Interface List Index: 4 Vlan200 Outgoing Packets:1100739 Bytes:870975864 slot 10 ========== (172. flags: Received Packets: 422488 Bytes: 334427892 Number of Outgoing Interfaces: 1 Outgoing Interface List Index: 4 Vlan200 Outgoing Packets:370782 Bytes:293426795 Campus_N7K1-VPC# show forwarding multicast outgoing-interface-list 4 slot 2 Outgoing Interface List Index: 4 Reference Count: 4 Vlan200 BRKRST-3066 © 2013 Cisco and/or its affiliates. Troubleshooting Multicast Forwarding L3 Hardware Entries– show forwarding multicast route Campus_N7K1-VPC# show forwarding multicast route group 239.0.

Q&A .

All rights reserved. Log into your Cisco Live portal and click the "Enter Cisco Live 365" button. Cisco Public 7 . and on-demand and live activities throughout the year.com/portal/login.com/mobile  Visit any Cisco Live Internet Station located Don’t forget to activate your throughout the venue Cisco Live 365 account for Polo Shirts can be collected in the World of access to all session material. Solutions on Friday 8 March 12:00pm-2:00pm communities.ww BRKRST-3066 © 2013 Cisco and/or its affiliates.  Directly from your mobile device on the Cisco Live Mobile App  By visiting the Cisco Live Mobile Site www. www.Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2013 Polo Shirt! Complete your Overall Event Survey and 5 Session Evaluations.ciscoliveaustralia.ciscoliveaustralia.

Cisco Public .BRKRST-3066 © 2013 Cisco and/or its affiliates. All rights reserved.