You are on page 1of 200

Advanced Firewalls

Session BRKSEC-3020

Agenda
  Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

2

Packet Flow

Understanding the Packet Flow
  To effectively troubleshoot a problem, one must first understand the packet path through the network   Attempt to isolate the problem down to a single device   Then perform a systematic walk of the packet path through the device to determine where the problem could be   For problems relating to the Cisco ASA/FWSM, always
Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol Determine the interfaces through which the flow passes

Note: All Firewall Issues Can Be Simplified to Two Interfaces (Ingress and Egress) and the Rules Tied to Both
BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public

4

219.1.1. Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside BRKSEC-3020 © 2011 Cisco and/or its affiliates.9 DST IP: 198.133. Cisco Public 5 .9 Packet Flow Eng Accounting Outside Server: 198. All rights reserved.133.1.25 SRC Port: 11030 DST Port: 80 Destination: Outside Servers Protocol: TCP   Interfaces Source: Inside Client: 10.219.Example Flow   Flow SRC IP: 10.25 With the Flow Defined.1.

0 no buffer Received 29519 broadcasts.Packet Processing: Ingress Interface Ingress Interface   Packet arrives on ingress interface   Input counters incremented   Software input queue is an indicator of load   No buffers indicates packet drops. 0 overrun. BW 1 Gbit full duplex 5912749 packets input. typically due to bursty traffic ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up. 0 CRC. 0 giants 0 input errors.1. All rights reserved.6214 IP address 10. address is 0003. Cisco Public 6 . 377701207 bytes. 0 abort 286298 packets output. line protocol is up Hardware is i82543 rev02 gigabit ethernet.255. subnet mask 255.470d. 18326033 bytes. 0 underruns input queue (curr/max blocks): hardware (4/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0) BRKSEC-3020 © 2011 Cisco and/or its affiliates.0 MTU 1500 bytes. 0 frame.1. 0 ignored.1. 0 runts.255.

1.1.1. drop and log TCP SYN or UDP packet. pass to ACL checks Established Connection: ASA-5540# show conn TCP out 198.219. bypass ACL check   If no existing connection TCP non-SYN packet.133.9:11030 idle 0:00:04 Bytes 1293 flags UIO Syslog Because of No Connection. Cisco Public 7 . and Non-SYN Packet: ASA-6-106015: Deny TCP (no connection) from 10.1.9/11031 to 198.133. All rights reserved. flow is matched.219.25/80 flags PSH ACK on interface inside BRKSEC-3020 © 2011 Cisco and/or its affiliates.Packet Processing: Locate Connection Existing Conn   Check first for existing connection   If connection exists.25:80 in 10.

1. Cisco Public 8 .9/11034 dst outside: 198. All rights reserved.219.Packet Processing: ACL Check ACL Permit   First packet in flow is processed through interface ACLs   ACLs are first match   First packet in flow matches ACE.1.255.0 255.1.0 any (hitcnt=1) Syslog When Packet Is Denied by ACL: ASA-4-106023: Deny tcp src inside:10.25/80 by access-group "inside" BRKSEC-3020 © 2011 Cisco and/or its affiliates. incrementing hit count by one   Denied packets are dropped and logged Packet Permitted by ACL: ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.133.1.255.

  After-Auto NAT entries BRKSEC-3020 © 2011 Cisco and/or its affiliates. FWSM best match) 4.  Match nat commands First Match First Match 1.3 Version 8. Cisco Public 9 .  Match existing xlates 3.Translation and NAT Order of Operations Match xLate For your reference Translation Matching Pre version 8.  nat 0 access-list (nat-exempt) 2.  Match static commands (Cisco ASA/PIX first match.3+ 1.  Auto NAT entries 3. All rights reserved.  Manual NAT entries 2.

Packet Processing: Inspections/Sec Checks Inspections Sec Checks   Inspections are applied to ensure protocol compliance   (Optional) customized AIC inspections   NAT-embedded IPs in payload Question!   Additional security checks are applied to the packet What command will show you if packets are being dropped   (Optional) packets passed to Content Security and Control (CSC) module one of the Inspection engines? by Syslog from Packets Denied by Security Check: ASA-4-406002: FTP port command different address: 10.21) to 209.2.168.202.165.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP BRKSEC-3020 © 2011 Cisco and/or its affiliates.21 (192.252. All rights reserved. Cisco Public 10 .1.

All rights reserved. pass packet to IPS (AIP) module BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 11 .Packet Processing: NAT IP Header Nat IP Header   Translate the IP address in the IP header   Translate the port if performing PAT   Update checksums   (Optional) Following the above.

0.4 Inbound Packets to 192.0 static (dmz..16. outside) 192.12.0 172.16.0 BRKSEC-3020 © 2011 Cisco and/or its affiliates.0/24 172.0 netmask 255.4 Get Routed to Inside Based on Order of Statics static (inside.e.16.0.0 172. outbound initial packet) the results of a global route lookup are used to determine egress interface   Example: Inside 172.168.12.0.168.16.168.0 netmask 255.16. outside) 192.0. Cisco Public 12 . not forwarded to the driver yet)   Egress interface is determined first by translation rules   If translation rules do not specify egress interface (e..0/16 Outside DMZ 172.12.255. All rights reserved.12.255.g.12.Packet Processing: Egress Interface Egress Interface   Packet is virtually forwarded to egress interface (i.255.

124.18. All rights reserved.220/59138 to dmz:172.168. even though the routing table may point to a different interface Syslog from Packet on Egress Interface with No Route Pointing Out Interface: %ASA-6-110003: Routing failed to locate next hop for TCP from inside: 192.76/23 BRKSEC-3020 © 2011 Cisco and/or its affiliates.Packet Processing: L3 Route Lookup L3 Route   Once on egress interface. an interface route lookup is performed   Only routes pointing out the egress interface are eligible   Remember: translation rule can forward the packet to the egress interface. Cisco Public 13 .103.

Layer 2 resolution is performed   Layer 2 rewrite of MAC header   If Layer 2 resolution fails—no syslog   show arp will not display an entry for the L3 next hop   debug arp will indicate if we are not receiving an ARP reply BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 .Packet Processing: L2 Address Lookup L2 Addr   Once a Layer 3 route has been found. and next hop identified.

DLY 10 usec Auto-Duplex(Full-duplex). 0 frame.0 4337255 packets input.36.103. 394043049 bytes. 0 giants 0 input errors. Auto-Speed(1000 Mbps MAC address 5475. MTU 1500 IP address 14. 0 CRC. Cisco Public 15 . is up. line protocol is up Hardware is bcm56801 rev 01.0.Packet Processing: Transmit Packet Xmit Pkt   Packet is transmitted on wire   Interface counters will increment on interface ASA-5585# show int Gig0/0 Interface GigabitEthernet0/0 "outside". All rights reserved. 0 no buffer Received 1957325 broadcasts. 0 underruns 0 pause/resume output BRKSEC-3020 © 2011 Cisco and/or its affiliates. subnet mask 255. 28855690 bytes. 0 abort 0 pause/resume input 0 switch ingress policy drops 282901 packets output. 0 runts.255.d05b. BW 1000 Mbps. 0 ignored.96.0fa6. 0 overrun.

Cisco Public 16 . All rights reserved.Agenda   Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Cisco Public 17 .   Architecture optimized for multi-flow traffic patterns   ASASM packet processing is also done in software.Cisco ASA — Understanding the Architecture   ASA processes all packets in software (via the central CPU) All packets are processed first in… usually also first out   ASA platforms have software imposed connection limits   Multi-CPU / Multi-Core systems hash packets in the same flow to the same CPU/core. unlike FWSM BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.   10 Gig interfaces hash flow to same RX ring.

  Each ACE uses a minimum of 212 bytes of RAM   However. Bound only by Memory. maximum performance may decrease (typically 10-15%) as you reach or exceed the Max Recommended ACEs. 5505 Max Recommended ACEs Tested ACEs Max Observed (from customers) 25k 5510 80k 80k 5520 200k 300k 5540 500k 700k 5550 700k 700k 2.77 million 5585 10/20/40/60 500k / 750k 1 / 2 million 500k / 750k 1 / 2 million ASA SM 2 million 2 million Note: Issue show access-list | include elements to see how many ACEs you have BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 .Maximum ACL Limits   No hard-coded limit on the number of elements (ACEs) in an ACL.74 million 5580 750k 1 million+ 2.

Available starting in 8.3(1) ASA-5585(config)# object-group-search access-control BRKSEC-3020 © 2011 Cisco and/or its affiliates.255 rules (ACEs) •  New command to reduce ACL memory impact for large ACLs. Cisco Public 19 .Warning .930 rules Single line ACL explodes to   Nested object-groups: •  Assume you add a SRC object-group to the above. which contains 25 additional sources •  Result: (10+25)x21x33 = 24.ACE Explosion   Object-groups: •  Sources (10 addresses) •  Destinations (21 addresses) •  Ports (33 ports) •  Result: 10x21x33 = 6. All rights reserved.

All rights reserved.3   Best used for new installations. Cisco Public 20 . or migration from other vendors access-group <access_list> global ASA Only Policy Ordering Interface Specific access-list Global access-list Default (implicit) deny ip any any BRKSEC-3020 © 2011 Cisco and/or its affiliates.Global ACLs Interface Independent Policies   Global ACLs introduced in version 8.

Object-NAT (Auto-NAT) (version 8.3+)
  Object NAT is the simplest form of NAT, and is defined within an object
Host NAT object network obj-WebServer host 10.3.19.50 nat (inside,outside) static 198.51.100.50 Network NAT object network Servers subnet 10.0.54.0 255.255.255.0 nat (inside,outside) static 203.0.113.0 Dynamic PAT (interface overload) object network InternalUsers subnet 192.168.2.0 255.255.255.0 nat (inside,outside) dynamic interface
BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public

21

Manual NAT (Twice NAT) (version 8.3+)
  Manual NAT should be used to translate the destination, or for policy NAT
object network ServerReal host 10.3.19.50 object network ServerTrans host 198.51.100.50 object network RemoteSite subnet 10.0.0.0 255.255.255.0 Static NAT nat (inside,outside) source static ServerReal ServerTrans

Static Policy NAT nat (inside,outside) source static ServerReal ServerTrans destination static RemoteSite RemoteSite
BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public

22

NAT Order of Operation version 8.3+

For your reference

  The ASA configuration is built into the NAT Table (show nat)   The NAT Table is based on First Match (top to bottom)
NAT Table Static NAT
Longest Prefix

Manual NAT Policies (Section 1)

First Match (in config)

Shortest Prefix

Dynamic NAT
Longest Prefix

Auto NAT Policies (Section 2)

Shortest Prefix

Manual NAT [after auto] Policies (Section 3)

First Match (in config)

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

23

Real-IP (version 8.3+)
  Finally, a reminder that with 8.3+ Real-IPs are used in ACLs

Real, Un-translated address object network obj-WebServer of internal Server host 10.3.19.50 nat (inside,outside) static 198.51.100.50 ! access-list allowIn permit tcp any host 10.3.19.50 eq 80 ! access-group allowIn in interface outside

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

24

All rights reserved. this has zero impact on packets processed in hardware   Note that FWSM packet processing is different from ASA BRKSEC-3020 © 2011 Cisco and/or its affiliates. with Some Packets Needing to be Processed in Software —via the Control Point (CP)   Packets processed in hardware have zero impact on CPU   Similarly. if the CPU is pegged at 100%.FWSM—Understanding the Architecture FWSM Process Most Packets in Hardware. Cisco Public 25 .

Security Checks and NAT in Hardware Fast Path NP 1 Fast Path NP 2 FWSM C6K Backplane Interface BRKSEC-3020 © 2011 Cisco and/or its affiliates. ACLs Fast Path Flow Identification. All rights reserved. AAA Cache.FWSM Architectural Overview Control Point (CP) Central CPU Software Hardware Session Manager NP 3 Control Point ACL Compilation. AAA. IPv6 in Software Session Manager Session Establishment and Teardown. Fixups. Syslog. Cisco Public 26 .

All rights reserved.2 / 4.200) 6.0 /4.498 (384) 256K (256K) 999.801) 8.990 (999.1 Configurable X X 56.0/4.417) 2764 (425) Cisco Public 4.990) 32K (32K) 4147 (1.704) 3.843 (283) 256K (256K) 999. Appendix A (Specifications) © 2011 Cisco and/or its affiliates.990) 32K (32K) 32 (32 per) 3942 (606) X X X 27 *Complete List in FWSM Docs.806 (11.1 2.2 (Multimode) 72. but vary based on single or multimode   Some limits include: Increase over 2.990 (999.3 Increase over 3.345) 4K (4K) 2K (2K) 2.744 (1.567 (14.990) 32K (32K) 5621 (1.FWSM—Hardware Limits See Appendix   FWSM has several hardware limits that should be considered in your network design   Limits are hard set.990 (999.3 (Multimode) ACEs AAA Rules Global Statements Static NAT Statements Policy NAT ACEs NAT Translations Connections Route Table Entries Fixup/Inspect Rules Filter Statements BRKSEC-3020 3.942 (606) 256K (256K) 999.1/3.1 (Multimode) 100.942 (606) 1K (1K) 2K (2K) 3. .537) 3747 (576) 3.627 (9.451 (992) 4K (4K) 2K (2K) 1.

801 ACEs Tree 9 : active = 14.801 ACEs Tree 12 : backup BRKSEC-3020 © 2011 Cisco and/or its affiliates. wasted reserved space Single Context Tree 0 : Active  100. Cisco Public 28 .801 ACEs Tree 1 : active = 14.567 (mirror of active tree) Tree 8 : active = 14.801 ACEs Tree 2 : active = 14.801 ACEs Tree 7 : active = 14. one backup) If you have less than 12 contexts.801 ACEs 177612 combined total ACEs Backup Tree:  100. ACL resources are divided in 13 equal partitions (12 active.801 ACEs Tree 10 : active = 14.801 ACEs Tree 5 : active = 14.567 ACEs Multi-Context Tree 0 : active = 14.801 ACEs Tree 3 : active = 14.801 ACEs Tree 11 : active = 14.801 ACEs Tree 6 : active = 14.FWSM—ACL Rule Limits   ACL rules are about the only hardware limit users encounter   In multimode.801 ACEs Tree 4 : active = 14. All rights reserved.

All rights reserved. it must classify it to determine where to send the packet (which context)   Packets are classified based on the following Unique ingress interface/VLAN Packet’s destination IP matches a global IP   FWSM has a single MAC address for all interfaces   ASA has single MAC for shared interfaces (physical interfaces have unique MACs) ASA Ver 7.2 introduces mac-address auto option to change this BRKSEC-3020 © 2011 Cisco and/or its affiliates.Classifier in Multimode   When the firewall receives a packet. Cisco Public 29 .

2 Inside VLAN 5 10.2 Inside VLAN 6 10.1 10.2.3 Shared Interface 30 .89 10.2 static (inside.14.168.4 VLAN 4 Inbound Packet CTX2 .2 CTX3 .1.3.3. outside) 10.Classifier in Multimode Example   Inbound traffic is classified to context CTX3.1.1.5.3.14.14. All rights reserved. Cisco Public DST IP VLAN 3—10.3.3.x CTX1 .1.2 BRKSEC-3020 © 2011 Cisco and/or its affiliates.1. based on the global IP in the NAT translation FWSM Inside 10.89 Outside MSFC SRC IP 192.

Multi-Context - Common Issues on FWSM
  Overlapping statics (globals) across contexts   Missing statics (globals), and unable to classify packets – check Admin context log
%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 72.163.4.161/80

  Forgetting to ‘monitor-interface’ for Failover   Forgetting to assign unique IP for each Transparent mode context   Transparent mode, multi-BVI, one routing table

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

31

Agenda
  Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

32

Failover Basics
  Active/Standby vs. Primary/ Secondary   Serial vs. LAN failover   Stateful failover (optional)   A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall   Both firewalls swap MAC and IP addresses when a failover occurs   Level 1 syslogs will give reason of failover
BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public

Internet

Stateful LAN/Serial Secondary (Standby) Primary (Active)

Corp

33

Verifying Failover Configuration
ASA# show failover Failover On Failover unit Primary Failover LAN Interface: failover Redundant5 (up) Unit Poll frequency 200 milliseconds, holdtime 1 seconds Interface Poll frequency 500 milliseconds, holdtime 5 seconds Interface Policy 1 Monitored Interfaces 2 of 250 maximum Version: Ours 8.2(2), Mate 8.2(1) Interface Last Failover at: 10:37:11 UTC May 14 2010 Monitoring This host: Primary - Active Active time: 1366024 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys) Interface outside (10.8.20.241): Normal Interface inside (10.89.8.29): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys) Interface outside (10.8.20.242): Normal Interface inside (10.89.8.30): Normal Stateful Failover Logical Update Statistics Link : stateful Redundant6 (up) Stateful Obj xmit xerr rcv rerr General 424525 0 424688 0 sys cmd 423182 0 423182 0
BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public

34

All rights reserved.What Triggers a Failover?   Power loss/reload (this includes crashes) on the Active firewall   SSM interface/module failure   The Standby becoming healthier than the Active firewall In LAN based Failover. what happens if the LAN interface communication is severed? BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 35 .

All rights reserved. only if all tests fail will the interface be marked failed BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 36 .What Triggers a Failover? (Con’t)   Two consecutive hello messages missed on any monitored interface forces the interface into testing mode   Both units first verify the link status on the interface   Next. both units execute the following tests Network activity test ARP test Broadcast ping test   The first test passed causes the interface on that unit to be marked healthy.

All rights reserved.How Well do you Understand Failover? What Happens When…   You disable failover? (By issuing no failover)   You RMA/Replace the Primary unit?   You don’t define Standby IP addresses? BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 37 .

mate is healthier Syslogs from Secondary (Standby) Firewall ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 38 .What to Do After a Failover   Always check the syslogs to determine root cause   Example: switch port failed on inside interface of active firewall Syslogs from Primary (Active) Firewall See Appendix ASA-4-411002: Line protocol on Interface inside. All rights reserved. changed state to down ASA-1-105007: (Primary) Link status ‘Down’ on interface 1 ASA-1-104002: (Primary) Switching to STNDBY—interface check.

conns replicated Stb Upgrade Complete BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 39 . and to “normalize” – approx 2 min Verify config. conns replicated Act Issue “failover active” Copy new image over and reboot Wait for failover to finish syncing.Failover – Zero Downtime Upgrades State Start State Secondary Issue “failover active” Act Stb Primary Stb Act Copy new image over and reboot Wait for failover to finish syncing. All rights reserved. and to “normalize” – approx 2 min Verify config.

Agenda   Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 40 .

Cisco Public 41 . All rights reserved.Troubleshooting Tools   Syslogs   Debug commands   Show commands   Packet capture   Packet tracer   TCP Ping BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved. Buffered BRKSEC-3020 © 2011 Cisco and/or its affiliates.Uses of Syslogs   Primary mechanism to record traffic to and through the firewall   The best troubleshooting tool available Archival Purposes Debugging Purposes Console Syslog Server Internet SSH Client Trap SNMP Server Syslog . Cisco Public 42 .

7.2 0 77 (77) 35 (112) 334 (446) 267 (713) 206 (919) 302 (1221) 258 (1479) Ver. 7. 8.3 0 95 (95) 57 (152) 408 (560) 324 (884) 246 (1130) 377 (1507) 269 (1776) Ver.4 0 109 (109) 63 (172) 448 (620) 357 (997) 265 (1242) 395 (1637) 276 (1913) 43 0 1 2 3 4 5 6 7 50 (137) 56 (143) More messages 363 (500) 281 (781) 218 (999) 337 (1336) 267 (1603) 384 (527) 315 (842) 237 (1079) 368 (1447) 269 (1716) © 2011 Cisco and/or its affiliates. 8.2 0 87 (87) Ver. Cisco Public .0 0 62 (62) 29 (91) 274 (365) 179 (544) 161 (705) 234 (939) 217 (1156) Ver. All rights reserved. 8.ASA Syslog Level vs.1 0 87 (87) Ver. 8. 8.0 0 78 (78) 49 (127) 361 (488) 280 (768) 216 (984) 335 (1319) 266 (1585) Ver. Number of Messages Log Description Level Emergenci es Alerts Critical Errors Warnings Notification s Information al Debugging BRKSEC-3020 Number of Messages (SUM) Ver.

but by default it is at level seven (debug) %ASA-7-111009: User ‘johndoe’ executed cmd: show run The problem is we don’t want to log all 1775 other syslogs that are generated at debug level © 2011 Cisco and/or its affiliates.What Are Modifiable Syslog Levels? [no] logging message <syslog_id> level <level>   Modifiable syslog levels Allows one to move any syslog message to any level Levels 0—Emergency 1—Alert 2—Critical 3—Errors 4—Warnings 5—Notifications 6—Informational 7—Debugging   Problem You want to record what exec commands are being executed on the firewall. Cisco Public BRKSEC-3020 44 . syslog ID 111009 records this information. All rights reserved.

Cisco Public 45 . how could you   Now our syslog looks as follows do it? %ASA-3-111009: User ‘johndoe’ executed cmd: show run   To restore the default syslog level ASA(config)# no logging message 111009 level 3 Tip: Use show logging message all to see the default level for any message BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.How to Create Modifiable Syslog Levels Solution [no] logging message <syslog_id> level <level>  Lower syslog message 111009 to level 3 (error) ASA(config)# logging message 111009 level 3 If you were only interested in logging one syslog message.

Logging – Common Issues   logging flash-bufferwrap – should only be used when logging to buffer at Level 1   logging history – should only be used when you really have an SNMP server that you want to receive all syslogs   logging console – should only be enabled while actively troubleshooting on the Console   logging standby – should only be used if you want to receive double the syslogs   logging permit-hostdown – should always be used with TCP syslogging BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 46 .

use with caution 3.  Debugs can negatively impact the CPU of the box.Debug Commands 1. of the specified type. is passing through the firewall before enabling the respective debug * Crypto Conditional Debugging Was Added to Cisco ASA/PIX 8. All rights reserved.  Know how much traffic.  Debugs are not conditional* 4. and also the performance of it.0 BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 47 .  Debugs should not be the first choice to troubleshoot a problem 2.

1.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.22 to inside:10.219.201. Cisco Public 48 .25 to 209.165.201.1.Debug ICMP Trace Internet http://www.1.165.2 to 198. or ICMP inspection must be enabled Example debug icmp trace output ICMP echo-request from inside:10.219.com   Valuable tool used to troubleshoot connectivity issues   Provides interface and translation information to quickly determine flow   Echo-replies must be explicitly permitted through ACL.cisco.1.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.201.133.165.2 BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.2 to outside:209.22 ICMP echo-reply from outside:198.133.1. All rights reserved.

Show Output Filters See Appendix show <cmd> | begin|include|exclude|grep [-v] <regular_exp>   Use output filters to filter the output of show command to only the information you want to see   To use them. Cisco Public 49 . All rights reserved. use the pipe character “|” followed by begin include exclude grep grep –v Start displaying the output beginning at the first match of the RegEx. at the end of show <Command>. and continue to display the remaining output Display any line that matches the RegEx Display any line that does not match the RegEx Same as include Same as exclude BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Cisco Public 50 .0(1)/FWSM 1.Show CPU Usage   Under normal conditions the CPU should stay below 50% (baseline as per network).1(1) BRKSEC-3020 © 2011 Cisco and/or its affiliates. if the CPU reaches 100% the firewall will start dropping packets   FWSM CPU is used for limited traffic processing. 1 minute: 4%. during ACL compilation CPU is expected to be near 100% until ACL is compiled   The show cpu usage command displays the CPU over time as a running average ASA# show cpu usage CPU utilization for 5 seconds = 5%. 5 minutes: 4% *First Introduced in Cisco PIX OS Version 6. All rights reserved.

Cisco Public 51 . and 5min ASA# # show processes cpu-usage PC Thread 5Sec 081aa124 d51ab230 0.2% 08070416 d51aa660 0.0% 0924fe95 d51a7528 0.0% 0.0% 0. All rights reserved.0% 088a6e14 d51a7138 0.0% 0.0% 08bcf736 d51a53b0 0. 1min.0% 0.0% 0.0% 0.0% 08685627 d51a3a18 0.0% 0. 1Min 2.0% 0.0% 0. 8.1(1.0% 0.0% 0..0% 0.0(4. 8..0% 08a6d7f6 d51a6f40 0.0% 08c2a91d d51a7f00 0.100).0% 0.0% 0. Currently not Available in FWSM BRKSEC-3020 © 2011 Cisco and/or its affiliates.0% 0.2(4.0% 08b99aec d5195d98 3.5% 5Min 2.0% 0.5).2(1).0% 0.0% 0.1% Process Dispatch Unit aaa dbgtrace netfs_thread_init Chunk Manager IP Address Assign QoS Support Module Logger netfs_mount_handler arp_timer ssh/timer ssh *First Introduced in Cisco ASA Version 7.0% 0851ca68 d51a3820 0. 8.0% 081a954c d51a96a0 0.11).0% 0.0% 0.0% 08b9ffab d5198ae0 0.0% 0.Show Processes cpu-usage   The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec.9% .0% 0.

NUMHOG: 18.0(1). LASTHOG: 10 LASTHOG At: 14:18:47 EDT May 29 2009 PC: 8b9ac8c (suspend) Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971 8ba02b4 8062413 CPU hog threshold (msec): 10. Currently not Available in FWSM BRKSEC-3020 © 2011 Cisco and/or its affiliates. and lead to a process running on the CPU longer than the minimum platform threshold ASA# show processes cpu-hog Process: ssh_init. Process = ssh_init.Show Processes cpu-hog   The show processes cpu-hog command displays a list of processes. PC = 8b9ac8c. Cisco Public 0x08BA77ED 52 . Traceback = 0x08B9AC8C 0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413 *First introduced in Cisco ASA Version 7.240 Last cleared: None   A corresponding syslog message is also generated Note: The Traceback syslog below does not signify a crash May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec. and the function stack (Traceback) which executed. All rights reserved. MAXHOG: 15.

Show Traffic   The show traffic command displays the traffic received and transmitted out each interface of the firewall ASA# show traffic outside: received (in 124. Cisco Public 53 .650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124. All rights reserved.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec ! BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved.show np blocks (FWSM Only)   The show np blocks command is used to see if the FWSM is over subscribed Data and Control packets dropped Data packets dropped Warning FWSM# show np blocks MAX FREE NP1 (ingress) 32768 32768 (egress) 521206 521206 NP2 (ingress) 32768 32768 (egress) 521206 521206 NP3 (ingress) 32768 32768 (egress) 521206 521206 THRESH_0 0 0 0 0 13 0 THRESH_1 0 0 0 0 460417 0 THRESH_2 550 0 92 0 4427509 0 BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 54 .

allocated 0 TCP PAT pool outside. 5772 most used TCP PAT from inside:192. allocated 2321 ASA-5585# BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 55 .2.3 ASA-5585# show nat pool TCP PAT pool outside.168.1. address 10.103.1. range 1-511.2.1.168.2.1. range 1024-65535.2/43756 flags ri idle 0:00:00 timeout 0:00:30 TCP PAT from inside:192.2/54464 flags ri idle 0:00:00 timeout 0:00:30 Added in version 8.2.Show Xlate and Show Xlate Debug   The show xlate command displays information about the translations through the firewall   You can limit the output to just the local or global IP ASA-5585# show xlate 5014 in use. address 10.1.220/57762 to outside:10. range 512-1023. address 10. allocated 1 TCP PAT pool outside.2.220/57761 to outside:10.2.2.103. All rights reserved.2.

0.0/24 Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static webserver-obj 14.1.36. Translated: 192.16.168.96/16 ASA-5585/admin# BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 56 .0.83/32 2 (inside) to (outside) source dynamic science-obj interface translate_hits = 37723. untranslate_hits = 0 Source . Translated: 14. untranslate_hits = 3232 Source . All rights reserved.168.0/16.36.36.32/32.Origin: 172.103.103.168.0/16.Origin: 192. Translated: 172.0/24.Show Nat Detail   The show nat command displays information about the nat table of the firewall   The detail keyword will display object definitions ASA-5585# show nat detail Manual NAT Policies (Section 1) 1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj translate_hits = 0.103.Origin: 192.Origin: 192.0.0/16 Destination .83 translate_hits = 0.168.16. Translated: 14.1. untranslate_hits = 0 Source .22.

outside back connection.incomplete. E .inside FIN.0(4) Flags: A .WAAS.133.inside back connection. a . 8.UDP SUNRPC.H.225. 64511 most used 8.1:123 dmz 10.MGCP.inbound data. idle 8s.outside acknowledged FIN.SQL*Net data. flags UIO.323. S .25/80 dmz:10. bytes 1431 BRKSEC-3020 © 2011 Cisco and/or its affiliates.inspected by service module TCP outside:198. C .Show Conn and Show Conn Detail Real Interface Names Added in 7.9. d .GTP t3-response k .133.SIP.H.1.group.1.2(4).25:80 dmz 10. Cisco Public Idle Time. m .awaiting outside SYN.GTP. J .0(4) ASA# show conn 2 in use. H . j .3:4101. M .18.124. R .1. timeout 2m. flags UIO UDP outside 172. I .18.9/123.up. U .9.Skinny media.initial SYN from outside. uptime 16s.awaiting outside ACK to SYN. G .CTIQBE media. X . idle 15s. s . uptime 10s. K . bytes 127 UDP outside:172.awaiting inside SYN. Bytes Transferred Connection Flags detail Adds Uptime 57 . q .2(4). h .GUP O . F .awaiting inside ACK to SYN. timeout 1h.9.outside FIN.SIP media.0. f . n .SMTP data.9:123 idle 0:00:13 flags – ASA# show conn detail and Timeout in 7.GTP data.9.219. Bytes 127. T . P .outbound data.inside acknowledged FIN.SIP transient.219. All rights reserved. g . t .3/4101. 64511 most used TCP outside 198.dump. i .1. 2 in use.1/123 dmz:10. B . idle 0:00:06. R .DNS. flags -. r . W . D .124.

the SYN is permitted by the access-list. and the connection is now considered up (U flag)   The outside device sends the first data packet. Cisco Public 58 . a translation (xlate) is built up. and now show A   The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake.Example—Connection Build Up   Firewall receives an initial SYN packet from the inside. and the connection is also created with the flags saA   The outside device responds to the SYN packet with a SYN+ACK. All rights reserved. the connection flags are updated to reflect this. the inside device has sent a data packet and the connection is updated to include the O flag 1 5 3 SYN+ACK SYN Data ACK Connection Flags 42 UI UIO saA U A Client Inside Outside Server BRKSEC-3020 © 2011 Cisco and/or its affiliates. the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection   Finally.

there are no more connection flags. the connection flags are updated to reflect this. All rights reserved. 59 . and now show UfFR   The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection. as the FIN passes through the firewall. thus.Example—Connection Teardown   Firewall receives a FIN packet from the inside. it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface   The outside device immediately responds to the FIN packet with a FIN+ACK. because the connection no longer exists 3 1 FIN+ACK ACK FIN Connection Flags 2 UfFRr UfUfFR Inside Client Outside Server Cisco Public BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved. Cisco Public 60 .Connection Flags—Quick Reference Outbound Connection Inbound Connection For your reference BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved. Cisco Public 61 . it will always have a teardown reason   The TCP teardown syslog is logged at level six   If you are having problems with connections abnormally closing. What does teardown reason and check the the Reset-O Termination reason mean in the Teardown TCP connection syslog? ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)] BRKSEC-3020 © 2011 Cisco and/or its affiliates.TCP Connection Termination Reasons   If a TCP connection is built through the firewall. temporally increase your logging level (or move the syslog down).

Cisco Public 62 .TCP Connection Termination Reasons— your For Quick Reference reference Reason Conn-Timeout Deny Terminate Failover Primary Closed FIN Timeout Flow Closed by Inspection Flow Terminated by IPS Flow Reset by IPS Flow Terminated by TCP Intercept Invalid SYN Idle Timeout IPS Fail-Close SYN Control Description Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout Flow Was Terminated by Inspection Feature Flow Was Terminated by IPS Flow Was Reset by IPS Flow Was Terminated by TCP Intercept SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer than the Timeout Value Flow Was Terminated Due to IPS Card Down Back Channel Initiation from Wrong Side BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

TCP Connection Termination Reasons— your For Quick Reference (Cont.) reference Reason SYN Timeout TCP Bad Retransmission TCP Fins TCP Invalid SYN TCP Reset-I TCP Reset-O TCP Segment Partial Overlap TCP Unexpected Window Size Variation Tunnel Has Been Torn Down Unauth Deny Unknown Xlate Clear Description Force Termination After Two Minutes Awaiting Three-Way Handshake Completion Connection Terminated Because of Bad TCP Retransmission Normal Close Down Sequence Invalid TCP SYN Packet TCP Reset Was Sent From the Inside Host TCP Reset Was Sent From the Outside Host Detected a Partially Overlapping Segment Connection Terminated Due to a Variation in the TCP Window Size Flow Terminated Because Tunnel Is Down Connection Denied by URL Filtering Server Catch-All Error User Executed the ‘Clear Xlate’ Command BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 63 . All rights reserved.

220/34077. Cisco Public 64 . flags UO.show local-host   A local-host entry is created for any IP tracked through the firewall   It groups the xlates. bytes 0 (output truncated) BRKSEC-3020 © 2011 Cisco and/or its affiliates. flags UO. idle 0s.220/34078.76/80 inside:192. bytes 0 TCP outside:172.168.168.168.103. connections. 0 maximum active.103. timeout 30s.103. idle 0s. uptime 0s. and AAA information   Very useful for seeing the connections terminating on servers ASA# show local-host ASA# show local-host detail connection tcp 50 Interface dmz: 0 active.18.18.76/80 inside:192. timeout 30s. 0 denied Interface inside: 1 active.124.220>. TCP flow count/limit = 798/unlimited TCP embryonic count to host = 0 Add ‘show local-host TCP intercept watermark = unlimited UDP flow count/limit = 0/unlimited detail connection Conn: arguments’ TCP outside:172. 0 denied local host: <192. 1 maximum active. uptime 0s.124. All rights reserved.

drop 0. drop 0. Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0. drop 0. lock fail 0. Cisco Public 65 . packet 0. packet 0. lock fail 0. packet 0. drop 0. packet 0.show service-policy   The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them ASA-5585/admin# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: ftp. bytes dropped 0 Inspect: h323 ras _default_h323_map. packet 0. reset-drop 0 ASA-5585/admin# . reset-drop 0 Inspect: h323 h225 _default_h323_map. reset-drop 0 Inspect: rtsp.. packet 0. All rights reserved. drop 0. lock fail 0. reset-drop 0 tcp-proxy: bytes in buffer 0. lock fail 0. packet 1215927. lock fail 0. reset-drop 0 Inspect: http.. drop 0. drop 0. lock fail 0. reset-drop 0 Inspect: icmp. drop 0. reset-drop 0 tcp-proxy: bytes in buffer 0. aggregate transmit 349 BRKSEC-3020 © 2011 Cisco and/or its affiliates. reset-drop 0 Inspect: rsh. packet 57. bytes dropped 0 Inspect: sqlnet.

9.8.9.show service-policy flow   Use to determine what policies a given flow will match in the Modular Policy Framework (MPF) ASA# show service-policy flow tcp host 10. Cisco Public 66 .6 host 10.8.3 eq sqlnet Action: Input flow: set connection timeout dcd BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.9.1.9.3 eq 1521 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside Class-map: oracle-dcd Match: access-list oracle-traffic Access rule: permit tcp host 10. All rights reserved.6 host 10.

flow drops are per flow   Some counters have corresponding syslogs ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) Invalid tcp length (invalid-tcp-hdr-length) Invalid udp length (invalid-udp-length) No valid adjacency (no-adjacency) No route to host (no-route) Reverse-path verify failed (rpf-violated) Flow is denied by access rule (acl-drop) First TCP packet not SYN (tcp-not-syn) Bad TCP flags (bad-tcp-flags) TCP option list invalid (tcp-bad-option-list) TCP MSS was too large (tcp-mss-exceeded) Bad TCP Checksum (bad-tcp-cksum) *Drop Counters Are Documented in the CMD Ref.show asp drop   Packets dropped in the Accelerated Security Path (ASP) will increment a counter   FWSM – applies only to traffic sent to the control-point   Frame drop counters are per packet. Cisco Public 10897 9382 10 5594 1009 15 25247101 36888 67148 731 10942 893 67 . All rights reserved. Under show asp drop BRKSEC-3020 © 2011 Cisco and/or its affiliates.

0.Packet Capture capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any} [eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]   Capture command first introduced in Cisco 7. All rights reserved. or copy it off in . Cisco Public 68 .1. FWSM need to use 3.2(3) and 8.0(3) added a real-time option   ASDM 6. or match line   Key steps Use the ‘match’ keyword to specify what traffic to capture (implicitly bidirectional) Define the capture and bind it to an access-list and interface View the capture on the firewall.5 or later   ASA 7.pcap format BRKSEC-3020 © 2011 Cisco and/or its affiliates.0 adds a capture wizard   Capture sniffs packets on an interface that match an ACL.

All rights reserved. Outside Capture Out Capture Outside Cisco Public 69 .) See Appendix   Traffic can be captured both before and after it passes through the firewall. one capture on the outside interface   Capture buffer saved in RAM (default size 512 KB)   Default is to stop capturing when buffer is full   Default packet length is 1518 bytes   Copy captures off via TFTP or HTTPS Inside Capture Capture In Inside BRKSEC-3020 © 2011 Cisco and/or its affiliates.Packet Capture (Cont. one capture on the inside interface.

Cisco Public 70 .Where Packets Are Captured in Packet Flow Ingress Packets Captured Egress Packets Captured   Packets are captured at the first and last points they can be in the flow   Ingress packets are captured before any packet processing has been done on them   Egress packets are captured after all processing (including L2 source MAC rewrite) BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public 71 .Capturing Packets Dropped by the ASP   Capture all packets dropped by the ASP ASA# capture drops type asp-drop all   Capture on a specific drop reason ASA# capture drops type asp-drop tcp-not-syn   Applies to both ASA and FWSM ASA# capture drop type asp-drop ? acl-drop all bad-crypto bad-ipsec-natt bad-ipsec-prot bad-ipsec-udp bad-tcp-cksum bad-tcp-flags Flow is denied by configured rule All packet drop reasons Bad crypto return in packet Bad IPSEC NATT packet IPSEC not AH or ESP Bad IPSEC UDP packet Bad TCP checksum Bad TCP flags BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public 72 . it is punted to the control-plane   The control-plane reads and displays the actions taken on the packet. All rights reserved.2 ASA Only   A packet tagged with the trace option is injected into the interface. or is dropped.Packet Tracer: Overview   Introduced in ASA ver 7. along with the associated lines in the configuration BRKSEC-3020 © 2011 Cisco and/or its affiliates. and processed in the data-plane   Each action taken on the packet is recorded in the packet itself   When the packet reaches the egress interface.

Packet Tracer: ASDM (Located off Tools Menu)
Define Packet

Action Matching Config Link Back to Edit Rule

Final Result

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

73

Packet Tracer: Example Output
ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80 Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: Found no matching flow, creating a new flow Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group in in interface inside access-list in extended permit tcp any any eq www Additional Information: Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map match-all inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect http service-policy global_policy global Additional Information:
BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public

74

Packet Tracer: Example Output (Cont.)
... Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information: Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255 ... Phase: 15 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.165.201.1 using egress ifc outside adjacency Active next-hop mac address 000a.f331.83c0 hits 0 >>>>Packet successfully forwarded to fast path<<<<

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

75

Packet Tracer: Tracing Captured Packet
  Create a capture using the trace option   Find the packet in the capture you want traced
ASA# 68 1: 2: 3: 4: 5: show capture inside packets captured 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: select that packet to be traced ...

Important!
.

ASA# capture inside access-list web interface inside trace

  Then

S S . P .

ack ack ack Ack

ASA# show capture inside trace packet-number 4

.

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

76

7 BRKSEC-3020 © 2011 Cisco and/or its affiliates.200.4. All rights reserved.TCP Ping   New troubleshooting tool added in ASA ver 8.225) 77 . Cisco Public www server (209.1   Why is it needed??? Consider the following… 10.1.1.165.

165. All rights reserved.TCP Ping   Previously – limited reachability tools: Ping and Traceroute   Access to client machine? What about Attempts to validate the path …but with ICMP PAT? NAT and/or ICMP Echo Request ICMP Echo Reply ICMP Echo Request ICMP Echo Reply 10. Cisco Public www server (209.200.1.1.225) 78 .7 BRKSEC-3020 © 2011 Cisco and/or its affiliates.

ACLs.1.51.165.7 injected on Inside interface Internal hosts are PATed to 198.2 TCP SYN sent to server inside 10.2 on Egress TCP SYN+ACK sent from server BRKSEC-3020 © 2011 Cisco and/or its affiliates.200. etc) Packet PATed to 198.51.100. Cisco Public 79 .225) ASA Datapath Validated (NAT.1.TCP Ping   Sources TCP SYN packet with Client’s IP and injects it into Client’s interface of the ASA Packet with SRC of 10.7 outside www server (209.1. All rights reserved.1.100.

1.TCP Ping – The Big Picture   Validates 2 of the 3 legs of the connection from client to server TCP path from client side of ASA to Server through the cloud -Validated- inside 10.200.7 outside www server (209.225) 1st Leg 2nd Leg 3rd Leg BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 80 .1. All rights reserved.165.

outside www server (209.TCP Ping .1.1.225 Target IP port: 80 Specify source? [n]: y Source IP address: 10.200.1.225) Cisco Public 81 . Sending 5 TCP SYN requests to 209. round-trip min/avg/max = 1/1/1 ms inside 10.1.200.165.165.200.7 BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. timeout is 5 seconds: !!!!! Success rate is 100 percent (5/5).7 starting port 3465.225 port 80 from 10.Example Specify Client’s source Interface Specify Client’s real IP Address asa# ping tcp Interface: inside Target IP address: 209.165.1.7 Source IP port: [0] Repeat count: [5] Timeout in seconds: [2] Type escape sequence to abort.1.

Agenda   Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 82 . All rights reserved.

Case Study Leveraging Smart Call Home .

  This is easily accomplished with SCH.com ASA Only   This will send a plain-text e-mail with the output of the command to the e-mail address specified. Example: Subject: CLI ‘show run’ output BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 84 . with the command in the subject line.Case Study: Smart Call Home Email CMD Output to You   Objective – Send the output of a command directly to your e-mail. All rights reserved. Use the command: call-home send <“cmd”> email <email_addr> Example: call-home send “show run” email userid@cisco.

  This is easily accomplished with SCH.Case Study: Smart Call Home Collecting Memory Diagnostics over Time   Objective – Memory appears to be depleting over time on your ASA. All rights reserved. for further investigation. Use SCH to collect the detailed memory output hourly. Cisco Public 85 . Setting a ”snapshot” alert-group to e-mail commands at a specified interval   Snapshot will contain the following command: show conn count show memory detail BRKSEC-3020 © 2011 Cisco and/or its affiliates.

com mail-server smtp-server.com sender from user@cisco.com priority 1 profile SENDCMD active destination address email user@cisco.com sender reply-to user@cisco. All rights reserved. Cisco Public 86 .cisco.com destination preferred-msg-format long-text destination transport-method email subscribe-to-alert-group snapshot periodic hourly BRKSEC-3020 © 2011 Cisco and/or its affiliates.Case Study: Smart Call Home Example Config service call-home call-home alert-group-config snapshot add-command “show conn count” add-command "show memory detail“ contact-email-addr user@cisco.

Case Study Intermittent Access to Web Server .

1. All rights reserved.50 Internet Web Server 10.Case Study: Intermittent Access to Web Server Problem   Most external clients are not able to load company’s web page NATed to 10.1.50 ASA-5510 Clients BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.1.1.168.50 HTTP Requests to 192. Cisco Public 88 .

Case Study: Intermittent Access to Web Server Traffic Spike BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 .

00% BRKSEC-3020 © 2011 Cisco and/or its affiliates.Case Study: Intermittent Access to Web Server   show perfmon indicates high number of embryonic connections ASA-5510# show perfmon PERFMON STATS: Xlates Connections TCP Conns UDP Conns URL Access URL Server Req TCP Fixup TCP Intercept Established Conns TCP Intercept Attempts TCP Embryonic Conns Timeout HTTP Fixup FTP Fixup AAA Authen AAA Author AAA Account VALID CONNS RATE in TCP INTERCEPT: Current 0/s 2059/s 2059/s 0/s 0/s 0/s 0/s 0/s 0/s 1092/s 0/s 0/s 0/s 0/s 0/s Current N/A Average 0/s 299/s 299/s 0/s 0/s 0/s 0/s 0/s 0/s 4/s 0/s 0/s 0/s 0/s 0/s Average 95. All rights reserved. Cisco Public 90 .

1.1. bytes 0.1.50. All rights reserved. idle 0:00:23. flags aB TCP outside 124.105.92. flags aB TCP outside 75.50:80.24. bytes 0.101.50:80.1.1. bytes 0.1. idle 0:00:02.7. idle 0:00:15.1.1. flags aB TCP outside 98.77:28138 inside 10.2.50:80.1.122:24542 inside 10.63:51503 inside 10.36. 54764 most used TCP outside 17. idle 0:00:12.133. bytes 0.82. bytes 0.239:18315 inside 10.202:20773 inside 10.1.72.50:80.230:44115 inside 10. bytes 0. bytes 0. flags aB TCP outside 82.41:22257 inside 10.75.50:80. bytes 0.50:80. idle 0:00:04.1.205:18073 inside 10.131:26224 inside 10.147.1.246.192:23112 inside 10.1. idle 0:00:02. flags aB TCP outside 95. idle 0:00:07.1.137. flags aB TCP outside 130.50:80. flags aB .128.50:80.50:80.108. flags aB TCP outside 26.1.106.1.115. flags aB TCP outside 37.1.1. idle 0:00:03. flags aB TCP outside 76.110.50:80.1.60:47733 inside 10. idle 0:00:12. idle 0:00:02.216:3496 inside 10.226. idle 0:00:25.170.236.50:80.1. bytes 0.1.1.223.195. BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.181. flags aB TCP outside 121.2.210.1.1. flags aB TCP outside 13.187.1.50:80. flags aB TCP outside 134.39:20591 inside 10.1.118:26093 inside 10.142. . Cisco Public 91 .1.110. idle 0:00:06. bytes 0.213. flags aB TCP outside 39. bytes 0.Case Study: Intermittent Access to Web Server   Issue show conn to see ‘who’ is creating the connections Random Sources Embryonic Conns ASA-5510# show conn 54764 in use.50:80. bytes 0. flags aB TCP outside 24.1.203.50:80.1.1.50:80.27.1. flags aB TCP outside 111. bytes 0. idle 0:00:13. flags aB TCP outside 99. bytes 0.202:32729 inside 10. idle 0:00:27.1.1.76. idle 0:00:27.1. flags aB TCP outside 66.1. idle 0:00:06. idle 0:00:02. bytes 0.50:80. bytes 0. bytes 0. bytes 0.172:27391 inside 10.133.27.1.181:37784 inside 10. bytes 0.229:21247 inside 10.204:56481 inside 10.1.185.1. idle 0:00:13. flags aB TCP outside 70. . flags aB TCP outside 30.50:80. idle 0:00:13.50:80.233.50:80. idle 0:00:29.50:80.1.109:23598 inside 10. bytes 0.34.

Cisco Public 92 . All rights reserved.Case Study: Intermittent Access to Web Server Traffic Permitted Connection Count Jumps SYN Flood Detected BRKSEC-3020 © 2011 Cisco and/or its affiliates.

50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 93 .Case Study: Intermittent Access to Web Server   Apply TCP Intercept to stop the SYN flood attack access-list 140 extended permit tcp any host 192. All rights reserved.168.1.

Case Study: Intermittent Access to Web Server Why did the Connection countFew Clients Represent Intercept drop after TCP TCP Intercept 50+ was applied? % of Traffic Applied BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 94 . All rights reserved.

All rights reserved.168.1.Case Study: Intermittent Access to Web Server   Apply per-client-max option to limit the number of connections any single client can establish access-list 140 extended permit tcp any host 192.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 95 .

All rights reserved. Cisco Public 96 .Case Study: Intermittent Access to Web Server per-client-max TCP Intercept BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Case Study: Intermittent Access to Web Server Attacks Being Mitigated Attacks Still Occurring BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 .

Agenda   Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 98 . All rights reserved.

com   TAC Security Show Podcast   Online learning modules (VoD Training)   Security RSS Feeds BRKSEC-3020 © 2011 Cisco and/or its affiliates.cisco. All rights reserved.Supportforums.Online Resources   Support Communities . Cisco Public 99 .

cisco. Cisco Public 100 . FWSM and PIX   Hundreds of Sample Configs   Troubleshooting Docs   FAQs http://supportforums. All rights reserved.cisco.Supportforums.com   Public wiki – anyone can author articles   Combines supportwiki and Netpro forums   Sections for: ASA.com/ BRKSEC-3020 © 2011 Cisco and/or its affiliates.

TAC Security Podcast   Great way to obtain valuable troubleshooting insights. Cisco Public 101 . which focus on providing in-depth information on a given feature. All rights reserved.cisco.   New episodes posted Monthly http://www.com/go/tacsecuritypodcast/ BRKSEC-3020 © 2011 Cisco and/or its affiliates.   Conversational shows.

Cisco Public 102 .TAC Security Podcast Episodes Search iTunes for TAC Security Podcast BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

cisco.cisco. All rights reserved.com for ASA Online Learning Modules   Direct link http://www.Online Learning Modules – VoD Training   Great way to learn about new features in the ASA   From www.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list. Cisco Public 103 .com select: Products and Services  Security  Network Security (expand)  Cisco ASA 5500 Series   Training resources   Online learning modules   Search cisco.html BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved.cisco. FWSM. ASDM https://supportforums.Security Hot Issues – RSS Feeds   Subscribe with an RSS reader   Receive weekly updates on the Hot Issues customers are facing   Separate feeds for: ASA.com/docs/DOC-5727 BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 104 .

Agenda   Packet Flow   Understanding the Architecture   Failover   Troubleshooting   Case Studies   Online Resources   Best Practices BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 105 . All rights reserved.

Cisco Public 106 . use TACACS+ or RADIUS with LOCAL as the fallback BRKSEC-3020 © 2011 Cisco and/or its affiliates.Cisco ASA/FWSM Best Practices   Enable ip verify reverse-path on all interfaces   Set embryonic and maximum connection counts on static and nat statements.1+ use per-client-max   Configure logging to syslog server   Move messages you want to see to lower levels. for 7. instead of raising logging levels and capturing messages you don’t want to see   Disable telnet access! Use SSH for management access   Enable authentication for management access (console/ SSH/telnet/enable).2. All rights reserved.

connection counts.   Keep config archives (and show tech ouputs) (smart call home)   Run the latest maintenance release in your train   Upgrade major feature trains only when you need new features.Cisco ASA/FWSM Best Practices   Restrict DMZ access inbound to your internal networks   Baseline CPU load. or after train has matured BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 107 . xlate counts. All rights reserved. and traffic (per interface)   Monitor stats using MRTG or other snmp graphing tools.

2.5 8.5 7.4 8.0.2 8.3.8 7.2 7.0.3 8.2 7.2.0 8.1.2 8.2.3 8.2 7.2.0.2 8.1 7.2 8.5.0 7.1 7.4 8.1 ASA-SM only BRKSEC-3020 © 2011 Cisco and/or its affiliates.3 7. Cisco Public 108 .0.0.1 8.1 8.5 8.4.4 8.1.7 7.2.1.2.4 7.1 7.0.0.1 7.0.5 ASA-5580 only EOL 8.1 8.6 7.0.2.2.2 8.3.2 8.ASA Software Trains 7.5 8.4 Bug Fixes Waterfall Down EOL EOL 7.3 8.2 7.2.1.2.1 8. All rights reserved.1 8.0.4.0.

Q&A .

Visit the Cisco Store for Related Titles http://theciscostores.com

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

110

1 1

Complete Your Online Session Evaluation
  Receive 25 Cisco Preferred Access points for each session evaluation you complete.   Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.   Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.   Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and ondemand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

111

1 1

Thank you.

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

112

Appendix
  Lucky You   This appendix contains extra information which you may find useful, but I just didn’t have enough time to cover in the lecture – or which was covered in previous years.   Enjoy… :-)

BRKSEC-3020

© 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

113

3 Memory Requirements SNMP OIDs to Monitor Example: Show Output Filters Code Base History Case studies Poor Voice Quality Out-of-order packet buffering TCP MSS issue Out of memory High CPU Capture Example             FWSM Additional Architecture Slides Failover Extras Packet Capture Example Online Tools ASDM Information to include when opening a TAC case © 2011 Cisco and/or its affiliates.Appendix           ASA 8. All rights reserved. Cisco Public BRKSEC-3020 114 .

Redirecting Debugs to Syslog   Problem Log only debug output to syslog   Solution Create a logging list with only syslog ID 711001 ASA(config)# logging list Networkers message 711001 . All debug messages are currently being redirected to syslog:711001 and will not appear in any monitor session Log on the logging list ASA(config)# logging trap Networkers BRKSEC-3020 © 2011 Cisco and/or its affiliates. Enable debug output to syslogs ASA(config)# logging debug-trace INFO: 'logging debug-trace' is enabled. Cisco Public . All rights reserved. 115 .

only the Security Plus or Unlimited licenses require the memory upgrade BRKSEC-3020 © 2011 Cisco and/or its affiliates.3 512 MB 1024 MB 2048 MB 2048 MB Upgrade Kit Part Number ASA5505-MEM-512= ASA5510-MEM-1GB= ASA5520-MEM-2GB= ASA5540-MEM-2GB= * For the 5505.3   New ASAs ship with the upgraded RAM installed ASA Model 5505 * 5510 5520 5540 Original Default RAM 256 MB 256 MB 512 MB 1024 MB Required RAM for version 8.ASA 8. All rights reserved. Cisco Public 116 .3 Memory Requirements   ASA Models 5505 – 5540 Require Memory Upgrades before upgrading to ASA version 8.

3.2.4.9.1.1.1.1.1.1.1.6 (Current total) •  1.109.6.3.2.9.40.3.SNMP OIDs   CPU usage •  1.9.1.4.1.1 (1 min) •  1.3.1.5.1.1.1. Cisco Public 117 .2.6.2.1 (5 min)   Connections •  1.1.1.3.9.1.1.1 (5 sec) •  1.3.n (in/out octets) •  Use SNMPwalk to verify the interfaces! For your reference BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.9.9.9.6.2.{10|16}.4.4.2.9.7 (Max total)   Traffic •  1.1.6.109.1. All rights reserved.5.5.2.1.9.4.1.2.1.9.6.4.1.1.1.1.147.40.3.1.109.6.147.2.1.

Trailing Spaces Are Counted BRKSEC-3020 © 2011 Cisco and/or its affiliates.5   Display the config. All rights reserved.Example: Show Output Filters show <cmd> | begin|include|exclude|grep [-v] <regular_exp> Examples   Display the interface stats starting with the ‘inside’ interface show interface | begin inside   Display the access-list entries that contain address 10.1.1. Cisco Public 118 .1.1.5 show access-list | grep 10. except for the access-lists show run | exclude access-list   Display only access-list entries that have non-zero hitcounts show access-list | grep –v hitcnt=0   Display a count of the number of connections each host has show local-host | include host|count/limit Note: You must Include a Space on Either Side of the Pipe for the Command to Be Accepted. Also.

Cisco PIX/ASA/FWSM Code Base History PIX 6.2(17) 3.0(2) 8.1(1) 6.1(10) 3. All rights reserved.2(1) 8.1(17) 119 .0(1) 7.1(3) 2.1(1) PIX/ASA 7.0(2) 4.3(1) Port Features Bug Fixes FWSM 1.3(1) 2.0(4) 4.0(1) Feature Releases 6. Cisco Public 3.0(11) Maintenance Releases SafeHarbor 3.1(1) 3.1(2) Time BRKSEC-3020 © 2011 Cisco and/or its affiliates.2(4) GD 3.2(2) SafeHarbor 3.1(1) SafeHarbor 4.2(1) 6.1(1) 2.3(2) 3.3(1) In Sync 7.1(2) 1.2(1) 4.2(1) Feature Releases 1.1(6) 3.0(1) 4.

Cisco Public 120 . All rights reserved.Case Study Poor Voice Quality Presentation_ID © 2010 Cisco and/or its affiliates.

Case Study: Poor Voice Quality Problem   Poor outbound voice quality at SOHO sites Outbound RTP Stream 100 Mbps ASA-5505 100 Mbps Cable Modem 2 Mbps WAN BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 121 .

All rights reserved.Case Study: Poor Voice Quality Solution: Traffic Shaping   What is traffic shaping. and why is it needed here?   Why won’t policing work?   Why won’t priority queuing alone work? Shape to 2 Mbps Cable Modem 100 Mbps ASA-5505 100 Mbps WAN 2 Mbps BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 122 .

use the command show service-policy shape BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. class-map voice-traffic ! match dscp af13 ef! !! policy-map qos_class_policy ! class voice-traffic ! priority! !! policy-map qos_outside_policy ! class class-default ! shape average 2000000 ! service-policy qos_class_policy! !! service-policy qos_outside_policy interface outside!   To view statistics on the operation of the shaper.Case Study: Poor Voice Quality – Configuration Example (Traffic Shaping) Solution   Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface. Cisco Public 123 .

Case Study: Poor Voice Quality Things to Keep in Mind:   Shaping can only be applied to the class class-default   Shaping only works in the outbound direction on an interface   The shaping value is in bits per second. All rights reserved. Cisco Public 124 . and must be a multiple of 8000   The shaping policy is applied to all sub-interfaces on a physical interface   Not supported on the ASA-5580 platform   Not supported in Transparent or Multi-context mode BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved. Cisco Public 125 .Case Study Out-of-Order Packet Buffering Presentation_ID © 2010 Cisco and/or its affiliates.

Case Study: Out-of-Order Packets   Inspections require ordered packets   Packets sent to the SSM (AIP and CSC) require ordered packets   Cisco ASA/PIX will buffer up to three packets by default   Buffering can be increased on ASA by using the queue-limit option under the tcp-map BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 126 .

168.9.16.2 Dropped on Network Buffer Dropped by Firewall Packet 10 Packet 11 Packet 12 Packet 13 Packet 14 Packet 15 BRKSEC-3020 © 2011 Cisco and/or its affiliates.30 Client Inside Outside Server 10.1. Cisco Public 127 . often caused by asymmetric traffic flows   If the out-of-order packet buffer isn’t large enough. All rights reserved.Case Study: Out-of-Order Packets Problem   Some networks have high numbers of out-of-order packets. traffic is dropped and packets must be retransmitted 192.

..9.255.0 255.255. TCP packet SEQ past window TCP packet buffer full . All rights reserved.Case Study: Out-of-Order Packet Buffering Example   How to detect? ASA# show asp drop Frame drop: . Cisco Public 128 .16. 46331 90943   How to fix? access-list OOB-nets permit tcp any 10...0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global BRKSEC-3020 © 2011 Cisco and/or its affiliates.

. All rights reserved. Cisco Public 129 ..Case Study: Out-of-Order Packet Buffering Example   How to verify? ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default . Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0 BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved.Case Study TCP MSS (Maximum Segment Size) Presentation_ID © 2010 Cisco and/or its affiliates. Cisco Public 130 .

Case Study: TCP MSS   MSS is the Maximum Segment Size—or the maximum amount of data that can be sent in a single packet   The MSS is set in the SYN packets   The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 131 .

168.1.Case Study: TCP MSS Problem   Some servers have broken TCP stacks and ignore the MSS advertised by the Client   The firewall will drop packets that exceed the advertised MSS 192.2 SYN MSS=1380 SYN+ACK MSS=1400 DATA=1390 BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 132 .9.16.30 Client Inside Outside Server 10. All rights reserved.

1.16.Case Study: TCP MSS Example   How to detect? ASA# show asp drop Frame drop: TCP MSS was too large 943 %ASA-4-419001: Dropping TCP packet from outside:10. reason: MSS exceeded. All rights reserved.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global BRKSEC-3020 © 2011 Cisco and/or its affiliates. MSS 1380.9.30/1025.168. Cisco Public 133 . data 1390   How to fix? access-list MSS-hosts permit tcp any host 10.9.16.2/80 to inside:192.

All rights reserved.Case Study: TCP MSS Example   How to verify? ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518 ASA# show capture mss-capture 0 packets captured 0 packets shown   How else could you verify? BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 134 .

Cisco Public 135 . All rights reserved.Case Study Out of Memory Presentation_ID © 2010 Cisco and/or its affiliates.

64 MB RAM pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ---------------------------Total memory: 67108864 bytes BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 136 .Case Study: Out of Memory Problem   Users are unable to access the Internet   No new connections are working   All old (long lived) connections continue to work Step 1: Check the Syslogs %PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error Step 2: Check the Amount of Free Memory Available Hardware: PIX-515E. All rights reserved.

1.102 PAT Global 209.165.201.Case Study: Out of Memory Step 3: What Eats Up Memory (RAM) on the Cisco PIX?   Cisco PIX image (run from RAM)   Configuration   IPSec database   Xlates (translations)   Connections What Can Eat Up 64 MB on a Cisco PIX-515E? Step 4: Let’s Check the Translations pixfirewall# show xlate 251 in use.1.1.1.165.201.25 Local 10.165.201.165.1.227(20276) Global 209.34(43543) A Small Global Pool Is Used.201.1.26(2254) Local 10.201. 258 most used PAT Global 209.1.1.201.26(2382) Local 10.165.132(52716) PAT Global 209.1.26(2378) Local 10. All rights reserved.125(12783) PAT Global 209. Overloading to a PAT Address Varied Source IPs BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.26(2255) Local 10.26(2379) Local 10.1.1. Cisco Public 137 .165.175(39197) PAT Global 209.

48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.1.101.57.194:80 in 10.1. 147456 most used TCP out 64. Cisco Public 138 .1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.1.102.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64. .205.Case Study: Out of Memory Step 5: Check the Connections pixfirewall# show conn 147456 in use.101.1.1.102.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.85:80 in 10.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.1.200.206.1.1.144.102.101.102.203:80 in 10.1.215:80 in 10.69:80 in 10.95. BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.   Q: Why is the connection count so high? .10:135 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.27:80 in 10.1.103.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.54:43703 idle 0:00:12 Bytes 0 flags saA .1.236:80 in 10.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.1.25.1.97:135 in 10.1.101.77:21 in 10.1.200:80 in 10.106:80 in 10.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.101.19.172:80 in 10.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.1.80.1. All rights reserved.103.21.1.31.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.1.102.1.147.1.35:80 in 10.22.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.101.86.1.

000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25. All rights reserved.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec Traffic Flow   Vast majority of traffic is coming in the inside interface and going out the outside interface BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public Inside Outside 139 .000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.Case Study: Out of Memory Take a Look at the Traffic Load pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.

147456 most used pixfirewall# show xlate count 251 in use. Cisco Public 140 . 258 most used Conn Count Is Very High. are generating the vast majority of connections   Most likely due to a virus on the host(s) BRKSEC-3020 © 2011 Cisco and/or its affiliates.Case Study: Out of Memory Step 6: Review What We Know and Take Action pixfirewall# show conn count 147456 in use. All rights reserved. but xlate Count Is Low   Many connections per xlate   Probably one. or a few hosts.

1.51>. TCP connection count/limit UDP connection count/limit include host|count/limit = 0/unlimited = 0/unlimited = 2/unlimited = 0/unlimited = 0/unlimited = 0/unlimited Only Show Lines That Have the Word host or count/limit in Them = 146608/unlimited = 0/unlimited   Host 10.1. and they are TCP-based connections BRKSEC-3020 © 2011 Cisco and/or its affiliates. TCP connection count/limit UDP connection count/limit local host: <10.1.1.1.1. Cisco Public 141 .1.1.1. .99 is eating up all the connections. TCP connection count/limit UDP connection count/limit .99>.1.131>. All rights reserved. local host: <10. TCP connection count/limit UDP connection count/limit local host: <10.Case Study: Out of Memory Step 7: Find the Host(s) Generating All the Connections pixfirewall# show local-host | local host: <10.236>. .

101.99:59163 idle 0:01:43 Bytes 0 flags TCP out 64.19:135 in 10.99:34580 idle 0:01:43 Bytes 0 flags TCP out 64.Case Study: Out of Memory Step 8: Now that We Found the Host.1. Cisco Public saA saA saA saA saA saA saA saA saA saA 142 .1.17.205.32.1.1.25.108.1.1. Let’s Look at the Connections It Is Generating pixfirewall# show local-host 10.1.1.1.1.1.1.99:7774 idle 0:01:43 Bytes 0 flags TCP out 64.103.182.120:135 in 10.100.102.99:41589 idle 0:01:43 Bytes 0 flags .1.101.100.99:16462 idle 0:01:43 Bytes 0 flags Destinations on TCP/135– TCP out 64.201. BRKSEC-3020 © 2011 Cisco and/or its affiliates.1. TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 Note: All Connections UDP connection count/limit = 0/unlimited Are Embryonic Xlate(s): Global 209.176.1.21.103.99:41116 idle 0:01:43 Bytes 0 flags TCP out 64.1.1.1. 0 denied local host: <10.141:135 in 10. 250 maximum active.1.1.21 Local 10.101.103.160:135 in 10.1. . . All rights reserved.218.1.157:135 in 10.99 Interface inside: 250 active.195:135 in 10.75:135 in 10.201.1.1.99>.99:8688 idle 0:01:43 Bytes 0 flags TCP out 64.102.45:135 in 10.1.99:2978 idle 0:01:43 Bytes 0 flags TCP out 64.99:30322 idle 0:01:43 Bytes 0 flags MS Blaster TCP out 64.99 Conn(s): TCP out 64.1.99:39193 idle 0:01:43 Bytes 0 flags Connections to Random TCP out 64.191:135 in 10.219:135 in 10.165.

All rights reserved.Case Study: Out of Memory   Cisco PIX provides two methods to limit the number of connections per host TCP intercept Max connections   Question: Which help because the source address is valid TCP intercept won’t One Can Be Used Here?   Limiting the maximum number of connections each internal host can have is the only option BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 143 .

0.1.99 pixfirewall(config)# show local-host 10. .0. .0.1. 250 maximum active. Cisco Public 144 .99 Interface inside: 250 active. 0 denied local host: <10.1.1.1. All rights reserved.0.1.0 50 0   Note: the local-host must be cleared before the new connection limits are applied pixfirewall(config)# clear local-host 10. BRKSEC-3020 © 2011 Cisco and/or its affiliates.Case Study: Out of Memory Step 9: Limit Infected Host(s) Impact on Network   Configure the MAX TCP connections for NATed hosts to be 50 pixfirewall(config)# nat (inside) 1 0.99>.0 0. The Infected Host TCP connection count/limit = 50/50 Is Limited to 50 TCP embryonic count = 50 TCP intercept watermark = unlimited TCP Connections UDP connection count/limit = 0/unlimited .

1.Case Study: Out of Memory Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit pixfirewall# show conn count 126 in use.255.0. All rights reserved.0 0.0 0 0 BRKSEC-3020 © 2011 Cisco and/or its affiliates.1.255 50 0 nat (inside) 1 0.1. Cisco Public 145 .0. 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ---------------------------Total memory: 67108864 bytes   Things look much better now   Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.0.99 255.0.255.1.99) which was infected with the virus? nat (inside) 1 10.

All rights reserved.Case Study High CPU Usage Presentation_ID © 2010 Cisco and/or its affiliates. Cisco Public 146 .

See http://www.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c. Cisco Public 147 .cisco.shtml BRKSEC-3020 © 2011 Cisco and/or its affiliates.High CPU Usage on the Cisco PIX Problem: Cisco PIX CPU Running Very High   A quick overview of the show processes command Number of msec This Process Has Been on the CPU pixfirewall(config)# show processes Hsi Lsi Lwe Lwe Crd Lsi PC 001eab19 001f00bd 00119abf 003e4425 001e26fb 00300a29 SP 008a5a74 00a28dbc 02d280dc 02d2a26c 0533940c 04c0f504 STATE 00557910 00557910 0055b070 00557dd8 00557d88 00557910 Runtime 0 0 0 74440 6070290 0 SBASE 008a4aec 00a27e44 02d27274 02d28324 05338484 04c0e57c Stack 3628/4096 3832/4096 3688/4096 6936/8192 3684/4096 3944/4096 Process arp_timer FragDBGC dbgtrace Logger 557poll xlate clean The Name of the Process For more Information on the Output of the show processes Command. All rights reserved.

These Processes Account for 44 Seconds of CPU Time ~ 73% The Interface Polling Processes Always Run. Cisco Public 148 . All rights reserved.High CPU Usage on the Cisco PIX Step 1: Determine What Process Is Eating the CPU   Take the difference in output of two show processes over a period of time   The following output was a diff of the processes taken one minute apart Process_Name Logger pix/intf3 557poll i82543_timer i82542_timer Runtime (msec) 25940 18410 9250 4180 2230 In One Minute. and Are not Counted in the CPU Usage BRKSEC-3020 © 2011 Cisco and/or its affiliates.

.18. 6172472 messages logged Logging to lab 172. 5919412 messages logged Logging to lab 172. All rights reserved.173. Over a Few Minutes pixfirewall(config)# show log Syslog logging: enabled Buffer logging: level alerts.123 BRKSEC-3020 © 2011 Cisco and/or its affiliates.173.123 History logging: disabled Notice the Change . let’s review what we have configured to log This Is Cumulative pixfirewall(config)# show log Since the Cisco PIX Syslog logging: enabled Standby logging: disabled Was Last Rebooted Console logging: disabled Monitor logging: disabled Buffer logging: level alerts. .High CPU Usage on the Cisco PIX Step 2: Focus on the Processes with High CPU Time   Logging is taking up much of the CPU. 0 messages logged Trap logging: level warnings. 0 messages logged Trap logging: level warnings.18. Cisco Public 149 .

123 172. and examine the buffered messages pixfirewall(config)# show log Buffer logging: level warnings. All rights reserved. .18.1.1.18.123 172.36.88 14.88 14.36.123 . .18.123 172.18.88 14.173.18.High CPU Usage on the Cisco PIX Syslog Server Is Controlled by a Different Group   Enable buffered logging to same level as syslog server.1.88 14.123 172.173. 400011: 400011: 400011: 400011: 400011: 400011: IDS:2001 IDS:2001 IDS:2001 IDS:2001 IDS:2001 IDS:2001 ICMP ICMP ICMP ICMP ICMP ICMP unreachable unreachable unreachable unreachable unreachable unreachable from from from from from from 172.173. Cisco Public 150 .88 on on on on on on interface interface interface interface interface interface lab lab lab lab lab lab BRKSEC-3020 © 2011 Cisco and/or its affiliates.36. 31527 messages logged Trap logging: level warnings.1.123 to to to to to to Cisco PIX’s Interface Address 14.173. 6453127 messages logged Logging to lab 172.1.173.173.36.18.88 14.173.36.123 172.1.36.18.

creating the exponentially increasing problem Syslog Server Lab Outside Syslog Message ICMP Unreachable IDS Syslog Message BRKSEC-3020 © 2011 Cisco and/or its affiliates.High CPU Usage on the Cisco PIX Examine IDS Configuration pixfirewall(config)# show run | grep audit ip audit name IDS info action alarm ip audit interface lab IDS   Syslog service was down on the syslog server   ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it   Cisco PIX’s IDS configuration also logged every ICMP unreachable message. All rights reserved. Cisco Public 151 .

5 minutes: 99% BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 152 . 1 minute: 50%.High CPU Usage on the Cisco PIX Solution   Bring back up syslog service on server   Take server offline   Configure Cisco PIX to not log IDS ICMP unreachable messages ip audit signature 2001 disable or no logging message 400011 pixfirewall# show run | grep signature ip audit signature 2001 disable pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%.

All rights reserved.High CPU Usage on the Cisco PIX Summary   Examine the DIFF of two show processes taken over a one minute interval   Find the process taking up the highest amount of CPU (excluding the polling processes)   Take actions to lower that process’s CPU time   Reexamine the CPU output. Cisco Public 153 . and repeat as necessary BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved. Cisco Public 154 .FWSM   Additional architecture information BRKSEC-3020 © 2011 Cisco and/or its affiliates.

3. Cisco Public 155 .FWSM Syslog Level vs.3 Emergencies Alerts Critical Errors Warnings Notifications 0 58 (58) 21 (79) 94 (173) 131 (304) 26 (330) Ver. All rights reserved. 3.0 0 67 (67) 29 (96) 318 (414) 199 (613) 178 (791) 255 (1046) 226 (1272) Ver. 2.2 0 67 (67) 29 (96) 306 (402) 196 (598) 169 (767) 248 (1015) 225 (1240) Ver.1 0 67 (67) 29 (96) 305 (401) 194 (595) 167 (762) 245 (1007) 225 (1232) Ver. Number of Messages Log Level 0 1 2 3 4 5 6 7 Number of Messages (SUM) Description Ver.1 0 67 (67) 29 (96) 318 (414) 199 (613) 178 (791) 259 (1050) 231 (1281) Informational 116 (446) Debugging 23 (469) BRKSEC-3020 © 2011 Cisco and/or its affiliates. 4. 4.

Cisco Public 156 . All rights reserved. rules are attempted to be pushed into hardware Successful download Access Rules Download Complete: Memory Utilization: 49% Failed download (exceeded HW memory) ERROR: Unable to add.FWSM and ACLs   ACLs on the FWSM are compiled on the control point and pushed down into hardware (NP 3)   During compile time. access-list config limit reached BRKSEC-3020 © 2011 Cisco and/or its affiliates. CPU should stay at ~ 99% ACL compile uses all free CPU cycles Allows compile to complete in shortest time possible   Once compile is complete.

FWSM and ACLs (Multimode)   Use show np 3 acl stats to see the current ACL resource utilization in that context FWSM/admin(config)# show np 3 acl stats ---------------------------ACL Tree Statistics ---------------------------Rule count : 9584 Bit nodes (PSCB's): 8760 Leaf nodes : 8761 Total nodes : 17521 (max 24260) Leaf chains : 6912 Total stored rules: 15673 Max rules in leaf : 3 Node depth : 32 ---------------------------Note: One ACE Does not Equal One Node BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public Total Number of ACEs This Is the Hardware Limit 157 . All rights reserved.

Cisco Public 158 . All rights reserved.FWSM and ACLs (Multimode)   Use show np 3 acl tree to see which ACL tree a context is mapped to ACL Tree Number FWSM# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 3 Context (004) Accounting -------------------------------------------Context Name BRKSEC-3020 © 2011 Cisco and/or its affiliates.

2 introduced resource-rule—allows further customization of a partition   FWSM 4.3 introduced resource acl-partition—set the number of ACL partitions allocate-acl-partition—assigns a context to a specific partition   FWSM 3. Cisco Public 159 .0 introduced resource partition—customize the size of individual partitions access-list optimization enable—merges and/or deletes redundant and conflicting ACEs without affecting the policy BRKSEC-3020 © 2011 Cisco and/or its affiliates.FWSM—ACL Rule Limits   FWSM 2. All rights reserved.

All rights reserved.Tree 0 Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 0 Context (004) Accounting -------------------------------------------BRKSEC-3020 © 2011 Cisco and/or its affiliates.FWSM and ACLs (Multimode)   Use the command resource acl-partition <num-ofpartitions> to reduce the number of active partitions created. default is 12   Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree FWSM(config)# context Accounting FWSM(config-context)# allocate-acl-partition 0 FWSM(config-context)# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map Both Use -------------------------------------------. Cisco Public 160 .

Configured Limit = Available to allocate 92156 92156 = 0 BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 161 . All rights reserved.FWSM—Resource Rule   FWSM 3.2 introduced resource-rule—allows further customization of a partition resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283 show resource-rule—displays information about the current rule allocation FWSM# show resource rule Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 1843 1843 10000 ACL 74188 74188 74188 Filter 2764 2764 5528 Fixup 4147 4147 10000 Est Ctl 460 460 460 Est Data 460 460 460 AAA 6451 6451 10000 Console 1843 1843 3686 -----------+---------+----------+--------Total 92156 92156 Partition Limit .

The <size> command leads to re-partitioning of ACL Memory.Configured Limit = Available to allocate 19219 19219 = 0 BRKSEC-3020 © 2011 Cisco and/or its affiliates. It will not take effect until you save the configuration and reboot. All rights reserved.0 introduced resource partition—allows customization of the size of individual partitions (multi-context mode) FWSM(config)# resource partition 10 FWSM(config-partition)# size 1000 WARNING: The rule max has been reset based on partition size 1000.FWSM—Resource Partition   FWSM 4.Configured Limit = Available to allocate 1000 1000 = 0 Cisco Public 162 . After FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 20 20 43 ACL 770 770 770 Filter 30 30 60 Fixup 80 80 160 Est Ctl 5 5 5 Est Data 5 5 5 AAA 70 70 140 Console 20 20 40 -----------+---------+----------+--------Total 1000 1000 Partition Limit . Before FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 384 384 833 ACL 14801 14801 14801 Filter 576 576 1152 Fixup 1537 1537 3074 Est Ctl 96 96 96 Est Data 96 96 96 AAA 1345 1345 2690 Console 384 384 768 -----------+---------+----------+--------Total 19219 19219 Partition Limit .

Cisco Public Fast Path Fast Path NP 1 NP 2 FWSM URL filtering (WebSense/N2H2) C6K Backplane Interface 163 . Session Manager NP 3 Control Point (CP) Central CPU   Traffic sourced from.323/RTSP) DNS XDMCP.FWSM and Control Point   The traffic that makes it to the control point is traffic that requires Layer 7 fixup (embedded NAT. or cmd inspection) FTP VoIP (SIP/SKINNY/H. All rights reserved. the FWSM also goes through the control point Syslogs AAA (RADIUS/TACACS+) Management traffic (telnet/SSH/ HTTPS/SNMP) Failover communications Routing protocols (OSPF/ RIP) etc. or destined to. etc. BRKSEC-3020 © 2011 Cisco and/or its affiliates.

BRKSEC-3020 © 2011 Cisco and/or its affiliates. Control Point (CP) Central CPU Session Manager NP 3 Fast Path NP 1 FWSM Fast Path NP 2 C6K Backplane Interface Control Point (CP) Central CPU   The fast path—NP 1 and 2 Performs per packet session lookup Maintains connection table Performs NAT/PAT TCP checks Fragmentation reassembly etc. All rights reserved.FWSM and Network Processors   The session manager—NP 3 Processes first packet in a flow ACL checks Translation creation Embryonic/established connection counts TCP/UDP checksums Sequence number randomization TCP intercept etc. Session Manager NP 3 Fast Path NP 1 FWSM Fast Path NP 2 C6K Backplane Interface 164 Cisco Public .

AFP. there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the firewall 4 3 2 1 4 2 3 1   This issue might be encountered when performing TCP throughput testing. It will then be enabled for all contexts on the firewall BRKSEC-3020 © 2011 Cisco and/or its affiliates.FWSM—Enabling the Completion Unit   Due to the FWSM’s NP architecture. backups   FWSM version 3. or passing high speed TCP flows through the FWSM   Examples: CIFS.1(10) and 3. Cisco Public 165 .2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the completion unit)   In multiple mode enter this command in the admin context configuration. FTP. All rights reserved.

Case Study Advanced Syslog Analysis .

Case Study: Advanced Syslog Analysis Problem – Find Services which are permitted through the firewall. yet the servers no longer exist   Get a fast Linux/Solaris machine with a decent amount of memory   Learn to use the following commands: •  cat •  grep. Cisco Public 167 . fgrep •  cut •  awk (basic) •  sort •  uniq •  Perl (advanced manipulation)   Pipe the commands to construct the necessary outputs! BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. egrep.

100.0.190/21 duration 0:00:30 bytes 0 SYN Timeout Reason BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 168 . All rights reserved.19.113.126/6243 to inside:10.Case Study: Advanced Syslog Analysis   Interesting syslogs appear as follows: Syslog ID Destination May 24 2010 23:19:53: %ASA-6-302014: Teardown TCP connection 1019934 for outside:203.

13/80 dmz:192.0.txt | grep "SYN Timeout" | awk '{print $13}' | uniq -c | sort -r -n 673 451 392 358 119 inside:10.0. Cisco Public 169 . All rights reserved.142/80   grep – used to find the syslogs we want   awk – used to print the destination column (IP/port)   uniq – used to print only unique entries.1.5.190/21 dmz:192. with a count   sort – used to display ordered list.5.19.67/1521 inside:10.168.100.168.Case Study: Advanced Syslog Analysis Results: syslogserver-sun% grep 302014 syslog.11/443 inside:10.0. highest count first BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved. Cisco Public 170 .Case Study FWSM – Slow Single-Flow TCP Throughput Move this case-study to the appendix Presentation_ID © 2010 Cisco and/or its affiliates.

Cisco Public 171 .Case Study: FWSM Slow TCP Throughput Problem   TCP based backups are taking longer than expected through the FWSM   iPerf performance testing is only showing ~450 Mbps through FWSM BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

4 3 2 1 4 2 3 1 TCP Flow   FWSM version 3. there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the FWSM. Cisco Public 172 . and it will be applied globally BRKSEC-3020 © 2011 Cisco and/or its affiliates.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order Enable np completion-unit 4 3 2 1 4 3 2 1 TCP Flow Note: In multi-mode add command to admin context. All rights reserved.Case Study: FWSM Slow TCP Throughput FWSM Only   Due to the FWSM’s NP architecture.

Failover   What to Do After a Failover   Additional Failover Commands BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 173 .

Cisco Public 174 .What to Do After a Failover   show failover state – will provide specific details about the failure reason. All rights reserved.   This information is not saved across reboots ASA# show failover state This host State Primary Failed Last Failure Reason Ifc Failure Inside: Failed None Date/Time 12:56:00 UTC May 6 2010 Other host - Secondary Active ====Configuration State=== Sync Done ====Communication State=== Mac set BRKSEC-3020 © 2011 Cisco and/or its affiliates.

3 and Cisco ASA/PIX 7. Cisco Public 175 . the reason for failover is saved in the failover history   This information is not saved across reboots ASA# show failover history ========================================================================== From State To State Reason ========================================================================== Disabled Negotiation Set by the CI config cmd Negotiation Just Active No Active unit found Just Active Active Drain No Active unit found Active Drain Active Applying Config No Active unit found Active Applying Config Active Config Applied No Active unit found Active Config Applied Active No Active unit found Active Failed Interface check ========================================================================== BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.0.What to Do After a Failover   Starting with FWSM 2.

ASA(config)# prompt hostname priority state ASA/sec/act(config)# BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 176 . All rights reserved.   failover reload-standby – only valid on Active unit   prompt – changes the prompt to display failover priority and state.Other Useful Failover Commands   failover exec mate – allows you to execute commands on the peer and receive the response back.

All rights reserved..]   Syntax keywords: Hostname Configures the prompt to display the hostname Domain Configures the prompt to display the domain Context Configures the prompt to display the current context (multi-mode only) Priority Configures the prompt to display the failover lan unit setting State Configures the prompt to display the current traffic handling state Slot Configures the prompt to display the slot location (when applicable)   Example FWSM(config)# prompt hostname domain priority state slot FWSM/cisco.Failover Prompt Display Configuration   The firewall’s prompt maybe changed to display certain keyword   Usage prompt <keyword> [<keyword> .. Cisco Public 177 .com/sec/actNoFailover/4(config)# BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Cisco Public 178 . All rights reserved.Capture Example BRKSEC-3020 © 2011 Cisco and/or its affiliates.

3.219.1.168.cisco.2 Capture In Inside 10.168.133. All rights reserved.25). the user is getting PATed to 192.com Outside Internet 198.2 Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.219.1.2.2 is having a problem accessing www.cisco. Cisco Public 179 .com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer Program BRKSEC-3020 © 2011 Cisco and/or its affiliates.2 Capture Out www.25 10.3.133.3.2 192.Capture Command: Example   Problem: user on the inside with an IP of 10.1.cisco.2.com (198.

133.219.pcap copy /pcap capture:in tftp://10.25 eq 80 Access-list 101 permit tcp host 198.168.3.3 copy capture copy capture:out tftp://10.Capture Command: Example   Step 1: create ACL for both inside and outside interface ! Outside Capture ACL Access-list 100 permit tcp host 192.168.133.3. All rights reserved.25 eq 80 host 192.1.25 eq 80 Access-list 100 permit tcp host 198.3.3.2 host 198.1.pcap pcap Or copy using https: https://<FW_IP>/capture/out/pcap BRKSEC-3020 © 2011 Cisco and/or its affiliates.pcap pcap copy capture:in tftp://10.cisco.3.1.2.219.1.0+ / FWSM 3.2   Step 2: create captures on both inside and outside interface capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518   Step 3: have inside user access www.1.219.5/out. Cisco Public 180 .25 eq 80 host 10.2 ! Inside Capture ACL Access-list 101 permit tcp host 10.1.5/in.5/in.0+ copy capture copy /pcap capture:out tftp://10.133.3.pcap ! PIX ver 6.com   Step 4: copy the captures off to a TFTP server ! ASA ver 7.2 host 198.x / FWSM 2.133.5/out.219.2.

No SYN+ACK BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 181 .Packet Capture: Example   Step 5: analyze captures with sniffer program Outside CAP Inside CAP Outbound SYN. All rights reserved.

Packet Capture: Limitations on FWSM   Capture functionality is available on the FWSM starting in 2.1(1) added support to capture packets in hardware Only ingress packets were captured Session Manager NP 3   FWSM 3. be careful not to flood the control point with too much traffic FWSM Fast Path Fast Path NP 1 NP 1 C6K Backplane Interface BRKSEC-3020 © 2011 Cisco and/or its affiliates.1(5) both ingress and egress transient packets can be captured which flow through hardware Capture requires an ACL to be applied Capture copies the matched packets in hardware to the control point where they are captured. Cisco Public 182 .3 However. only packets processed by the control point could be captured Control Point (CP) Central CPU   FWSM 3. All rights reserved.

Cisco Public 183 .Online Tools   Networking professionals connection   Bug toolkit   Output Interpreter BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public 184 . All rights reserved. and Anyone Can Answer Regular Ask the Expert Events on Certain Topics BRKSEC-3020 © 2011 Cisco and/or its affiliates.Networking Professionals Connection Online Open Forum to Ask Questions Anyone Can Ask a Question.

Networking Professionals Connection Online Open Forum to Ask Questions Anyone Can Ask a Question. Cisco Public 185 . All rights reserved.cisco. and Anyone Can Answer Regular Ask the Expert Events on Certain Topics http://www.com/go/netpro BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Cisco Public 186 . All rights reserved.Bug Toolkit On the Support Tools and Resources Page BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved.Bug Toolkit—Product Selection Select Security. then Cisco ASA 5500 Series BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 187 .

Cisco Public 188 . All rights reserved.Bug Toolkit—Advanced Search Version Search Keywords Severity Status BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Bug Toolkit—Search Results Select Link to View Details of Bug BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 189 .

Cisco Public 190 . All rights reserved.Bug Toolkit—Bug Details First Fixed-In Releases BRKSEC-3020 © 2011 Cisco and/or its affiliates.

Output Interpreter Linked off the Technical Support and Documentation— Tools and Resources Section on CCO Great Tool for Catching Configuration Errors Paste in the show run Output and Hit submit BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 191 .

All rights reserved.com/cgi-bin/Support/OutputInterpreter/home. Cisco Public 192 .cisco.pl BRKSEC-3020 © 2011 Cisco and/or its affiliates.Output Interpreter: Example Output Warning: Unused Statics Warning: Unapplied Crypto Map Warning: Invalid Crypto Map https://www.

Cisco Public 193 . All rights reserved.ASDM BRKSEC-3020 © 2011 Cisco and/or its affiliates.

1 and 8.ASDM   Run as a standalone application using the ADSM Launcher   This allows for one-stop access to multiple firewalls   ASDM 6. 3. All rights reserved.1 releases BRKSEC-3020 © 2011 Cisco and/or its affiliates.2 works with ASA 8.0 releases   ASDM 6.1F works with FWSM 4. ASA 8.0. Cisco Public 194 .0 adds Upgrade Wizard to upgrade ASA and ASDM software direct from cisco.2. and 3.com   ASDM 6.2.

Memory. Interface Traffic Real-Time Syslogs BRKSEC-3020 © 2011 Cisco and/or its affiliates. Conns/Sec. All rights reserved. Cisco Public 195 .ASDM Home Page Device Information CPU.

Cisco Public 196 .Using ASDM for Monitoring Up to Four Different Graphs Can Be Displayed Great for Monitoring Trends BRKSEC-3020 © 2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public 197 . All rights reserved.ASDM: Editing Rules from the Log Viewer Select Log Entry from Viewer Right-Click on Message to View or Edit Associated Rule BRKSEC-3020 © 2011 Cisco and/or its affiliates.

All rights reserved.ASDM: Syslogs Explained BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 198 .

please open a TAC case http://www.com/techsupport/servicerequest/   At a minimum include: Detailed problem description Output from show tech   Optionally include: Syslogs captured during time of problem Sniffer traces from both interfaces using the capture command (capturing only the relevant packets.cisco.Opening a TAC Case   If after using all your troubleshooting tools you still cannot resolve the problem. and saved in pcap format) BRKSEC-3020 © 2011 Cisco and/or its affiliates. Cisco Public 199 . All rights reserved.