Thursday, October 9, 2008
Common commands you should master when working with the Cisco IOS
#1: The “?”
It may seem entirely too obvious that you should know how to type ? to ask for help when using the Cisco IOS. However, the Cisco IOS is completely different from other operating systems when it comes to using the question mark (help key). As the IOS is a command-line operating system with thousands of possible commands and parameters, using the ? can save your day.
You can use the command in many ways. First, use it when you don’t know what command to type. For example, type ? at the command line for a list of all possible commands. You can also use ? when you don’t know what a command’s next parameter should be. For example, you might type show ip ? If the router requires no other parameters for the command, the router will offer CR as the only option. Finally, use ? to see all commands that start with a particular letter. For example, show c? will return a list of commands that start with the letter c.
#2: show running-configuration
The show running-config command shows the router, switch, or firewall’s current configuration. The running-configuration is the config that is in the router’s memory. You change this config when you make changes to the router. Keep in mind that config is not saved until you do a copy running-configuration startup-configuration. This command can be abbreviated sh run.
#3: copy running-configuration startup-configuration
This command will save the configuration that is currently being modified (in RAM), also known as the running-configuration, to the nonvolatile RAM (NVRAM). If the power is lost, the NVRAM will preserve this configuration. In other words, if you edit the router’s configuration, don’t use this command and reboot the router–those changes will be lost. This command can be abbreviated copy run start. The copy command can also be used to copy the running or startup configuration from the router to a TFTP server in case something happens to the router.
#4: show interface
The show interface command displays the status of the router’s interfaces. Among other things, this output provides the following:
* Interface status (up/down)
* Protocol status on the interface
* Utilization
* Errors
* MTU
This command is essential for troubleshooting a router or switch. It can also be used by specifying a certain interface, like shint fa0/0.
#5: show ip interface
Even more popular than show interface are show ip interface and show ip interface brief. The show ip interface command provides tons of useful information about the configuration and status of the IP protocol and its services, on all interfaces. The show ip interface brief command provides a quick status of the interfaces on the router, including their IP address, Layer 2 status, and Layer 3 status.
#6: config terminal, enable, interface, and router
Cisco routers have different modes where only certain things can be shown or certain things can be changed. Being able to move between these modes is critical to successfully configuring the router.
For example, when logging in, you start off at the user mode (where the prompt looks like >). From there, you type enable to move to privileged mode (where the prompt looks like #). In privileged mode, you can show anything but not make changes. Next, type config terminal (or config t) to go to global configuration mode (where the prompt looks like router(config)# ). From here, you can change global parameters. To change a parameter on an interface (like the IP address), go to interface configuration mode with the interface command (where the prompt looks like router(config-if)#). Also from the global configuration mode, you can go into router configuration using the router {protocol} command. To exit from a mode, type exit.
#7: no shutdown
The no shutdown command enables an interface (brings it up). This command must be used in interface configuration mode. It is useful for new interfaces and for troubleshooting. When you’re having trouble with an interface, you may want to try a shut and no shut. Of course, to bring the interface down, reverse the command and just say shutdown. This command can be abbreviated no shut.
#8: show ip route
The show ip route command is used to show the router’s routing table. This is the list of all networks that the router can reach, their metric (the router’s preference for them), and how to get there. This command can be abbreviated shipro and can have parameters after it, like shiproospf for all OSPF routers. To clear the routing table of all routes, you do clear ip route *. To clear it of just one route, do clear ip route 1.1.1.1 for clearing out that particular network.
#9: show version
The show version command gives you the router’s configuration register (essentially, the router’s firmware settings for booting up), the last time the router was booted, the version of the IOS, the name of the IOS file, the model of the router, and the router’s amount of RAM and Flash. This command can be abbreviated shver.
#10: debug
The debug command has many options and does not work by itself. It provides detailed debugging output on a certain application, protocol, or service. For example, debug ip route will tell you every time a router is added to or removed from the router.
Mitigate problems using Cisco IOS debug
This article is intended for the intermediate level network administrator who has never encountered circumstances that would require debugging the Cisco IOS. But if you've used debug before, perhaps this article will give you some suggestions for ways to use debug that you haven't previously tried.
Key points
Before I introduce 10 ways to use Cisco IOS debug, here are a few tips and caveats:
- · The techniques I'm including are not the only ways to use debug. In fact, there are hundreds of ways to use debug. Listing A: shows all the possible subcommands that can be debugged in IOS 12.2 (the version I was running). You will see that there are more than you would ever want to use. In addition, each subcommand has its own subcommands. I'll be focusing on some of the debug commands I consider most common and useful.
- · Using Cisco IOS debug can be dangerous and should first be used on a test router and then used carefully on a production router. I don’t want anyone to get fired because he crashed the company’s production router after trying debug commands from this article.
- · By default, debug messages will go to the console and will not be sent to any log. To log debug output (or any system output) to the IOS system log, you must turn on logging buffered in global configuration mode. You can see if logging is on by issuing the show logging command on your router. The output looks like Listing B. As you can see, there are three kinds of logging: console, monitor, and buffer. There are also different levels of logging.
- · If you want to see debug output on a terminal (or Telnet session) that is not the console, use the terminal monitor command.
- · You can also set up “conditional debugging” to debug only when certain conditions apply. See Cisco's Troubleshooting and Fault Management to learn more about this.
- · If you turn on some debugging that is filling up your screen, the quickest way to turn it off is by running the un all command. The un is a built-in macro for undebug.
Now let’s move on and look at 10 ways to use Cisco IOS debug. We'll start with number 10 and work our way up to the most useful debug command.
10. Debug all
By far, the most dangerous debug command is debug all. This command will most likely crash a production router. The Cisco IOS is nice enough to ask, “Are you sure?” before doing it. Only use this command if you are desperate, you know you are on a test router, and/or you are having trouble isolating the proper subcommand to debug. To satisfy your curiosity of what it looks like to run it, Listing C shows the output from executing the command and then issuing show debug to see what it turned on.
9. Debug with ISDN and dialer
Debugging is helpful when troubleshooting Cisco router dialup configurations. In my experience, these technologies rarely seem to work the first time. The best way to configure IOS dialup technologies (or any networking technologies) is layer by layer, testing each layer after configuring it. I try to think of a multilayer cake, where the icing is the successful testing of each OSI layer after each configuration. When something doesn’t work, debug is usually a must. Useful debug commands for ISDN and dialup networking appear below:
- · debug isdn events debugs all ISDN events. Start with this one first before trying q921 or q931, as those are very verbose and events may tell you all you need to know.
- · debug isdn q921 is Layer 2.
- · debug isdn q931 is Layer 3.
- · debug dialer events will tell you about the calls (i.e., the reason the call was initiated and/or disconnected). Here is some sample output:
00:18:52: BRI0/0 DDR: Attempting to dial 8358661
00:19:22: BRI0/0:2 DDR: disconnecting call
This can be especially helpful if you have a router that keeps calling when you don't want it to.
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:41:09: BRI0/0 DDR: cdp, 273 bytes, outgoing uninteresting (no list matched)
8. PPP authentication
If you configured PPP authentication on your dialup lines (which you should, for security reasons), you could end up with username/password mismatches on each side. Finding out that this is the reason that the line won’t come up can be impossible without using debug ppp authentication.
Here's some sample output of missing username/password information on one router:
00:32:30: BR0/0:1 CHAP: O CHALLENGE id 13 len 23 from "r2"
00:32:31: BR0/0:1 CHAP: I CHALLENGE id 2 len 23 from "r1"
00:32:31: BR0/0:1 CHAP: O RESPONSE id 2 len 23 from "r2"
00:32:31: BR0/0:1 CHAP: I FAILURE id 2 len 26 msg is "Authentication failure"
This is sample output of an incorrect password on one router:
00:47:05: BR0/0:1 CHAP: O CHALLENGE id 25 len 23 from "r2"
00:47:05: BR0/0:1 CHAP: I CHALLENGE id 19 len 23 from "r1"
00:47:05: BR0/0:1 CHAP: O RESPONSE id 19 len 23 from "r2"
00:47:05: BR0/0:1 CHAP: I FAILURE id 19 len 25 msg is "MD/DES compare failed"
7. Debug {topology} packet
As I mentioned above, it is best to troubleshoot networking technologies using a layered approach, working your way up the OSI model from Layer 1. (I give credit to Bruce Caslow and his book Cisco Certification: Bridges, Routers, and Switches for CCIEs for carefully documenting this approach.)
Using that model, for whatever network topology you are working with, you can use the debug command to see that your Layer 2 packets are being encapsulated properly. (Of course, I'm assuming that all the cables are connected at Layer 1.) For instance, let's say that you work with frame-relay and you're having trouble getting packets across a link. You've verified that the link is up, but there is still no data passing on it. You could try
debug frame-relay packet
Then, you could attempt to ping the interface on the remote router. If the ping passes, you will get the following:
01:03:22: Serial0/0:Encaps failed--no map entry link 7(IP)
This tells you that encapsulation failed for the IP packet into the frame-relay protocol. It even tells you that it failed because there is no frame-relay map statement. If you fix that and try it again, you might find that you get no frame-relay errors. But then the packet still may not pass. Thus, you'll need to go up to Layer 3 and try:
debug ip packet
You might get this output:
01:06:46: IP: s=1.1.1.2 (local), d=11.11.11.11, len 100, unroutable
This tells you that there is no IP route for your packet to traverse. Thus, you know you need to add a route.
You can insert whatever networking topology you are working with on the debug {topology} line. Some other possibilities are:
- · debug atm packet
- · debug serial packet
- · debug ppp packet
- · debug dialer packet
- · debug fastethernet packet
6. Debug crypto (IPSec and VPN features)
I won’t go into a lot of detail and examples here since IPSec and VPN are huge topics that can get very complex. But I do want to point out that you can troubleshoot IPSec connections with debugging commands. Because of the popularity of VPN and IPSec, it is important to know how to use debug commands for these technologies. Some of the possibilities are:
- · debug crypto isakmp
- · debug crypto ipsec
- · debug crypto engine
- · debug ip security
- · debug tunnel
In addition, debug ip packet, as mentioned below, is useful in troubleshooting IPsec connections.
If you want to learn more on this topic, I recommend going to Cisco’s IP Security Protocols Section, choosing the protocol you are interested in (IPSec, IKE, etc.), and looking at the different configurations along with their recommended troubleshooting methods.
5. Debug IP routing
Using debug with IP routing protocols can be useful as well. For example, to know if you have a route flapping (a route being added and removed frequently), you can use debug ip routing.
If you had a flapping route, you might see something like:
01:30:56: RT: add 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:31:13: RT: del 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:31:13: RT: delete subnet route to 111.111.111.111/32
01:31:13: RT: delete network route to 111.0.0.0
01:32:56: RT: add 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:33:13: RT: del 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:33:13: RT: delete subnet route to 111.111.111.111/32
01:33:13: RT: delete network route to 111.0.0.0
This could indicate a routing loop in your network. On the other hand, it could indicate a link going up and down, perhaps a bouncing dialup interface or frame-relay interface.
4. Debug ip {routing protocol}
You can insert many options with debug ip. Issuing the debug ip ? command will display them. We've included the options in Listing D.
Most routing protocols (e.g., OSPF, EIGRP, IGRP, and BGP) are included in this list. Each has its own extensive options that can be debugged. (You can see all the options for a particular protocol by issuing the debug ip {routing protocol} ? command.) For instance, the only way to know that your two OSPF routers are not forming an adjacency due to mismatched authentication types is to issue debug ip ospf adjacency.
You might see the following output, telling you that the authentication types are mismatched:
01:39:46: OSPF: Rcv pkt from 12.12.12.11, Serial0/0 : Mismatch Authentication type. Input packet specified type 0, we use type 2
In this case, if it weren't for debug, you could be scratching your head for a while.
3. Debug list
An interesting debug command that isn’t widely known is debug list. This command doesn’t actually debug anything. Instead, it sets an interface and/or access list for the next debug you enter. So if you are going to issue some command that you know will produce a lot of output but that doesn’t have an option to limit that output with an access command, you can first issue the debug list XXX command (where XXX is an access list number you have already created) and then run the command you want to debug on. An example would be:
debug list 1
debug dhcp detail
DHCP client activity debugging is on for access list: 1 (detailed)
2. Log an access list to system or syslog
You can use the log option at the end of an access list to log packets that are permitted or denied. This could enable you to log packets that are denied on a router being used as a firewall or to control access at an important network point. Remember that a firewall doesn’t have to just protect you from the Internet; it can protect higher security departments (such as accounting), be placed between two business partners, or just keep some traffic (like P2P traffic) off your network. While this isn’t technically a debug command, it can certainly be used for debugging and troubleshooting. Let's look at a couple of examples.
First, suppose that you want to allow only BGP traffic in your network and keep track of all other traffic that attempts to enter a link. Here's a sample configuration:
Interface Serial 0/0
Access-group 100 in
access-list 100 remark Begin -- Allow BGP IN and OUT
access-list 100 permit tcp host 1.1.1.1 host 2.2.2.2 eq bgp
access-list 100 permit udp any host 2.2.2.2 gt 33000
access-list 100 remark End
access-list 100 deny ip any any log
All traffic will go to your Cisco router’s log if you have “logging buffered” turned on or to a syslog server if you configured one.
For the second example, let's say you are trying to nail down an access list to permit only the proper traffic to go over your dialup link. You use an access list and log the existing traffic to get an idea of what is being carried so that you can configure the correct access list. Here's a sample configuration:
Interface BRI 0/0
Access-group 100 in
Access-group 100 out
access-list 100 permit ip any any log
If you look at the output in the router’s log, you will see that an ICMP packet and a TCP packet went across your link:
02:03:43: %SEC-6-IPACCESSLOGDP: list 100 permitted icmp 1.1.1.2 -> 1.1.1.1 (0/0), 1 packet
02:06:25: %SEC-6-IPACCESSLOGP: list 100 permitted tcp 1.1.1.2(0) -> 1.1.1.1(0), 1 packet
1. Debug IP packet detail XXX (access list number)
My number-one debug technique is using an access list to see all the traffic going to and from some destination, just like a basic sniffer/protocol analyzer. The limitation to this one is that you can’t see the packet payload and can see almost nothing of the headers.
Since you can do this with an access list, you can nail it down to a particular host, protocol, port, or network range, as well as using a uni- or bi-directional access list. I know it isn’t a real protocol analyzer, but it could still come in handy—and it's already built into the IOS. The sample configuration below shows the logging of all Telnet packets to your router. (These could just as well be a host on one of your router’s networks.)
access-list 101 permit tcp any any eq telnet
debug ip packet detail 101
IP packet debugging is on (detailed) for access list 101
Listing Eshows the sample output of what the debug output would look like.
You can see from the output that you get the IP source, destination, interfaces, source and destination port numbers, sequence numbers, acknowledgement numbers, window sizes, and TCP communication flags (SYN, ACK, FIN).
Troubleshoot Cisco routers and switches using the debug commands - Part 3
What are the three most common mistakes made when using Debug?
Using Debug can be a risky proposition, and even experienced admins have made mistakes when using it.
I’d say the number one common mistake is to forget that you have left Debug on in a production environment. Sometimes, we get so focused on resolving the issue that when we get it resolved, we are on to the next “opportunity” and forget to issue the no debug command to turn off debugging. I think that many a network admin can attest to horror stories of when they brought their router to its knees because they forgot this simple task of turning off Debug.
The second common mistake would be not realizing the effect on your router of issuing a lot of Debug commands at the same time. Remember that the router’s job is to forward packets, not to monitor processes and generate Debug messages. For example, you are having a problem with the packets on your router, so you issue the Debug statement debug ip packet. Then you decide that you want to view the events on the RIP protocol. Now, you have two separate Debug statements that are being processed and sent to the console. Debug statements are processed at a higher priority than other network traffic, so, needless to say, these Debug statements can jeopardize your router’s performance.
The third common mistake made with the Debug command is entering debug all or debug ip packet detail on a production router. Either one of these commands can crash a heavily loaded production router. Luckily, there is an “are you sure” prompt before these take effect; however, that hasn’t prevented every debug-related catastrophe. You should be as specific as possible when using Debug, and then turn it off as quickly as possible. Also, always test your Debug commands on a test router before using them in a production environment.