Section 35.2. Objective 2: Advanced Network Configuration and Troubleshooting


35.2. Objective 2: Advanced Network Configuration and Troubleshooting

For this Objective, make sure that you focus on troubleshooting, not just knowing the names of commands. By troubleshooting, we mean being able to:

  • Use the applications and configuration files discussed earlier in the chapter

  • Review configuration files for simple problems (e.g., misspellings and mis-entered lines)

  • Review log files for problems

  • Generally keep your cool when faced with balky systems

Although the LPI Exams don't really test you on the last item, we assume that many people reading this book are using it to learn essential Linux skills, as the LPI Exam examines real-world knowledge.

This section contains scenarios that help you learn more about the depth of troubleshooting skills you will need. We'll cover practical uses of netstat, tcpdump, ethereal (and the associated tethereal application), lsof, and netcat (nc). It is also important for you to understand ping and the utilities shown earlier in this chapter in a troubleshooting situation. But we'll start with one of the most basic utilities in networking.

35.2.1. Simple Connectivity Example: Telnet

Suppose an employee reports that the intranet web server is unavailable. What do you do? Open up a web browser? That won't provide much more informaiton than the employee already told you. Instead, try Telnet. Don't get incredulousTelnet is a terrific troubleshooting tool.

Depending on your location, you can use Telnet to log in to the system and check the process list to see if the web server is running. If it is, issue the command telnet localhost 80 while still on the web server host, specifying that you want to talk to port 80 (HTTP) on the server. A sample session looks like this:

 # telnet localhost 80 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. HEAD / HTTP/1.0 HTTP/1.1 302 Found Date: Mon, 26 Jul 2005 13:14:44 GMT Server: Apache/1.3.29 (Debian GNU/Linux) PHP/4.1.2 mod_ssl/2.8.16 OpenSSL/0.9.7c X-Powered-By: PHP/4.1.2 Location: http://intranet.company.test/ Connection: close Content-Type: text/html; charset=iso-8859-1 Connection closed by foreign host. 

The typical HTTP response from the web server tells you it can respond to requests properly on port 80. But this test bypasses any possible network problem, because you're on the same host as the web server when you run Telnet.

Now run the Telnet command from another host:

 # telnet intranet.company.test 80 

The outcome of this test determines what you do next: success will suggest you query the employee more carefully to find out why his particular request failed, while failure suggests you look at the components of your network that permit communication, including DNS and the web server configuration itself.

Simple as it may be, Telnet may be one of your most valuable tools when troubleshooting networks. We would never willingly use Telnet to configure a remote system, but it is very useful for troubleshooting. netcat is much more capable, of course. However, netcat is not always available, while Telnet usually is. You would not use SSH when troubleshooting, because it will try to encrypt the connection, causing the connection to fail.

The ability to speak directly to the service's listening port at a low level using Telnet or netcat is often invaluable, because client software such as a browser works at too high a level. It does not give an administrator the required level of control over situations. The great strength of using Linux is that you can break down troubleshooting into network connectivity, DNS resolution, and so on, using various tools. It is vital that you use command-line tools, because they provide the most information. A Telnet test to see whether the remote service is listening gives a quite clear answer to whether this is a network problem or a client/server application problem.

35.2.1.1. Using tcpdump

tcpdump is another extremely useful tool. Contrary to its name, it does a good deal more than display information from the TCP headers of packets; it actually understands a wide range of protocols. So you can use tcpdump to read packet captures created by various applications (e.g., ethereal and Snort).

tcpdump is text based; it does not have a fancy GUI or even an ncurses interface. The independence from the X Window System is a blessing, because many times you will be asked to monitor traffic from a host that does not have X running.

Some still might think that tcpdump's lack of a fancy GUI indicates the tool is unsophisticated. They would be mistaken. tcpdump is quite sophisticated. It can sniff traffic on many types of networks, including 802.1Q VLANs. It can sniff on standard hub-based networks as well as switched-based networks. There is one overriding factor regarding tcpdump: you need to use the right options and syntax. Otherwise, you will be lost.

The reason your choice of options is so important is that a production network provides tcpdump with far too much information. That is the main difficulty in using tcpdump. You need to tell it to be selective.

The syntax for tcpdump is:

 tcpdump [options] [expression] 

Some of the most commonly used options are:


-c count

Capture count packets, then stop.


-D

List the devices (interfaces) that are configured on the system. If you are lucky, the device will have an accompanying description that will help you know whether the interface can capture packets.


-i interface

Listen to interface. If set to any, tcpdump listens to all interfaces.


-q

Quiet operation: show less information on each capture. (The manpages refer to -q as "quick" as well.)


-r file

Read a previously written tcpdump file.


-s 0

Set the "snaplength" to 0. tcpdump thus reads the whole packet instead of just the first 68 bytes, which is the default.


-t

Suppress timestamp entries.


-v, -vv, -vvv

Print a lot of information; the more v s in the option, the more verbose the output.


-w file

Write the displayed information to file. This file may later be analyzed at your leisure using either tcpdump in read mode (with the -r file option) or ethereal.

35.2.1.2. tcpdump expressions

It is not enough to know tcpdump's options; the volume of traffic displayed makes it unlikely you will find what you need to know unless you also use expressions. The ones given here are quite useful. Make sure that you understand them before you take the exam.


not port ssh

Filter out any SSH traffic. Extremely useful if you run tcpdump from a shell on a remote host, so you won't see your own login connection.


Tip: It is more accurate to say that the not port ssh command suppresses traffic that uses port 22. The SSH protocol uses this port by default. If, for some reason, some other application was using that port, you would not view that traffic, either.

ip proto \\icmp

Specify that only ICMP traffic is to be monitored.


dst 192.168.0.1

Monitor only network traffic destined for host 192.168.0.1.


src 192.168.0.1

Monitor only for network traffic originating from host 192.168.0.1.


host 192.168.0.1

Monitor only network traffic involving host 192.168.0.1, both ways.


host 192.168.0.1 and host 192.168.0.2

Monitor only traffic between these two hosts, both ways.


src 192.168.0.1 and dst 192.168.0.2

Monitor only traffic from 192.168.0.1 to 192.168.0.2, not vice versa.

As these sample expressions show, you can manipulate the options by using familiar Boolean operations, expressed as words or punctuation: not (!), and (&&), or (||).

Some examples of how you can use tcpdump to monitor network traffic follow.

35.2.1.3. Example #1

In this example, you will view all network traffic on the second Ethernet interface (eth1) of the host having the IP address of 192.168.45.1, with the exception of SSH traffic:

 tcpdump -i eth1 -v -vv -vvv -s0 not port ssh and host 192.168.45.1 

This command might be useful if you are on a host on a local area network and want to see everything traveling to and from a hub.

35.2.1.4. Example #2

The following example captures all network traffic on interface eth0 destined to port 80 on the host 192.168.0.20 and writes the raw data to the file /tmp/web:

 tcpdump -i eth0 -w /tmp/web -s0 dst 192.168.0.20 and dst port 80 

This command actually doesn't need to specify the interface, because tcpdump defaults to the eth0 interface.

35.2.1.5. Example #3

In this example, the expression captures a brief view of the handshake that occurs when an email client contacts a POP-3 daemon residing on host 192.168.0.20:

 # tcpdump -t -n -q host 192.168.0.20 and host 192.168.1.18 and port 110 tcpdump: listening on eth0 192.168.1.18.1035 > 192.168.0.20.110: tcp 0 (DF) [tos 0x10] 192.168.0.20.110 > 192.168.1.18.1035: tcp 0 (DF) 192.168.1.18.1035 > 192.168.0.20.110: tcp 0 (DF) [tos 0x10] 192.168.0.20.110 > 192.168.1.18.1035: tcp 113 (DF) 192.168.1.18.1035 > 192.168.0.20.110: tcp 0 (DF) [tos 0x10] 192.168.1.18.1035 > 192.168.0.20.110: tcp 14 (DF) [tos 0x10] 192.168.0.20.110 > 192.168.1.18.1035: tcp 0 (DF) 192.168.0.20.110 > 192.168.1.18.1035: tcp 36 (DF) 

tcpdump captures the packets in "quiet mode," meaning it provides only an overview showing the IP addresses involved as well as source and destination ports. As the example shows, the source port is an arbitrary port above 1023 on the client IP. The output is useful for debugging.

35.2.1.6. Example #4

For this final example, we have set an improper static route to the client IP address on the destination POP-3 server. The result of this misconfiguration is that all of the return packets are lost. The tcpdump output shows that the client tries to initiate a connection, but cannot get a response from the POP-3 server:

 # tcpdump -t -n -q host 192.168.0.20 and host 192.168.1.18 and port 110 tcpdump: listening on eth0 192.168.1.18.1037 > 192.168.0.20.110: tcp 0 (DF) 192.168.1.18.1037 > 192.168.0.20.110: tcp 0 (DF) 192.168.1.18.1037 > 192.168.0.20.110: tcp 0 (DF) 192.168.1.18.1037 > 192.168.0.20.110: tcp 0 (DF) 192.168.1.18.1037 > 192.168.0.20.110: tcp 0 (DF) 

tcpdump output such as this is usually caused either by asymmetric routingwhere the return packets leave the host on another interface from the one that received trafficor when a service has broken down but still keeps listening to the TCP port.

Make sure that you are familiar with how to exclude packets, as well as how to "drill down" into the packets that you have captured using tcpdump. Not only will you find it easier to pass the exam, but you will also be a better networker.

35.2.1.7. ethereal and tethereal

Although tcpdump is versatile and powerful, ethereal has much to offer it as well; it is an excellent GUI-based packet sniffer. You can use ethereal, as you can tcpdump, to obtain a detailed analysis of network traffic. It may be used in interactive mode to capture network activity on the fly or in file mode to analyze an input file captured earlier. ethereal is capable of reading various packet capture files, including those created by tcpdump. A text-only version called tethereal is usually installed along with ethereal.

The syntax for ethereal and tethereal is:

 etheral [options] [file] tetheral [options] [file] 

A couple of frequently used options, which are the same as the corresponding tcpdump options, are:


-c count

Capture only count packets.


-s snaplength

Set the amount of each packet read. Set to 0 to ensure the entire packet is captured.

tethereal's output from a simple ping packet (ICMP request) looks like this:

 # tethereal -V -r /tmp/ping Frame 1 (98 bytes on wire, 96 bytes captured)     Arrival Time: Apr 26, 2004 06:20:32.926621000     Time delta from previous packet: 0.000000000 seconds     Time since reference or first frame: 0.000000000 seconds     Frame Number: 1     Packet Length: 98 bytes     Capture Length: 96 bytes Ethernet II, Src: 00:c0:f0:30:1d:5e, Dst: 00:01:02:f7:7c:4b     Destination: 00:01:02:f7:7c:4b (3com_f7:7c:4b)     Source: 00:c0:f0:30:1d:5e (Kingston_30:1d:5e)     Type: IP (0x0800) Internet Protocol, Src Addr: 192.168.1.18 (192.168.1.18), Dst Addr: 192.168.0.20 (192.168.0.20)     Version: 4     Header length: 20 bytes     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)         0000 00.. = Differentiated Services Codepoint: Default (0x00)         .... ..0. = ECN-Capable Transport (ECT): 0         .... ...0 = ECN-CE: 0     Total Length: 84     Identification: 0x0000 (0)     Flags: 0x04         0... = Reserved bit: Not set         .1.. = Don't fragment: Set         ..0. = More fragments: Not set     Fragment offset: 0     Time to live: 64     Protocol: ICMP (0x01)     Header checksum: 0xb832 (correct)     Source: 192.168.1.18 (192.168.1.18)     Destination: 192.168.0.20 (192.168.0.20) Internet Control Message Protocol     Type: 8 (Echo (ping) request)     Code: 0     Checksum: 0xb958     Identifier: 0x3507     Sequence number: 0x0000     Data (54 bytes) 0000  4b aa 8c 40 3b b2 0b 00 08 09 0a 0b 0c 0d 0e 0f   K..@;........... 0010  10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f   ................ 0020  20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f    !"#$%&'( )*+,-./ 0030  30 31 32 33 34 35                                 012345 

As the output shows, every network layer and protocol involved are analyzed.

Figure 35-3 shows an example of ethereal viewing packets on an Ubuntu system.

Figure 35-3. ethereal in action


The ethereal layout makes it much easier to view the important details of packets than the output of either tcpdump or tethereal. ethereal also provides fairly effective ways to measure network traffic volumes. You can filter packets either during a packet capture, using expressions, or after the capture.

35.2.1.8. The lsof command

The lsof command lists all open files on your system. Because Linux systems treat everything, including open network sockets, as files, lsof is useful when troubleshooting network connections. By using a few of its options, as well as piping it through grep, you can determine the origin of network connections, as well as find open ports on your system.

The syntax for using lsof is:

 lsof [options] 

Some of the most frequently used options include:


-i

List IP sockets. Accepts parameters that can restrict the sockets listed, such as -i tcp, -i udp, or -i 192.168.1.1.


-n

Do not resolve hostnames; can make lsof run faster.


-P

Do not resolve port names; can make lsof run faster

All the examples in this section include the -i option, because we want to see only sockets and not other open files on the system. Some examples further restrict the types of sockets viewed to a particular protocol.

35.2.1.9. Example #1

This example tells lsof to report only open network files (i.e., sockets) and to ignore hostnames:

 # lsof -i -n COMMAND    PID    USER   FD   TYPE DEVICE SIZE NODE NAME dhclient  1047    root    5u  IPv4   2068       UDP *:bootpc portmap   1113     rpc    3u  IPv4   2183       UDP *:sunrpc portmap   1113     rpc    4u  IPv4   2187       TCP *:sunrpc (LISTEN) rpc.statd 1132 rpcuser    4u  IPv4   2217       UDP *:1024 rpc.statd 1132 rpcuser    5u  IPv4   2207       UDP *:884 rpc.statd 1132 rpcuser    6u  IPv4   2220       TCP *:1024 (LISTEN) cupsd     1267    root    0u  IPv4   2487       TCP 127.0.0.1:ipp (LISTEN) cupsd     1267    root    2u  IPv4   2488       UDP *:ipp sshd      1470    root    3u  IPv6   2567       TCP *:ssh (LISTEN) xinetd    1483    root    5u  IPv4   2605       TCP 127.0.0.1:1025 (LISTEN) sendmail  1501    root    4u  IPv4   2650       TCP 127.0.0.1:smtp (LISTEN) 

35.2.1.10. Example #2

This example further restricts the listing to TCP-based connections:

 # lsof -i tcp -n COMMAND    PID    USER   FD   TYPE DEVICE SIZE NODE NAME portmap   1113     rpc    4u  IPv4   2187       TCP *:sunrpc (LISTEN) rpc.statd 1132 rpcuser    6u  IPv4   2220       TCP *:1024 (LISTEN) cupsd     1267    root    0u  IPv4   2487       TCP 127.0.0.1:ipp (LISTEN) sshd      1470    root    3u  IPv6   2567       TCP *:ssh (LISTEN) xinetd    1483    root    5u  IPv4   2605       TCP 127.0.0.1:1025 (LISTEN) sendmail  1501    root    4u  IPv4   2650       TCP 127.0.0.1:smtp (LISTEN) 

35.2.1.11. Example #3

This example lists only UDP-based connections:

 # lsof -i udp -n COMMAND    PID    USER   FD   TYPE DEVICE SIZE NODE NAME dhclient  1047    root    5u  IPv4   2068       UDP *:bootpc portmap   1113     rpc    3u  IPv4   2183       UDP *:sunrpc rpc.statd 1132 rpcuser    4u  IPv4   2217       UDP *:1024 rpc.statd 1132 rpcuser    5u  IPv4   2207       UDP *:884 cupsd     1267    root    2u  IPv4   2488       UDP *:ipp 

35.2.1.12. Example #4

In this example and the next, notice how you can use grep to isolate the port used by the network socket:

 # lsof -i -n -P | grep sendmail sendmail  1501    root    4u  IPv4   2650       TCP 127.0.0.1:25 (LISTEN) 

This example uses the -P option, which tells lsof not to convert port numbers to names. As a result, you see (in the right-most column, just after the colon) that sendmail is running on port 25, the standard SMTP port.

35.2.1.13. Example #5

In this example, notice that the +P option is used, so that the name of the protocol instead of the port number is listed:

 # lsof -i -n +P | grep sendmail sendmail  1501    root    4u  IPv4   2650       TCP 127.0.0.1:smtp (LISTEN) 

Notice now that you see smtp in the port field. The +P option is the default.

One possible use of lsof is in combination with the fuser commandfor example, to kill certain processes associated with suspicious network connections or to stop malfunctioning network processes.

35.2.1.14. The netstat command

netstat, like lsof, can find standard ports open on your system, as well as ports opened by applications such as netcat. netstat can help you determine TCP- and UDP-based connections and let you view your system's routing table.

netstat's syntax is:

 netstat [options] 

Frequently used options include:


-a

List all sockets.


-e

Extended mode. Use twice for more details.


--inet

List only IP connections.


-l

Show only listening sockets.


-n

Do not resolve IP addresses to hostnames.


-p

Show the process ID (PID) and name of the program holding the socket.


-r

Show the host's routing table (similar to the route command).

35.2.1.15. Example #1

List only open IP-based network connections and show the PID, as well as the program and username that opened the file:

 # netstat -apne --inet 

This command also places output into extended view (-e option). This offers more information, including the network socket's inode and the associated user.

35.2.1.16. Example #2

You can also list all of the ports that are currently listening:

 # netstat -le --inet 

35.2.1.17. Example #3

To view the system's routing table and ignore name resolution, enter:

 # netstat -nre 

35.2.2. netcat (nc)

netcat allows you to establish, analyze, and read connections between systems. It is a very useful program, and you should count it as one of the more useful weapons in your troubleshooting arsenal. You can also use netcat to open ports on your local system, which has various uses. For example, you can place your system behind the firewall and run netcat on your system to open up a port (e.g., port 80). You can then conduct a penetration test using a system outside of the firewall to see whether the firewall is properly blocking packets to internal systems. Conveniently enough, you can use netcat to simulate both the client and the server in this case.

netcat is installed by default on many systems. If it is not installed, obtain it by using your packaging tool (e.g., rpm, YaST, dpkg, apt, or Synaptic) or download the source code from http://netcat.sourceforge.net.

The syntax is:

     nc [options] hostname [ports]     nc -l -p port [options] [hostname] [ports] 

On many Linux systems, nc is the actual binary and the netcat file is a symbolic link to it.

netcat is capable of connecting to and listening on any port. It can use both TCP and UDP. Because netcat is able to both listen and send, it may be used for file transfers between systems. Many systems are not configured with Samba, FTP, or SSH; netcat acts as a useful, quick, adhoc file transfer solution in these cases.

Finally, you can also use netcat to analyze the protocol of a network client by directing the client to a listening netcat process. You can then view the connection's state to determine how the host handles connections.

35.2.2.1. Example #1

When you start netcat with host and port arguments, the service running on the given host and port are contacted. Whatever you enter into netcat will be sent to the service, and any response will be shown to you. Following is a simple example.

First, tell netcat to read port 80 on your own system:

 $ nc localhost 80 

Once you connect, netcat will keep the connection persistent. You may be tempted to think that the connection has "hung," but that is not the case. netcat is simply keeping it open, which allows you to then formulate a request. In this case, because you have opened a connection to your web server, create a simple HTTP request, imitating a basic web client:

 HEAD / HTTP/1.0 

The web server will then issue a response, similar to that shown here:

 HTTP/1.1 302 Found Date: Mon, 26 Jul 2005 04:27:12 GMT Server: Apache/1.3.29 (Debian GNU/Linux) PHP/4.1.2 mod_ssl/2.8.16 OpenSSL/0.9.7c X-Powered-By: PHP/4.1.2 Location: http://www.company.test/ Connection: close Content-Type: text/html; charset=iso-8859-1 

What you see will vary, depending on how verbose your version of the Apache server is and the nature of the request you type. Notice in the preceding example how the web server has returned quite a bit of information. Legitimate uses of this information include determining the exact configuration of the web server. Illegitimate uses of this tool include trying to determine a server's configuration so that an intruder can attack its weaknesses.

35.2.2.2. Example #2

In this example, netcat transfers files from one host to another. Suppose that host A is the one receiving the file. The following command on host A allows netcat to listen to traffic on an arbitrarily chosen port and receive a file:

 # nc -l -p 1234 | tar xvfp - 


Tip: In order for netcat to obtain a network socket, you have to run it with root privileges.

Now that host A is listening and ready to receive, you can send a file from host B using netcat on that system:

 # tar cfp - /some/dir | nc -w 3 hostA 1234 

35.2.2.3. Example #3

You can also use other applications instead of tar: for example, dd or cat. In the following example, netcat writes a complete backup of the /dev/hda1 partition to a file on a remote host. On host A, the receiving end, enter:

 # nc -l -p 1234 | dd of=backup_hda1 

On host B, the host containing the hda1 device to be copied, issue the following command:

 # dd if=/dev/hda1 | nc -w 3 hostA 1234 

You now know how to use the tools referred to in the LPI 202 Exam. You also understand the advanced networking concepts and troubleshooting tools assumed in the exam. Now, find a "live" Linux system and experiment with what you have found. Once you do that, you will be ready to pass this portion of the exam.




LPI Linux Certification in a Nutshell
LPI Linux Certification in a Nutshell (In a Nutshell (OReilly))
ISBN: 0596005288
EAN: 2147483647
Year: 2004
Pages: 257

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net