Section 41.6. Determining the Cause of Inaccessible Services | LPI Linux Certification in a Nutshell (In a Nutshell (OReilly))

41.6. Determining the Cause of Inaccessible Services

To confirm that a problem exists with a service, any number of commands can be used. netstat is very useful, but so are applications such as nmap, lsof, and strace. You can also inspect your system's configuration files to see whether unexpected changes have been made to your configuration.

41.6.1. Using netstat

While netstat is useful, it does not always reveal all information concerning a service. Review the following netstat output:

 # netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address     Foreign Address    State tcp   0      0      0.0.0.0:6000      0.0.0.0:*          LISTEN tcp   0      0      127.0.0.1:53      0.0.0.0:*          LISTEN tcp   0      0      0.0.0.0:22        0.0.0.0:*          LISTEN tcp   0      0      10.0.0.5:34386    80.232.36.131:22   ESTABLISHED tcp   0      0      10.0.0.5:35191    80.232.36.131:993  ESTABLISHED tcp   0      0      10.0.0.5:35190    80.232.36.131:993  ESTABLISHED tcp   0      0      10.0.0.5:35189    80.232.36.131:993  ESTABLISHED tcp   0      0      10.0.0.5:35188    80.232.36.131:993  ESTABLISHED tcp   0      0      10.0.0.5:34387    10.0.0.1:22        ESTABLISHED tcp   0      0      10.0.0.5:35193    10.0.0.4:993       ESTABLISHED tcp   0      0      10.0.0.5:35196    10.0.0.4:993       ESTABLISHED tcp   0      0      10.0.0.5:35187    10.0.0.4:993       ESTABLISHED udp   0      0      0.0.0.0:32779     0.0.0.0:* udp   0      0      127.0.0.1:53      0.0.0.0:*

The first connection shown indicates that this host presumably has an X server running, because X uses port 6000. The second connection shows that a DNS server is presumably listening on port 53, ready for a zone transfer. The next lines show port 22 open, so there is likely an SSH server open, as well.

Notice that we are not authoritatively saying that X, DNS, and SSH are listening on these ports. This is because netstat does not actually analyze the connections. Without additional arguments, netstat simply listens for an open port. It can also map that port to a name in the /etc/services file, but the preceding output didn't give the names.

41.6.2. nmap

The nmap command is no better in this case. While it can provide you with a useful listing of open ports, it will not give you details about the actual service that opened that port. nmap can also determine the type of operating system being scanned, but it cannot inform you what actual service opened the port.

41.6.3. Finding Authoritative Information with lsof and strace

While you can specify extra options to netstat to gain additional information, using lsof will provide you with extensive information about the actual service listening on the port. This command actually names names, as it were, providing you with detailed information about the daemon and user that opened the port. You will also receive information such as the daemon's PID, as well as connection details and the system call information that you normally would not see.

Another invaluable command for troubleshooting network problems is the strace command. By running an application or daemon under strace, you can read exactly what the application was doing at the time of failure. From this output, you can then begin to identify the problem and formulate a solution.

41.6.4. Unexpected Changes to Files and Settings

Review the /etc/hosts.allow and /etc/hosts.deny files and make sure that they are unchanged. It is possible that a service or another administrator updated these files. One way to verify whether a system is configured to use TCP wrappers is to review the /etc/inetd.conf file or the /etc/xinet.d directory. Look for references to tcpd. If you see a reference, this daemon is protected.

You can usually tell the difference between a daemon that is simply not responding and one that is protected by TCP wrappers. If a daemon is protected by TCP wrappers, the connection will hang for a brief period of time, then fail. This short period of time reflects the amount of time it takes for tcpd to analyze its settings and respond to the connection. If the connection is immediately refused, the daemon is simply not responding.

Also, review the iptables or ipchains settings. Make sure that a rule has not been added by an application such as Portsentry or by another administrator. Use the iptables -L command, for example, to discover settings.

Tip: Remember that iptables and ipchains settings apply to all network connections, not just those protected by TCP wrappers.

Failure to consult such files and features may cause you to take much more time solving a problem. The writers of the LPI Exams know that less experienced administrators fall into this trap.