Section 32.5. Objective 5: Troubleshooting System Resources

32.5. Objective 5: Troubleshooting System Resources

Sometimes the problems we need to troubleshoot aren't with hardware or with specific applications; the problems are with the system itself. A console message of a garbled terminal console or a wrong user home directory means there is something wrong with the system and not with an application per se.

32.5.1. Environment Variables and Shells

Many applications use variables to modify their behavior. Some variables have local scope (meaning they are visible only in the current instance of the shell), while others have global scope (meaning they are visible in the current shell and in child shells). These latter variables are called environment variables. A local variable can be made an environment variable via the export command.

To view your current variables and their settings, issue the set or env command.

Some common variables are shown in table Table 32-2.

Table 32-2. Common Variables Environment

Variable	Purpose
EDITOR	Determines which editor is invoked by programs requiring user interaction
LANG	Determines which language to use (e.g., en_US.UTF-8, en_US, etc.)
PAGER	Determines which program to use to page through files (e.g., more, less, etc.)
PATH	Colon-separated list of directories to search for executable file names
PS1	Character(s) to use for the command prompt
SHELL	Current shell
TERM	Determines which terminal type is in use (e.g., vt100, linux, etc.)
UID	Current user ID
USER	Current username

While bash is the most popular login shell on most Linux systems, there are quite a few other shells in use. The file /etc/shells lists the installed shells:

 /bin/sh /bin/bash /sbin/nologin/bin/ash /bin/bsh/bin/ksh /usr/bin/ksh/usr/bin/pdksh/bin/tcsh /bin/csh /bin/zsh 

Technically, /sbin/nologin is not a shell; it is used to stop users from being able to log in (/bin/false is used on some systems for the same purpose). Why would you not want a user to log in? Not all users are people; many applications set up a user account that does not need to log in but is required to run the application; one example is the user apache that runs the httpd daemon.

32.5.2. Shell Environment

Even if your users log in graphically and never see a command line, they are still running inside of a shell . The shell is configured via environment variables set in configuration files. There are two types of configuration files: global and local. Generally, when the shell starts, it will read a global configuration file and then read a local configuration file. Normally the settings in the local configuration file override the settings from the global configuration file.

The global configuration that most shells read is /etc/profile. Variables set here apply to every user on the system, although each user can override them; PATH and HOSTNAME are often set in /etc/profile. Since many kinds of shells will be reading /etc/profile, the syntax of the commands has to be Bourne shell-specific, e.g. you should never put a zsh-style command in /etc/profile because it will cause an error when the bash shell attempts to read /etc/profile.

After the global configuration file is read, the local (usually shell-specific) configuration files are read. The local files are located in the user's home directory. We will discuss the bash-related configuration files since it is the most popular shell on Linux systems.

bash can be invoked in several different ways: as a login shell, an interactive nonlogin shell, a noninteractive shell, or a Bourne shell. How it is invoked determines which files get read. If bash is invoked as a login shell (e.g., the login command launches it), bash reads the following files in order: ~/.bash_profile, ~/.bash_login, and ~/.profile. If bash is launched as an interactive, non-login shell (e.g., from the command line or from a application such as KDE konsole or Gnome Terminal), bash reads just the ~/.bashrc file. If bash is run non-interactively (e.g., to run a script), bash reads the environment variable $BASH_ENV to determine which file to read. Finally, if bash is run as sh, it reads the filename listed in the $ENV variable, if it is defined. Of course, this being Linux, there are multiple command-line options to bash that change each of these behaviors. Fortunately, they are not on the test.

So when the user complains that the terminal is acting "funny," you need to keep in mind which shell she is using, how the shell was invoked, and which configuration files, if any, are being read.

32.5.3. Editors

Many editors are available in the Linux/Unix world. However, this world falls into two general camps when it comes to editors: you are either a vi user or an Emacs user.

vi is a difficult editor to get used to, especially if you're used to dealing with modern-day word processors. It has been around longer than Emacs, and it shows. Since it was invented in the days before arrow keys, 64 MB of RAM, and other pleasantries, vi requires very little in the way of system resources, doesn't require arrow keys to navigate, and lets you edit very efficiently. There are clones of vi, such as popular VI Improved (http://www.vim.org), that add many nice features such as block copy/edit and integrated help, but you still need to know how to edit using basic vi.

Emacs , on the other hand, is a large, resource-hungry editor that can do the most amazing things. Through its built-in ELISP programming language, Emacs can be coerced into doing things such as fetching and reading newsgroups and running games. While you can do just about anything you want within vi, you generally invoke more complex functions by running external programs within the editor.

Whichever one you choose is based on what you like, but as a system administrator, you must be comfortable with vi. vi is ubiquitous; it is installed on every Unix/Linux machine and even system rescue disks. Since vi has been around for so long, it can run on machines with few resources, it can run from a floppy, and during system recovery sessions, it will be the only editor available to you.

32.5.4. Kernel Parameters

One of the more interesting features of Linux is that you can modify the running kernel not only by adding and removing modules, but also by changing the parameters of the kernel itself. Of course, you can't change everything about the kernel, but you can change a number of parameters. Examples range as far as whether the system uses IP Forwarding, the minimum and maximum speed of your RAID drives, and whether to ignore ARP requests.

Assume our Linux box is set up as a router with two network interface cards and that it has stopped routing packets for reasons unknown to us. One of the requirements for a Linux router is the ability to forward packets from one NIC to another. This ability is turned off by default. How would we troubleshoot this?

There are two ways of doing this research: via the /proc filesystem or the sysctl command. The sysctl command is a front-end for /proc. You must have procfs compiled in your kernel for this to work.

Most of the entries in /proc are read-only, but a few are read/write. Even though the entries look like files, the only way to read them is by cat-ing the "file":

 # cat /proc/sys/net/ipv4/ip_forward 0 

This shows us that IP Forwarding is turned off. We can turn it on by writing to the "file":

 # echo "1" > /proc/sys/net/ipv4/ip_forward # cat /proc/sys/net/ipv4/ip_forward 1 

The value is good only until the next reboot, so to keep this setting, you would place the echo command in rc.local and it will get run at every reboot.

How did we know to change the entry /proc/sys/net/ipv4/ip_forward? How you know which /proc entries are writable? One way is to traverse the entire /proc directory looking for files with write permissions.

An easier way to do all this is via the sysctl command. Typing sysctl -a or sysctl -A lists all of the changeable entries:

 # sysctl -a | more sunrpc.tcp_slot_table_entries = 16 sunrpc.udp_slot_table_entries = 16 sunrpc.nlm_debug = 0 sunrpc.nfsd_debug = 0 sunrpc.nfs_debug = 0 sunrpc.rpc_debug = 0 dev.parport.parport0.devices.lp.timeslice = 200 dev.parport.parport0.devices.active = none dev.parport.parport0.modes = PCSPP,TRISTATE dev.parport.parport0.dma = -1 --More-- 

If you go far enough down the list, you'll see the ip_forward entry we modified earlier:

 net.ipv4.ip_forward = 1 

Using sysctl to change or view the value, you would issue:

 # sysctl net.ipv4.ip_forward=0 # sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 0 

Just as in the previous method, the value set with sysctl is good only until the next boot. If you want to make the change permanent, place the key = value line in the file /etc/sysctl.conf.