Hack76.Use a Serial Console for Centralized Access to Your Systems

Hack 76. Use a Serial Console for Centralized Access to Your Systems

Keep a secret backdoor handy for midnight emergencies.

Imagine the following scenario. It's 3 A.M., and you're the administrator on call. All of a sudden, you're jolted awake by the pager rattling itself off the side of your nightstand. A critical server isn't responding to network polls, and you're unable to SSH into it to determine what the problem is. You are now faced with a tough decisionno one wants to get dressed and head into the office at 3 A.M., but this server is essential to your company's online presence. What do you do? The good news is that with proper fore-sight and planning, you can avoid this kind of decision altogether with a console server.

A console server is a device to which you can connect the consoles of multiple systems. You can then connect to the console server to get easy access to any of those systems. Devices that enable you to connect multiple serial ports and quickly switch between them are readily available from many different vendors. A quick Google search for "serial console server" will list more potential vendors than you probably want to know about.

This hack explains how to configure your Linux systems so that they can use serial ports for console output rather than the traditional graphical displays that we're used to on Linux systems. Not only are serial consoles inexpensive compared to multiple graphical displays, but they are easy to access remotely and fast because there is no graphical overhead.

8.9.1. The Options

Before you rush off to implement a console server, you need to consider several options. Various commercial options are available that provide many different flavors of console server. However, the method we're going to discuss here is a bit more do-it-yourself, and can ultimately be much cheaper to implement than a commercial option.

Another option to explore is whether or not your hardware already supports serial port console access via the BIOS. If it does, this might all be a moot point for you. However, this kind of hardware support is fairly rare, so odds are you're going to have to decide between an expensive proprietary method, or an easy-to-implement open source method.

If you're still reading, it would appear that you've decided to go the easy open source route. Good for you! The first thing to keep in mind when designing your console server is its physical deployment. The server will need to be kept fairly close to your critical servers. It will also need to have one or more serial ports available. A variety of vendors provide multi-serial-port PCI cards, so find the one that seems to best suit your situation and stay with it. If you only need to connect to one or two devices, consider sticking with the onboard serial ports typically found on most servers.

8.9.2. Start at the Beginning: The Bootloader

We'll now begin the process of configuring the console client, which, confusingly enough, is your production server. We need to configure the bootloader to both send output and receive input via the serial port. This isn't as difficult as it might sound, so have no fear. Several bootloaders are available for Linux, but by far the most prevalent are GRUB and LILO. In this hack we'll cover setting up console access through GRUB. Though LILO is certainly an effective bootloader and is capable of performing the same functions as GRUB, it doesn't contain as many of the features that make GRUB an attractive choice for this application.

When we configure the bootloader to redirect system input and output, we're actually indirectly configuring the Linux kernel to redirect the system's I/O. These configurations are made by modifying the configuration files for GRUB, thereby changing the way that GRUB boots the Linux kernel. GRUB's configuration file can be found under the /etc (or sometimes /boot/grub) directory, and is aptly named grub.conf (on some distributions this file may be named menu.lst).

Before we dive into configuring the bootloader, let's take a moment to examine a typical grub.conf file:

 # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that #   all kernel and initrd paths are relative to /boot/, eg. #   root (hd0,0) #   kernel /vmlinuz-version ro root=/dev/hda3 #   initrd /initrd-version.img #boot=/dev/hda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Fedora Core (2.6.11-1.27_FC3) root (hd0,0) kernel /vmlinuz-2.6.11-1.27_FC3 ro root=LABEL=/ initrd /initrd-2.6.11-1.27_FC3.img 

Some people recommend removing the splashimage directives, because graphical images are not appropriate for serial consoles. However, I've never had an issue with this. Whether or not you'll need to remove these directives will largely depend on the version of GRUB you're using. If it's fairly recent, it should be able to ignore these lines without an issue. Otherwise, simply comment out or remove the splashimage reference.

Now that you have that worked out, let's modify the configuration file to redirect all input and output to the serial port. The standard settings for serial port communications are 9,600 baud, no parity, and 8 bits. It's important to remember these settings, as they will become necessary later when you need to configure the console server to communicate with the client. To pass these settings on to the kernel, add the following lines to the top of your grub.conf file.

 serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1 terminal --timeout=30 serial console 

These lines should appear directly above your default directive. Most of the flags passed are self-explanatory, but let's look at the ones that may not be so clear. The --unit flag tells the kernel to redirect everything to the first serial port the kernel can identify. This will remain the same boot after boot, so you don't need to worry about it changing. The --word directive is used to set the number of bits that are to be used in communications with the console server. This can be set to 5, 6, 7,or 8. Take note that almost everything communicates using either 7 or 8 bits, and 8 bit is a nearly industry-wide standard. Using a lower number for this option can end up coming back to bite you should ASCII values greater than 127 be displayed. The --parity flag is used in this case to disable the use of parity error checking of the data transmitted from one end of the null modem connection to the other. The -stop directive is used to set the method by which a null modem data transmission is terminated.

The second line we added instructs the kernel to use both the serial port and the console to display output. Whichever one receives input first becomes the default console. Herein lies one of the above-mentioned features that makes GRUB an excellent choice for this project. By specifying both the serial and console options, we are able to effectively utilize two different devices as the console.

Once you've made the changes, reboot the server so the new GRUB directives can take effect. We'll now move on to configuring your console server to communicate with the client.

8.9.3. Putting It All Together

First, you need to make sure you have a serial cable connecting your console server to the client. (Be sure to connect the serial cable to the same port you configured the kernel to redirect I/O to, or you'll end up staring at a whole lot of nothing!) Then you'll need a program to communicate via the null modem. There are several available, but for overall ease of use and maximum features, I recommend minicom. To start minicom, run the following command as root:

 # minicom -s 

This will bring you directly into the minicom configuration screen. Select "Serial Port Setup" and change the configuration to match what you set up earlier in GRUB. Once that's done, save your changes and exit. You should be taken to the main minicom screen. If you don't see anything, hit Return once, and you should be greeted with the login prompt for your server. Congratulationsas long as your network remains alive, you now have remote console access to your server!

For more information about installing and configuring minicom, see "Turn Your Laptop into a Makeshift Console" [Hack #37].

Once everything is assembled and minicom is installed, all you have to do it to SSH to your console server, start minicom with the correct serial device, and access the console of your troubled system. Voilà! What could be easier?

8.9.4. Where to Go from Here

What we've created here is the most basic application of a console server. As I mentioned earlier, this can be expanded with the addition of a multi-port serial card. With a little extra time spent cabling and configuring your bootloaders, you can effectively deploy console servers across your entire network. Another trick to keep in mind is the deceptively simple RJ45 to DB9 serial adaptor. These little guys allow you to use a strand of cat5 network cable to connect to a serial port. I have actually used them in conjunction with patch panels to provide myself console access to network equipment from my desk. You can pick up one of these lifesaving gadgets from any networking supply company for under a few dollars.

Another way to increase the usefulness of your console server is to include a modem attached to the console server in your setup. This can be configured to accept incoming calls, thereby allowing you to connect to your console server over the phone line in the event of a network outage. I would highly recommend this, as the remote console server does you no good if your problem lies somewhere else on the network.

You'll only need to use a serial console server once to prove to yourself that it's well worth the half-hour or so it might take you to get it configured. At 3 A.M., when you're able to reboot a server remotely and bring it back online without so much as putting on a pair of pants, you'll agree that the foresight for such an occasion is priceless.

8.9.5. See Also

• http://www.linuxjournal.com/article/7206

Brian Warshawsky