Hack 34 Historical Link State Monitoring

Comprehensively track the performance of your wireless links over time.

The monitoring tools mentioned so far give you a good instantaneous reading of received signal and noise levels for a given radio link. While useful for proving a link and installing your gear, remember that radio conditions change over time. Doing the occasional "spot check" doesn't really give you the full picture of what is going on.

For example, take a look at Figure 3-31. This displays radio data for a one-mile link, averaged over several days. You can see that in the middle of each day, the signal drops by as much as 6 dB, while the noise remains steady (remember that these are really negative numbers, so in this graph, a smaller number is better for signal). The repeating pattern we see indicates the effect of thermal fade. This particular link is a simple waveguide antenna mounted in the middle of a low sloping roof. As the roof (and the rest of the environment) heats up, the perceived signal is apparently less. At night, when things cool down, the perceived signal increases. The effect of thermal fade in this installation was later mitigated (by about 2 or 3 dB) by relocating the antenna, placing it closer to the edge of the roof. With less hot roof in the path, the effect of the day's heat was reduced. Nothing can completely eliminate the effects of the sun on a long distance path, but without a historical graph, it would be difficult to account for the effect at all.

Figure 3-31. This link shows a good deal of thermal fade in the middle of the day.

Figure 3-32 shows another interesting artifact, this time averaged over several weeks. This link is about a mile-and-a-half long, and still shows some of the effects of thermal fade. But look at the noise readings. They are all over the map, reading as high as -89 and jumping to well below -100. This most likely indicates the presence of other 2.4 GHz devices somewhere in the path between the two points. Since the noise remains steady for some time and then changes rapidly to another level, it is probably a device that uses channels rather than a frequency-hopping device. Given the wide variety of perceived noise, I would guess that the most likely culprit is a high-powered 2.4 GHz phone somewhere nearby. It is probably impossible to ever know for sure, but this data might warrant changing the radio to a different channel.

Figure 3-32. A link with a noisy environment.

These graphs were generated using Linux and some freely available tools. A modest Linux server can monitor a large number of radio devices, and the requirements on each of the APs is quite small. I will assume that you are using a DIY AP running Linux [Hack #51], although similar techniques can be used for just about any kind of radio device.

You can monitor signal strength using any available TCP port above 1024 on the APs. Port 10000 is usually a good candidate. On each of the APs to be monitored, as well as on the machine doing the monitoring, add lines like these to /etc/services:

mon0    10000/tcp mon1    10001/tcp mon2    10002/tcp

Then add a line like this to the /etc/inetd.conf on the monitored machines:

mon0  stream  tcp     nowait  nobody  /usr/local/bin/iwconfig iwconfig eth1

Be sure to use the path to iwconfig for your system, and specify the device to be monitored at the end. You can add as many iwconfig lines as you like, using a different port each time:

mon1  stream  tcp     nowait  nobody  /usr/local/bin/iwconfig iwconfig wlan0

If you need more ports, add more lines to your /etc/services. When all of your radio cards are set up, restart inetd. You can verify that the changes have taken effect by using telnet:

pebble:~# telnet localhost mon0  Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. eth1      IEEE 802.11-DS  ESSID:"NoCat"  Nickname:"pebble"           Mode:Managed  Frequency:2.457GHz  Access Point: 00:02:2D:1C:BC:CF           Bit Rate:11Mb/s   Tx-Power=15 dBm   Sensitivity:1/3             Retry limit:4   RTS thr:off   Fragment thr:off           Power Management:off           Link Quality:14/92  Signal level:-83 dBm  Noise level:-96 dBm           Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:6176330           Tx excessive retries:7880  Invalid misc:0   Missed beacon:0 Connection closed by foreign host. pebble:~#

To collect the radio data, you need netcat installed on the machine that does the collection. netcat is a free and insanely useful utility available at http://www.atstake.com/research/tools/network_utilities/. Think of it as a scriptable telnet program that will handle just about any kind of network data you want to throw at it. I use nc to connect to each of the machines, and scrape the results with a Perl script.

With the nc binary installed on your system, use this wrapper to collect the data:

#!/usr/bin/perl  -w use strict; my $ip = shift; my $port = shift; ($ip && $port) || die "Usage: $0 ip port\n"; open(NC, "nc $ip $port |") || die "Couldn't spawn nc: $!\n"; while(<NC>) {   if(/Signal level:-?(\d+).*Noise level:-?(\d+)/) {     print "$1 $2\n";     exit;   } } die "Warning: couldn't find signal and noise!\n";

I call it /usr/local/sbin/radio.pl, and invoke it with the IP address and port number on the command line. It simply returns two unsigned numbers representing the current signal and noise readings:

rob@florian:~$ radio.pl 10.15.6.1 mon0 83 96 rob@florian:~$

In case you haven't recognized it already, the tool that actually stores the data and draws the pretty graphs is Tobi Oetiker's excellent rrdtool. Like many powerful tools, rrdtool can be imposing at first (sometimes to the point of frustration). Efficient data collection and accurate visual presentation isn't as simple as you might think. Of course, rrdtool will handle just about any data type you track, but getting it working can be tricky the first time.

Personally, I prefer to use a tool like Cacti (http://www.raxnet.net/products/cacti/) to manage my rrdtool graphs for me. It is simple to install, has an easy to use web interface, and does the dirty work of managing your rrdtool databases. With Cacti installed, simply set up radio.pl as a data input script, and create a data source for each radio to be monitored. See the Cacti documentation for more details.

When properly configured, Cacti automatically generates Daily, Weekly, Monthly, and Yearly averages for all of your data sources.

If you prefer to roll your own rrdtool, then use a command like this to create your database:

/usr/local/rrdtool/bin/rrdtool create \  /home/rob/radio/radio.rrd \  --step 300 \  DS:signal:GAUGE:600:0:150 \  DS:noise:GAUGE:600:0:150 \  RRA:AVERAGE:0.5:1:600 \  RRA:AVERAGE:0.5:6:700 \  RRA:AVERAGE:0.5:24:775 \  RRA:AVERAGE:0.5:288:797 \  RRA:MAX:0.5:1:600 \  RRA:MAX:0.5:6:700 \  RRA:MAX:0.5:24:775 \  RRA:MAX:0.5:288:797

Be sure that the rrdtool binary is in your PATH, and change the print "$1 $2\n"; line in radio.pl to something like the following:

`rrdtool update /home/rob/radio/radio.rrd --template signal:noise N:$1:$2`;

Add a call to radio.pl to a cron job that runs every five minutes or so to regularly collect the data:

*/5 * * * *                /usr/local/sbin/radio.pl 10.15.6.1 mon0

Finally, once you have collected some data, you can generate a graph like Figure 3-32 with this command:

/usr/local/rrdtool/bin/rrdtool graph - \  --imgformat=PNG \  --start="-604800" \  --title=" Radio" \  --rigid \  --base=1000 \  --height=120 \  --width=500 \  --upper-limit=105 \  --lower-limit=60 \  --vertical-label="dBm" \  DEF:a="/home/rob/radio/ radio.rrd":signal:AVERAGE \  DEF:b="/home/rob/radio/ radio.rrd":noise:AVERAGE \  AREA:a#74C366:"Signal" \  LINE1:b#FF0000:"Noise" \  > graph.png

While the other tools mentioned in this chapter will give you a good instantaneous estimate of how well your link is doing, rrdtool, netcat, and this simple script will give you impressive historical data that can be invaluable for troubleshooting your network. In addition to signal and noise, rrdtool can also track uptime, network traffic, associated clients, transmission errors, and any other data you can think of. The reward of having clear historical graphs is well worth learning to deal with the complexity of rrdtool.