The Methodical Approach and the Need for a Methodology

Oh no you saynot more management speak! Please, I get enough of that already! Fear not; we promise that we won't waste your time with YAUM (Yet Another Useless Methodology). We want you to find your problem and fix it quickly. So you can call this a process, a method, a way, or if you like, call it a methodologywhatever works for you. What we don't want to do is fill your head with some useless babble. This methodology is hard won from years of solving problems.

It all started many years ago. Painted on the wall of our parents' garage was a slogan: "1^st law of wrenchin': always check the spark plugs!" Scott painted this on the wall as a reminder of a valuable lesson we learned as teenagers. You need a plan for fixing things, and you need some "laws" to make sure you don't waste a lot of time figuring out what your problem is when it's staring you in the face.

As it was, we had cars, as many young men do in high school, which were the primary focus of our budding engineer minds. They were complex, they were fun, they took us places to have fun, and they were essential parts of teenagers' social lives. Without a car, life was a little less fun, and things were much farther away. However, our cars were used and needed worksometimes a lot of workto keep running. So we learned to take care of our cars, to fix them when they broke down, and to modify them to make them better. After all, what kind of engineer is happy with something as it is out of the box? Everything could be a little better, couldn't it? All of this meant that we had to figure out what was wrong with them while not wasting too much time doing so. We paid for those muscle cars, and we wanted to drive them! Enter the first law of wrenchin'.

The first law of wrenchin' was the result of a marathon session of trying to figure out what was wrong with one of our cars (Mike had a 1972 Pontiac GTO; Scott still has a 1970 Oldsmobile 442). For days we struggled to figure out why the motor wasn't performing as it should. We changed the timing, retuned the carburetor, drained the gas tank, uncorked the headers, and so on. We did everything we could think of...except check the spark plugs. That was too easy! It couldn't be the spark plugs, we told ourselves. Five minutes after we stopped convincing ourselves it wasn't the spark plugs, we pulled them out. They were covered with soot. The result of all our other twiddling had made already fouled plugs an absolute mess. In short, the problem was absolutethere was no way we could have gotten that GTO to run correctly, and the only solution was possibly the simplest and easiest to execute. After we had that figured out and had replaced the spark plugs, a five-minute job, we had that 1972 GTO running perfectly. What was really galling about this experience was that we knew better! Of course it could be the spark plugs; we just didn't check them, even though all the signs that this was the root cause were staring us in the face, as it were. The idea that the spark plugs were the cause of the problem just seemed too simple. We thought, wrongly, that it had to be something more complex.

Back then, our problem was caused by the lack of a clear system for problem solving or a mechanism for working through the problem to rule things out before leaping to what we might have thought was the root cause. We needed to work our way up the motor, ruling things out before moving on to the next component, and we had to learn to start with the simple things first. From that moment on, we decided that we wanted to spend more time using our cars as opposed to fixing them. So the journey started toward putting a plan in place to help prevent those forays into endlessly searching for the root cause of a problem. Over the years, we learned to work through the problem carefully, and we discovered that other people had come up with well thought out means of troubleshooting as well. It is that hard won experience and knowledge that we present here to help you to troubleshoot more effectively.

With Linux firewalls, this need for specific steps and a clearly defined approach is no different. There are so many variables in a modern firewall that it's easy to overlook the real source of any problem. For instance, we once had a big high-end SMP DEC Alpha Linux box that we used as a high-end packet switch and firewall. It worked perfectly in our lab in Virginia. We shipped it to our ISP, and all the network cards simply stopped working. We were puzzled, so we got on the box from a console and found that the network cards were up, that they had IPs, but that they simply could not talk to anything.

The NIC driver module was loaded, and the machine had worked without any problem for weeks on our lab's network. The co-location facility's network should have been no different, except for one small change: The ISP's network was Ethernet, and our lab was FastEthernet. We didn't expect that this would create any problems. After all, the network cards were designed to support both types of network, Ethernet and FastEthernet, and we had worked with these drivers before on both mediums. Our network was faster, and all we did was put the box on a slower network. Stepping down, as it were, shouldn't have made any difference. We were flummoxed. Why would that seemingly insignificant difference cause the cards to stop working?

The simple reason was that we were running the wrong driver. This might seem obvious now, but the cards we were using were DEC cards, and the wrong drivers would load and work with the wrong card, and they would work on a FastEthernet network! In our case, the system thought the card was Tulip FastEthernet card, when the cards were really DE500s. To make matters worse, they even look the same.

So back in our lab, the Tulip driver worked fine at FastEthernet speeds and never once complained about the fact that the card was not a Tulip. Couple this with the fact that we only stocked DEC NIC cards, Tulips, and DE500s, and you can see how this mistake was made. They look the same, and even the OS thought they were the same when it detected them. Fortunately, we had a simple process that we will illustrate here to work through this problem quickly, and we had the system up in short order. Total time to debug this problem and fix it was under five minutes.

The point here is that the best and quickest way to arrive at the root cause of your firewall problem is through the application of a well thought out process of elimination. It's very tempting sometimes to leap to a conclusion about what's causing the problem with your firewall, but unless you're lucky, you can actually make the problem worse. Take our word for it...relax, sit down, and work through the problem step by step. Stick with the approach we have outlined in this book, and you'll soon be tackling the thorniest firewall problems like a guru!