Section 19.12 Finding Cracker-Altered Files

   


19.12 Finding Cracker-Altered Files

Now, it is time to analyze your file system for cracker-installed differences. First, run fsck on the compromised file system, because you brought the system down abruptly and on the possibility that the cracker corrupted it accidentally or intentionally. This author has found tar backups to be excellent for most applications. Certainly, if you do backups of running databases, you are using software that is part of the database package and you will have to consult its documentation. Some of these programs probably generate normal files that you then can back up with tar or equivalent.

The rest of this section assumes that GNU tar backups of your system have been done. If you use a different backup scheme, refer to that product's documentation for comparing tape to disk; not all offer a useful way to do this. If you have tar-format backups, you can use GNU tar's -d feature to show the differences between your tape backups and your disk. It is assumed that you created the tape via

 
 cd / tar -cf /dev/rmt0 . 

or similar and that the compromised file system now is mounted as /mnt2 on the trusted system.

Be sure that your backup media is set Read/Only so that you do not accidentally write over it if you give the wrong flag to tar. It is very important to remember that the crackers might first have gotten control of your system a long time ago and that recent backups might be compromised, so you also might want to compare the disk to older backups.

Trap Door Warning

Many crackers will place their Trojan horse programs where root might execute them accidentally. /tmp, /, /root, /usr/tmp, and /var/tmp are such places, so do a

 

[View full width]

/bin/ls -Fa /mnt2/tmp /mnt2 /mnt2/root \ /mnt2/usr/tmp /mnt2/var/ graphics/ccc.giftmp

to look for any Trojan horses.


Ensure that "." is not in your $PATH search path or at least at the end of your $PATH search path. This will protect against an intruder placing a Trojan horse in /tmp that might be called /tmp/ls or /tmp/rm and waiting for you to do cd /tmp and invoke some common program such as ls or rm (say, to clean up /tmp). This would cause the Trojan horse to be invoked as root. Rename any such Trojan horses to something obvious, for example,

 
 mv /mnt2/tmp/ls /mnt2/tmp/lsCRACKED 

or

 
 mv /mnt2/bin/tar /mnt2/bin/tarCRACKED 

and chmod it to 0 so that it cannot be executed.

The tar -d flag is used to cause tar to note the differences between the tape backup and the respective files on disk. This -d flag is used in place of the -c that creates a tape or -x that extracts from a tape. Other tar modifiers, such as -v for verbose or -f to specify the tape device, also may be used.

If your tar backup is on /dev/rmt0 (raw magnetic tape number 0) and your compromised file system is mounted as /mnt2, do the following from csh:

 
 cd /mnt2 touch scanned sleep 2 /bin/tar -df /dev/rmt0 . >&! diffs 

or the following from sh (bash):

 
 cd /mnt2 touch scanned sleep 2 /bin/tar -df /dev/rmt0 . > diffs 2>&1 

Although the diffs file will be on the compromised file system, only the existing data on this possibly compromised file system is compromised; all new data created by your trusted kernel using trusted utilities (from the trusted boot floppy) can be trusted. You put the diffs file here because if you booted from floppies, there will be very little space left in the RAM disk that was created during booting. (Note that this Trojan scanning technique can be done without additional hardware and without relying on the system's normal root file system that might have been compromised anywhere.)

In the diffs file there will be a dialogue of all of the files on tape whose disk copies are different or missing. Expect plenty of differences in /tmp, /usr/tmp, /var/tmp, /var/spool/mail, users' directories, /home/*/.netscape, etc. Although you should expect some files to be deleted (from the disk copies), you will need to check for permission changes and differing contents that might be Trojan horses. Some of those deleted files may be the work of the crackers; you will need to determine this from studying the backups, talking with your users, etc.

There should be no differences in /bin, /sbin, etc. You should expect differences in /etc/passwd, /etc/group, /etc/shadow (if used), /etc/hosts, /etc/sendmail.cf, etc. These differences must be checked very carefully by extracting these files from tape to somewhere else and comparing with the latest "disk" versions using the diff program.

Recall that presently your current working directory is /mnt2. Issue the following commands to extract the "tape" versions of files in /etc and /var/spool/mail:

 
 mkdir tape chmod 755 tape cd tape /bin/tar -xf /dev/rmt0 etc var/spool/mail 

These files each can be checked for alterations via the following:

 
 diff etc/passwd ../etc      | more diff etc/group ../etc       | more diff etc/shadow ../etc      | more diff etc/hosts ../etc       | more diff etc/sendmail.cf ../etc | more etc. 

19.12.1 Interpreting tar -d's Output

In the previous chapter, you explored how to get a list of files on disk that have changed since the tape backup or which have been removed. Here, the interpretation of the tar -d command is discussed. Typical output might include:

 
 /bin/date: Mode differs /bin/ls: Uid differs /bin/su: Mod time differs /bin/su: Size differs /dev/hdb7: Device numbers changed /dev/null: Device numbers changed /dev/tty: Mode or device-type changed /lib/libc.so.6: Symlink differs 

As you can see, GNU tar will detect any change at all in the file, including contents, modification time (the only time-stamp stored on tape), permission bits, type of file, UID, or GID. Unfortunately, GNU tar will not tell you exactly what changed, requiring you to issue ls commands, extract files from the tape and run the diff command on some files, and generally cause you to waste a lot of time.

I recommend that you modify GNU tar (open source is wonderful) so that a new flag will cause the differences in inode data (UID, GID, time, permissions, type) to be listed. An -e extension could be added to have differing files be extracted from tape with extension appended to the name (and repeated if that file also exists on disk).

You marked the time that you started reading these files with the touch scanned command and waited more than a second, using the sleep 2 command. You now use the find command to find all files that have not been read since this time via the following commands. (Note that for the access time to be updated, the /mnt2 file system must not be mounted Read/Only when the tar -d command was invoked.)

Issue the csh commands:

 
 cd /mnt2 find . ! -anewer scanned -print >&! added 

or the sh commands

 
 cd /mnt2 find . ! -anewer scanned -print > added 2>&1 

You now can do

 
 more diffs added 

to study what the cracker has done.

A cracker can hide his files anywhere on the system, but a popular place is in /dev. They are easy to find in that location because there should be only devices and directories there, with only the ordinary files MAKEDEV* and, on some distributions, README*. Cracker files may be found there very efficiently with

 
 find /dev -type f -ls 

This find command could be placed in root's crontab for daily execution to detect whether any crackers install their files there. To avoid being bothered with files that should be there, have find not tell you about expected files. Because the output of a crontab entry is mailed to the account automatically, you do not need to pipe the output to a mailer. The following will work for Red Hat and many other distributions:

 
 find /dev -type f ! -name MAKEDEV -ls -exec file '{}' ';' 

19.12.2 Speeding Up the Check with RPM

RPM, Red Hat's Package Manager, can be used to verify all files that were created from the Red Hat packages. It uses a database that stores the ownership, permission, and MD5 checksum for each file in each package. It is a self-contained program, which allows this checking even on a system where anything might have been compromised. This is because only a few files need to be checked for validity. Assume that you booted from the rescue disks, that they contain the md5sum program, and that our normal root file system is mounted as /mnt2.

Note that since the rescue root file system is copied to the created RAMDISK, a

 
 mkdir /mnt2 

command or any other command applied to the RAMDISK will not be remembered across a reboot. RPM's -Va flag will cause it to verify the correctness of all installed packages, based on its databases. First, you need to validate the rpm program itself and its database. Operating from your rescue disks, the following command will generate the MD5 checksums for these files that may be compared to their correct values that can be determined from backup (or which you wrote down previously):

 
 md5sum /mnt2/bin/rpm /mnt2/var/lib/rpm/* 

A more sophisticated solution is to store the checksums in a file and make use of md5sum's -c flag to compare the checksums stored in this file to those of the target files on disk automatically. This may be done with this command:

 
 md5sum -c /rpm.md5 

If you were exceptionally well prepared and followed the advice in "Store the RPM Database Checksums" on page 599, you have a tar-format floppy with this file. If so, pop the floppy in the drive and issue the command

 
 tar -xvf /dev/fd0 /rpm.md5 

If you did not generate these checksums prior to being compromised, restore bin/rpm and var/lib/rpm/* and generate the checksums.

Now that you have proven the rpm program and its database trustworthy, from the operating rescue disks, issue the following command to validate your system:

 
 /mnt2/bin/rpm -Va --root /mnt2 

Certainly, many of the configuration files will have changed so you will need to inspect them visually or "diff" them against your backups. You might want to save rpm's output to the RAMDISK or floppy drive for more careful analysis.

19.12.3 RPM Repairs

Repairing (restoring) the damaged packages can be as easy as using rpm's -F flag to "freshen" the on-disk versions of packages. In the example from the previous section, you were booted from the rescue disk floppies and had mounted your damaged hard disk root file system on /mnt2. If you then insert your distribution CD-ROM and mount it on /mnt/cdrom, the following command will repair any cracker-induced damage to the packages:

 
 /mnt2/bin/rpm -Fa --root /mnt2 \    /mnt/cdrom/RedHat/RPMS/*.rpm 

You will need to correct any damaged configuration files manually, because RPM usually will not alter the existing ones. RPM does this in order to not undo your normal changes to the configuration files. This invocation also can be used to update buggy packages, especially those with security bugs, over "the net" from the distribution provider. To install packages over the net, simply provide a URL specifying the desired RPM as either an http-style or ftp-style URL.

For example, if you have a Red Hat installation and want to upgrade your named program (BIND) to correct that security bug, the following command will do this in one step:

 
 rpm -vhU \    ftp://updates.redhat.com/9.7/i386/bind-12.2.2.i386.rpm 

If you are reinstalling the same version of a package that already was installed on your system but which has become corrupted, possibly due to a cracker-installed Trojan, you need to add the --force flag after the regular flags. This is because RPM is smart enough to not re-install the same version of an already installed package. Thus, if a cracker subsequently broke in via another avenue and then overwrote your secure version of named with an earlier insecure version, you would issue the following command:

 
 rpm -vhU --force \    ftp://updates.redhat.com/9.7/i386/bind-12.2.2.i386.rpm 

19.12.4 Recovering Databases

From a SysAdmin perspective, databases are large binary files that change frequently over time. Most other files on Linux either are programs that do not change over time or are ASCII text files whose contents easily may be inspected and compared to previous versions via diff. For Web commerce sites and other systems that have databases which a cracker might attack, advance preparation needs to be made. One valuable technique is the use of audit trails. With preparation, you or, hopefully, the Database Administrator, can start with the database in a known "good" configuration.

The database's audit trail will need to be analyzed for the cracker's transactions. You should concentrate on parts of the audit trail around the time of the suspected break-in. Look for suspicious entries such as changes of address or purchasing patterns that are different from before. A database search for new accounts and recent changes of address would be helpful.

These transactions should be stopped pending investigation that may include phone calls or e-mail to the customer. After the audit trail has been cleaned up, one recovery would be to apply the audit trail's valid transactions to the known "good" database. On a busy site this could take a whole lot of effort, so you may want to call in other people in your company who are involved with ordering.

Understand that a database is just another file and that a cracker can alter times of transactions and otherwise obfuscate his actions, including directly altering the binary database file. Your Database Administrator may want to "unload" the current database into ASCII files and do the same for a restored known "good" version and have you run diff between these, looking for the mark of the cracker.

In the case of corruption, the Database Administrator might need to "unload" the current database (or a recent backup) into ASCII files. He then may need to correct bogus entries and then "load" the database from scratch. This would be a good exercise for the fire drills discussed in "Fire Drills" on page 582. After taking a full backup, you might want to have one Database Administrator alter the database either by adding bogus entries or actually corrupting it, perhaps altering the capitalization of some data in a harmless way. You will want to ensure adequate resources to allow this unload/analyze/load procedure, particularly disk space.

19.12.5 Peripheral Damage

Be aware that any peripherals attached to the compromised system can also be altered. This includes any tapes in the drive or juke box intended for backups, any writable floppy in the drive that might be intended for "secure" booting, and CD-RWs in the burner.

If you have bank check forms in a printer, these could have false information. Clearly, your system might have printed checks totaling thousands or millions of dollars to a cracker's phony company. In this case, closing the associated bank accounts and stopping your outgoing mail and contacting the post office might be necessary. If you can isolate which check numbers might be suspect, putting a "stop payment" against these might be sufficient but riskier than closing the account. If the cracker knows your account number and bank name, he can print phony bank checks on his own system. This is reason enough to close the associated bank accounts!

If you have purchase order forms in a printer, the cracker might have generated purchase orders for merchandise with a shipping address of the cracker's post office box or drop point.

19.12.6 Theft via Evil Electrons

The cracker even might have placed fraudulent orders via e-mail or the Web from your system. He might have used credit card numbers found on your system, either your corporate card numbers (possibly found in e-mail) or those of your customers. Theft of credit card numbers is big business. Some well-publicized credit card number thefts recently happened to thousands of customers of America Online due to a breach in AOL's security, more than 300,000 customers of CD Universe due to a bug in their Microsoft Web server software, and thousands of customers of Pacific Telephone.

A cracker simply might have relied on your company's good credit. He might have started a Web browser running on your system with the DISPLAY environment variable pointing to his system. The opportunities and methods are vast once your system has been compromised.

19.12.7 How the Kernel Can Be Damaged

Besides damage to the file system, crackers can alter data in the running kernel. This can be things such as the system's IP address, network routes, UID, etc., of running processes, and what processes are running. Most of these things are trivial to alter if the cracker has made himself root. These changes can allow Denial of Service attacks and also can control what other systems your system trusts. For example, modifying the running NFS server can allow the cracker's system to have full control of your entire file system.

A very sophisticated cracker can modify the system to run "secret" processes that a ps command will not show. This can be done with simple modifications to either the kernel or ps. One of his processes could be listening on an otherwise unused port for more cracker commands.


       
    Top


    Real World Linux Security Prentice Hall Ptr Open Source Technology Series
    Real World Linux Security Prentice Hall Ptr Open Source Technology Series
    ISBN: N/A
    EAN: N/A
    Year: 2002
    Pages: 260

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net