Section 10.2 Knights of the Realm (Forensics) | Real World Linux Security Prentice Hall Ptr Open Source Technology Series

10.2 Knights of the Realm (Forensics)

I came into my client's office after the New Year's holiday and had just started reading my e-mail and sipping my coffee when the Network Administrators came over to my cubic office space with worried looks. "Some of our systems were broken into over the holiday and we suspect that yours may have been one. They used an exploit in mountd that Red Hat 5.1 systems are vulnerable to. Your system is 5.1, isn't it?"

"Yes," I sighed. Sure enough, the telltale signs were there in the log file. See "Buffer Overflows or Stamping on Memory with gets()" on page 252 for a discussion on the exploit. Now the unpleasant work of decontaminating my system had to begin. It is important to note that this was a somewhat new and unknown vulnerability having been reported by CERT fewer than three months previously. I would have considered it unusual for a small unknown workstation to have suffered such a new and unknown exploit.

Of course, I was working very hard on a development project and security was not foremost on my mind. I did not follow the security mailing lists regularly, relying on word of mouth to alert me to problems.

See "The Seven Most Deadly Sins" on page 27. This is #7, Procrastination.

Normally I did not run mountd and NFS because I knew that they were security holes and I did not need them anyway. The server I was developing configured NFS, among many other things, so I had left it running unintentionally. Certainly, I should have operated it only when I needed it to reduce the window of vulnerability.

See "Turn Off NFS, mountd, and portmap" on page 98 and, of course, "Firewalls with IP Chains and DMZ" on page 514. It was irresponsible not to, at least, block Internet access to NFS, mountd, and portmap. Certainly, all three things were done right after the horse, I mean, break-in.

This was the only time in my quarter-century, ouch, of using UNIX and Linux that a system under my administration was cracked. It might not be the last. Although it took a bug in a program running as root, it also required errors on my part.

Fortunately I was good about backing up to tape every few weeks and keeping the server project's source changes in CVS on another system. Fortunately GNU tar (and my own highly customized spin-off of UNIX tar) have a feature to compare the contents of the disk with the tape. Typical invocation would be the following:

 tar -df /dev/rmt0 /

Typical output might include the following. Files that are identical between disk and tape are not listed unless the "v" modifier is included.

 /bin/date: Mode differs /bin/ls: Uid differs /bin/ls: Data differs /bin/su: Mod time differs /bin/su: Size differs /dev/hdb7: Device numbers changed /dev/null: Device numbers changed /dev/tty: Mode or device-type changed /lib/libc.so.6: Symlink differs

As you can see, GNU tar will detect any change at all in the file, including contents, modification time (the only time-stamp stored on tape), permission bits, type of file, UID, or GID. Unfortunately, GNU tar will not tell you how the item (UID, contents, etc.) has changed, requiring you to issue ls commands, extract files from the tape, and run the diff command on some files, and generally causing a lot of lost time.

It is recommended that you modify GNU tar open source is wonderful so that a second "v" will cause the differences in inode data (UID, GID, time, permissions, type) to be listed (and disable listing of identical files unless a third "v" is supplied). An -e extension could have differing files be extracted from tape with the extension appended to the name (and repeated if that file also exists on disk). For more details, see "Finding Cracker-Altered Files" on page 697.

So what did the crackers do to the system? They ran an IRC 'bot, left a few files under .mh, which had not existed previously, and in /tmp, and left the system undamaged.

Top