Monitoring AIDE for Bad Things

Okay, so you have this shiny new filesystem integrity-checking tool all set up and running. But now what? Now you sit and wait for something to happen. Usually nothing does, and even when it appears that something bad might have happened, many times it hasn't.

AIDE will continue to monitor the filesystem according to the rules you configured. Thanks to the cron job, you'll receive reports nightly containing the files and the attributes for those files that have changed since the database was initialized or last updated. Many times these changes will be completely benign. Recall the example from the beginning of the chapter. If you add a user, files such as /etc/passwd and /etc/shadow will change. AIDE will notice and report accordingly, assuming that you're checking /etc. However, if you didn't add a user or make other changes to /etc/passwd or /etc/shadow, it might bear closer examination to make sure that an attacker hasn't altered either one of those important files.

Of course, there are other files that AIDE will be reporting on. You should closely monitor the AIDE report for files that were altered unexpectedly. For example, the files /bin/su or /usr/bin/passwd should rarely be altered and then only by certain known software updates. Therefore, if a file such as /bin/su shows up in an AIDE report, you need to look into it immediately to see when and why that file changed. Taking this example one step further, assume for a moment that AIDE ran overnight through its normal process. In the morning you awake to find an email containing some of the following lines:

 AIDE found differences between database and filesystem!! Start timestamp: 2004-03-09 20:24:33 Summary: Total number of files=1488,added files=0,removed files=0,changed files=1 Changed files: changed:/bin/su Detailed information about changes: File: /bin/su  Size   : 23176               , 0  Permissions: -rwsr-xr-x            , -rwxr-xr-x  Mtime  : 2002-04-07 10:59:14        , 2004-03-09 20:24:31  Ctime  : 2002-04-08 20:45:36        , 2004-03-09 20:24:31  Inode  : 98564               , 100922  MD5   : XTJK0hKy/492djesGoBx7A==     , 1B2M2Y8AsgTpgAmY7PhCfg== 

The AIDE check that you scheduled found something. You can quickly tell from the summary line what has been found:

 Summary: Total number of files=1488,added files=0,removed files=0,changed files=1 

Next you'll see a slightly more detailed summary of the files that have been added, changed, or removed since the database was initialized or last updated. In this case, there's only one file that's been changed, /bin/su:

 Changed files: changed:/bin/su 

Finally, you see the gory details of exactly what AIDE found for the files in the report. The items you see in the detail section of the report correspond to the type of check that you configured in aide.conf; you do remember that, right? In this case, six items changed, the file size, permissions, Mtime (modified time), Ctime (creation time), inode, and MD5 hash:

 File: /bin/su  Size   : 23176               , 23948  Permissions: -rwsr-xr-x            , -rwsr-xr-x  Mtime  : 2002-04-07 10:59:14        , 2004-03-09 20:24:31  Ctime  : 2002-04-08 20:45:36        , 2004-03-09 20:24:31  Inode  : 98564               , 100922  MD5   : XTJK0hKy/492djesGoBx7A==     , 1B2M2Y8AsgTpgAmY7PhCfg== 

Based on this report, it's easy to see that something changed with /bin/su and probably not for the better. Knowing that I didn't perform any updates to the server in the past few days, I can reasonably assume that a break-in has occurred and someone has replaced the normal su program with one of his own. An attacker might do this so that he can steal the root password or, more likely, he would do it so that he always has a way to get root access into the server. The attacker's trojaned su program might always allow users to go to root if they type a certain password (not the real root password) or if they log in as a certain user. In this manner, no matter what I change the root password to, the attacker can always get root access.

Without a utility such as AIDE I might not see that /bin/su has changed. It's still roughly the same size and the permissions are the same. A more skilled attacker would've also changed the Ctime to match an earlier time, to make it appear that the file is older than it really is. In that way, an ls -la on the file wouldn't have given many clues as to the real status of this important program.

The /bin/su example is just one of many in a long line of similar examples. Attackers can and will replace your programs with their own versions to hide their activity or give them elevated privileges. For example, a common attack practice is to place a trojaned ps program on a server to hide the attacker's processes. Performing a ps awux to look at what processes are running wouldn't reveal the attacker's processes if the trojaned version is in place. Another attacker might replace the cron scheduling program with one of his own that automatically runs his attack again after you've cleaned up from it the first time!