Chapter 12: Automatic Logfile Scanning

This chapter is about logfiles and logfile scanning, another critical facet in your total security architecture. It is important to create a policy and business strategy for dealing with system and application logfiles, because logs are vital company information that can be overlooked or undervalued. Logfiles are useful for three main reasons. First, logfiles help with troubleshooting system problems and understanding what is happening on the system. Second, logs serve as an early warning for both system and security events. Third, logs can be indispensable in reconstructing events, whether you have determined an intrusion has occurred and are performing the follow-up forensic investigation or just profiling normal activity. This chapter will start out discussing the importance of logfiles for yourself and your company. Then, there will be an explanation of the importance of centralized logging and the steps in establishing a centralized log server. After you see how to set up centralized logging, there will be instructions how to take it up a notch with encrypted centralized logging. We will look at how to search logfiles for different types of activity either manually or with searching tools. Last, we will look at a couple of examples of what hacker activities look like in the logs and how to make changes to compensate.

Logfiles at a Personal Level

At a personal level, logfiles can give you an idea about what the different parts of your system are doing. As you work more and more with Linux, you will discover that the best way to get in touch with what is going on with the software you are running is by looking at logfiles. Logs can show you what is going right and what is going wrong. When you see errors reported , you should take steps to remedy the situation before the situation escalates, whether it s hardware failure, failed access attempts, or worse .

At a professional level, logfiles can provide a useful profile of activity. It is much easier to determine something is wrong if you are familiar with what the normal business routine looks like. From a security standpoint, it is crucial to be able to distinguish normal activity from the activity of someone attacking your server or network. It is not enough to merely gather logs, you need to protect and process them too. This chapter will recommend a process for log protection and for log processing. After becoming familiar with best logging practices, you should be able to detect intruders before they penetrate your defenses or take steps to prevent recurrence if the worst has happened . Good security is only possible through good logging.

You might need to make some configuration modifications to get the right amount of logging. Many applications can be made to log activity at different levels. Sometimes you want only critical things logged, while other times you may want to increase the verbosity of an application to a debug level. After a few weeks of gathering logs, you will have a useful archive that profiles the activity of your system and your network. An archive of logs can be useful when you notice a certain type of activity for the first time. You can immediately refer to the past logs to see if the activity looks familiar in some way. You might discover an activity that happens rarely but regularly, like once a month. You won t notice the regularity of these types of events until you have months of activity to profile.

Profiles of activity can also be extremely useful in capacity planning. When determining how much to invest in new equipment, you need to know how often certain activities have happened in the past. On the Web, people are concerned with the number of hits a certain web page has received. Other important metrics such as knowing the number of logins, e-mail messages, or transactions are crucial in determining the correct amount of resources to meet your future needs. Logs also provide statistics to back up the recommendations you make to management.

Everyone hopes that they will never need to provide proof of abuse to law enforcement. If the worst case scenario becomes a reality and you need to substantiate your claims or provide information to law enforcement, it will be a lot easier if you have prepared in advance for the situation. If you intend to take some legal action, officials will ask for logs validating your claims. If you don t have any documentation, you won t have a leg to stand on, so it s important to design and implement policy for dealing with logs from a legal basis.

Use Centralized Logging

While working as a senior consultant for a major government agency's web site, a junior staff asked for help changing to root using the su command. He couldn't change into the root user , and he thought the root password had been changed. Investigations revealed that the root password had not been changed and that anyone could log in as root normally, they just couldn't become root with su . Further investigation showed that modifications had been made to the su binary. These changes were sophisticated enough that it was unlikely the staff had made them. Management was alerted, but disregarded security's recommendation to take the site down as being too disruptive and too conservative. Since there were not any logs collected or archived, it was impossible to determine what had occurred or when. Over the next several days, the hacker continued the rampage until they made their presence known by overtly changing the agency's homepage. The system needed to be taken offline and reinstalled. Again, had there been any logs showing what was done to break in, changes could have been made to prevent a recurrence. Management then again disregarded recommendations about restoring the system entirely from distribution instead of backups , and the hacker came back and wreaked havoc again. A comprehensive log policy would have prevented this from happening. If the policy had not been sufficient to prevent the occurrence in the first place by log searching, it certainly would have been sufficient to prevent its recurrence. Knowing what normal activity looks like is crucial. Otherwise, it is difficult ”if not impossible ”to distinguish appropriate activity from inappropriate.

 

Logs record what happens and when it happens. If unauthorized personnel are trying to access your computer, they will undoubtedly make many attempts before they succeed. If they do get in on their first attempt, it is because they have inside information and logging their activity is equally important. If your computer is compromised, and you are logging activity, you will have some record of what happened before the breach and what happened that permitted the breach.

If logfiles are only stored locally on your system, it is extremely likely that the person who breaks into your system will attempt to eradicate any evidence of what happened. The information on what IP(s) the attacker came from and what was done to break into your system may be altered or the files may be entirely deleted. Without logging information, it is unlikely that you will prevent hackers from penetrating your system s defenses repeatedly. However, if logs are stored somewhere else, the person needs to break into yet another system before he can delete the logs. In most organizations, the solution to the fragility of local log storage is a centralized log collection system.

Another reason for centralized logfile collection is the ease of administration it provides. When all logged information is centralized, it is easier to use the logs to obtain information about activity on the network. Searches can pinpoint suspicious activity across multiple systems, and the onerous task of visiting many systems and examining the logs is not necessary.