12.5. Logging

Linux activities are recorded into several logs, and the information logged can reveal many interesting things. For example, you can use the log information to discover hackers, when they entered your system, where they came from, what they did in the system, when they left, and other important things. Because logs are one of the security tools, I will consider them in more detail. This information will allow you to exercise tighter control over your domain.

12.5.1. Main Commands

Information about current system users is stored in the /var/run/utmp file. However, trying to view this file using, for example, the cat command will not produce any legible results. Data in the file are stored not in the text but in the binary format and can be viewed only with the help of special commands (programs). Some of these commands are given here.

The who Command

The who command shows who is currently logged into the system and how long they have been logged in. The information is extracted from the /var/run/utmp file. It is displayed in the following format:

 robert tty1  Dec 8 10:15 root   tty2  Dec 8 11:07

The first entry tells you that user robert is using terminal one (ttyl) and entered the system on December 8 at 10:15.

Most hackers execute this command when entering a system to see whether the administrator is logged in. If they see that the root user is in the system, beginning hackers hightail it because they are afraid their knowledge is not sufficient to remain in the system undetected.

This is yet another reason you should not log into the system as root. You are better off logging in as a regular user; you can always switch to a privileged user when you need more rights. For a privileged user in my system, I created a user account named something other than root and set its UID to zero. Using this account, I have complete access to the system yet do not advertise being there. This way I can lay in wait for unsuspecting hackers and observe their actions to learn how they obtained access to my system.

The users Command

The users command displays user names of all users currently logged into the system.

This information is also stored in the /var/run/utmp file, but only while the user is logged in; it is deleted when the user logs out. The permanent history of logins is stored in the /var/log/wtmp file. This is a binary file, and its contents can only be viewed using special programs.

The last Command

The last command shows when a certain user logged into (and logged out of) the system. The command takes a user name as the parameter. For example, the following command shows the login history for user robert:

 last robert

The command's execution produces results looking like the following:

 robert tty1 Thu Dec 2 12:17 - 12:50 (00:33)

From this entry, you can tell that the user took the ttyl terminal, logged into the system on December 2, and remained logged for 33 minutes, from 12:17 to 12:50. If a user logged into the system over the network, information about the host, from which the login was made, will also be shown.

Executing this command for regular user, such as yourself, will produce quite a lengthy list, containing records of all logins for the specific user since the /var/log/wtmp file was created. The number of displayed entries can be limited by specifying it with the -n key. For example, the following command displays information about the last five logins for user robert:

 last -n 5 robert

The lastlog Command

The lastlog command displays all users and their last login times. An example of the listing produced is shown in Listing 12.1.

Listing 12.1: Listing produced by the lastlog command

 Username         Port     From             Latest root             ftpd2022 192.168.77.10    Mon Feb 21 12:05:06 +0300 2005 bin                                        **Never logged in** daemon                                     **Never logged in** adm                                        **Never logged in** lp                                         **Never logged in** sync                                       **Never logged in** shutdown                                   **Never logged in** halt                                       **Never logged in** mail                                       **Never logged in** news                                       **Never logged in** uucp                                       **Never logged in** operator                                   **Never logged in** games                                      **Never logged in** gopher                                     **Never logged in** ftp                                        **Never logged in** nobody                                     **Never logged in** vcsa                                       **Never logged in** mailnull                                   **Never logged in** rpm                                        **Never logged in** xfs                                        **Never logged in** apache                                     **Never logged in** ntp                                        **Never logged in** rpc                                        **Never logged in** gdm                                        **Never logged in** rpcuser                                    **Never logged in** nscd                                       **Never logged in** ident                                      **Never logged in** radvd                                      **Never logged in** squid                                      **Never logged in** mysql                                      **Never logged in** flenov           ftpd2022 192.168.77.10    Mon Feb 21 12:05:06 +0300 2005 named                                      **Never logged in** robert           tty1                      Mon Feb 21 12:10:47 +0300 2005

The list is divided into the following four columns :

User name taken from the /etc/passwd file
The port or terminal, to which the connection was made
The computer's IP address for network logins
The login time

This command can be used to control system accounts. Their latest login time is shown as **Never logged in** , because these accounts cannot be used to log into the system (they have dummy command-interpreter shells such as /bin/false, /dev/null, and /sbin/ nologin ). If you notice that one of these accounts was used to log into the system, this means that hackers changed the account's configuration settings and are using it to work in the system.

Simply changing the command shell for the /etc/passwd file will create a back door unseen by the administrator. But executing the lastlog command will bring these machinations with the system accounts to light.

Pay attention to the connection type and address. If something is suspicious, this may be an attack at the inception stage.

The lsof Command

The lsof command can be used to determine, which files and by which users are currently open . The result of its execution looks similar to the following:

 COMMAND PID  USER   FD  TYPE  DEVICE   SIZE     NODE NAME init     1  root  cwd   DIR     3,2   4096        2 / init     1  root  rtd   DIR     3,2   4096        2 / init     1  root  txt   REG     3,2  26920   635256 /sbin/init init     1  root  mem   REG     3,2  89547   553856 /lib/ld-2.2.5.so init     1  root  10u   FIFO    3,2          195499 /dev/initctl keventd  2  root  cwd    DIR    3,2   4096        2 / keventd  2  root  rtd    DIR    3,2   4096        2 / kapmd    3  root  10u   FIFO    3,2          195499 /dev/initctl

But this list is far from being complete. Even if there is only one user currently logged into the system, there can be a couple of dozen files open. But if there are several users logged in, the number of open files increases , because the same file can be opened more than once by different users. These are mostly system configuration files.

12.5.2. System Text Logs

The next logs considered are text format files. They can be viewed using the cat command or any text editor.

The /var/log/messages file contains the main information about user logins, failed authorizations, launches and shutdown of services, and so on. Information about all of these events cannot fit into one file and is split into several messages.X files, where X is the file number.

This is the most important log for any administrator. If hackers try to pick the password to some account, you will be able to notice this because the file will grow rapidly and contain numerous failed authorization entries. Fig. 12.1 shows an example of the contents of this file.

Figure 12.1: A screenshot of the /var/log/messages file contents

Another text log is stored in the /var/log/secure file. This is the most important file and you should check it as often as possible, paying close attention to each entry. This file contains information about remote logins to the system: who, when, and from what IP address. For example, you may discover an entry of the chief accountant connecting to the FTP server but from an address that does not belong to him. This is enough to sound the alarm.

The same file contains information about changes to the user and group lists. Hackers seldom use the root account for their dirty deeds, creating an inconspicuously-named but zero UID account. If such an account was created not manually, by editing the corresponding files, but with the help of Linux commands, this activity will be logged in the /var/log/secure file.

Hackers know about this and, therefore, add users manually. After all, this is not that difficult: All it takes is to add a single entry to the /etc/passwd and /etc/shadow files. But even then, spotting a user logging in that you did not create, you can suspect something goes wrong.

Yet to spot suspicious entries, you have to be vigilant. The most experienced and, therefore, the most dangerous hackers have some ingenious tricks up their sleeves. For example, a hacker can see that there is a regular user named robert on your system and create a user account named rodert. The difference may seem obvious, but with scores of user accounts to keep track of, you have to pay too close attention to notice it.

The log of the sendmail mail server is stored in the /var/log/maillog file. The information in the file is stored in the following format:

 Jan 16 13:01:01 FlenovM sendmail[1571]: J0GA11S01571: from=root, size=151, class=0, nrcpts=l, msgid=<200501161001.j0GA11S01571@flenovm.ru>, relay=root@localhost

The file contains information about by who, when, and to whom messages were sent.

I remember an administrator of a server hacked who manually inspected directories for foreign files. In my opinion, it would have been much easier to examine log records for this. If the hackers did not clean up the logs before leaving the system, you can gather lots of information from them.

But although logs can be informative, they cannot be relied on. Smart hackers always clear away their tracks and delete incriminating records from the logs; thus, inspecting directories manually may be useful, but check the log first.

12.5.3. The FTP Server Log

After breaking in to a system, hackers often upload their own programs on the server to raise their privileges or create back doors. They can use FTP for this purpose. Information about connections to the FTP server can be obtained from the /var/log/secure file, and the uploaded files are recorded in the /var/log/xferlog file.

Information is stored in the FTP server log in the text format, the same as in the mail server log. From my experience, the FTP server is the most frequent source of security- related problems. The server program is quite good, but hackers most often try to take over a user account that has FTP capabilities to be able to upload their hacking tools on the server. Examining the FTP log file, you can expeditiously determine what was uploaded to your server and by whom.

The following is an example of the contents of the log file:

 Sun Jan 16 13:21:28 2005 1 192.168.77.10 46668 /home/flenov/ sendmail.cf b _ o r flenov ftp 0 * c

The preceding entry tells you that a user from address 192.168.7.10 downloaded the /home/flenov/sendmail.cf file at 13:21 on January 16, 2005.

FTP is the most dangerous protocol because it can be used to download confidential data (for example, password files), or upload a hacker's data to the server (for example, a rootkit or a Trojan horse). You should learn to understand each record in the log to know what happens to files in the system. The function of each parameter in the log is as follows :

A full date showing the day of week, month, date, time, and year.
The session duration or time taken to download or upload the file.
The name and IP address of the remote host.
The file size in bytes.
The full path of the uploaded or downloaded file.
The transmission type a for ASCII or b for binary.
File manipulations C means the file was compressed, U means the file was uncompressed, T means the file was process with the tar program, and _ means no file processing took place.
The transfer direction o for download or i for upload.
The user type a for anonymous, g for guest, or r for real.
The local user name. For anonymous users, this will be the ID string.
The service name, usually ftp .
The authentication method for no authentication or 1 for RFC931 authentication.
The user identifier. If the user is not determined, the ID is an asterisk.
The transfer outcome c for successful or i for interrupted .

If you have never worked with the FTP log before, I recommend that you do this now and study carefully the example entry and entries from your own FTP log. You always have to approach a problem already prepared and not study it after it arises; otherwise , you are doomed to lose.

12.5.4. The Squid Proxy Server Log

The main log of the squid proxy server is stored in the /var/log/squid/access.log file. This is a text file, in which each entry consists of the following fields:

The starting time of the connection or event.
The session's duration.
The client's IP address.
The results of the request processing. It can be one of the following:
- TCP_HIT The necessary copy was found in the cache.
- TCP_NEGATIVE_HIT The object was cached negatively; the object request returned an error.
- TCP_MISS The object was not found in the cache.
- TCP_DENIED The service request was denied .
- TCP_EXPIRED The object was found but is outdated .
- TCP_CLIENT_REFRESH A forced refresh is requested .
- TCP_REFRESH_HIT During the refresh attempt, the server indicated that the object had not changed.
- TCP_REFRESH_MISS After the refresh attempt, the server returned a new version of the object.
- TCP_REF_FAIL_HIT The object in the cache is outdated, but the attempt to obtain a fresh copy failed.
- TCP_SWAPFAIL The object should be in the cache but is not there.
The number of bytes received by the client.
The request method GET, POST, HEAD, or ICP_QUERY .
The URL of the requested object.
The ident field (- if not available).
The result of the request to the other caches PARENT_HIT (object found), PARENT_UDP_HIT_OBJECT (object found and returned in a UDP request), or DIRECT (object requested from the original server).
The type of the MIME contents.

There are two other proxy server logs. These are the following:

cache.log This log stores information about the cache, saved and deleted object, and the like. I have never experienced a need to consult this log.
useragent .log This log stores records of the User Agent field from request headers. This information can be faked easily, and I have already shown that this file can be easily changed for squid requests .

12.5.5. The Web Server Log

Apache server logs are stored in the access.log and error.log files located in the /var/log/httpd directory. These logs contain information about user accesses and activities.

Logs are in the text format and can be viewed by anyone , including hackers. Because logs contain user passwords, they constitute a security danger.

It is impossible to dispense with keeping the logs, because they are necessary. But you should do everything possible to make them inaccessible to unauthorized people. At the least, I always change the logs' default directory. From my experience, hackers seldom examine the httpd.conf file but look for the logs in their default directories. If the logs are not there, hackers think that they are disabled.

In the /var/log/httpd directory, create the following empty files: access_log, access_log.l, access_log.2, access_log.3, access_log.4, error_log, error_log.l, error_log.2, error_log.3, and error_log.4. To add a touch of authenticity to the files, slap copies of some actual data into them, only make sure that they don't contain anything important. Any hacker worth being called a hacker will easily see that the data are old by the file change date and the dates inside the file but will be unlikely to suspect something fishy about this. The important thing is for the file change date to be the same as the dates in the file's entries.

To simplify creating such files, you can temporarily enable the logs in the default directory to have them collect some information and disable them afterwards.

Then modify the ErrorLog and CustomLog directives in the httpd.conf configuration file to point to another directory to store these logs. In this simple way, you'll have most hackers scratching their heads trying to figure out the story with the log files.

12.5.6. The Log Keepers

Logs in the /var/log directories are kept by the syslogd and klogd daemons. Only the former is on the list of automatically started services in the setup program, with its start-up parameters also determining the start-up parameters of the klogd daemon. If yours is a standalone system or you simply do not require system events logging, you can disable these services to save processor resources. For a server system, however, I do not recommend disabling these services. If you do not see the necessity for using logs now, when you run into the first problem or break-in, you will see all the advantages they offer.

The syslogd program logs systems messages. The klogd program is used to log kernel messages. The log settings are stored in the /etc/syslog.conf file. Listing 12.2 shows an example of this file's contents.

Listing 12.2: The contents of the syslog.conf file

 # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.*                                          /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! #.info;mail.none;authpriv.none;cron.none         /var/log/messages # The authpriv file has restricted access. authpriv.*                                       /var/log/secure # Log all the mail messages in one place. mail.*                                           /var/log/maillog # Log cron stuff. cron.*                                           /var/log/cron # Everybody gets emergency messages. *.emerg                                          * # Save news errors of level crit and higher in a special file. uucp,news.crit                                   /var/log/spooler # Save boot messages also to boot.log. local7.*                                         /var/log/boot.log

The directives' functions can easily be deduced from the corresponding comments. All directives have the following format:

 name.level

The name part is the name of the parameter to be logged. These are the following:

kern Kernel messages
auth Security and authorization-violation messages
authprith Privileged-access use messages
mail Mail programs messages
cron Messages from the cron and at task schedulers
daemon Service-generated messages
user Messages from user programs
uucp UNLX-to-UNIX copy (UUCP) messages, which are rarely used today
news News messages
lpr Print messages

The level can be one of the following ( lowest to highest level):

* Logging all system messages
debug Debugging information
info Informational messages
notice Notices
warn Warnings
err Errors
crit Critical messages
alert Requiring immediate operator intervention
emerg Emergency, so further operation is not possible

Messages of the specified level and higher are logged. For example, specifying the err level means that messages of the err, crit , and emerg levels will be logged.

The more errors logged, the greater the hard disk workload and the more resources used. To enhance the system's productivity, it is advisable to place the /var partition, in which the logs are stored, on a separate hard drive. In this way, system events can be logged in parallel with the system's operation. Make sure, however, that the /var partition is large enough to hold all logs.

Moreover, in my systems, I move logs from their default location to make it more difficult for hackers to find them and delete the information about their activities in the system. But this is not enough. An experienced hacker will examine the /etc/syslog.conf file and discover the new location of logs.

But you can pull a better one on hackers with only standard Linux means. In my system, I have a task scheduled in the cron service that every hour makes a backup copy of the /var directory. In this way, even if the hackers clean up the log I can always find out about them from the backup log copies.

It is even better if you can have another Linux server installed in the network. Then you will be able to send all log messages to this server, which will make it even more difficult for hackers to reach them. In this case, hackers will have to break in to another server to be able to clean up the logs. If the server where the logs are stored is dedicated to this mission only and there are no unnecessary ports open, breaking in to it may turn out to be too difficult.

To send the log over the network, the /etc/services file has to contain the following entry:

 syslog 514/udp

Moreover, the following entry has to be added to the /etc/syslog.conf file:

 messages     @address

The messages parameter specifies, which messages are to be sent to the server. Specifying it as *.* will send all messages. To send only critical messages, this parameter has to be set to *.crit .

The @address parameter is the address of the server, to which the messages are to be sent. For example, the following entry sends all messages to the log.myserver.com server:

 *.*   @log.myserver.com

But there is one problem here: To determine the IP address, DNS is necessary. However, when the system is booted , the syslog service starts before DNS, making determining the IP address impossible. This problem is solved by entering the server's name and its corresponding IP address in the /etc/ hosts file.

Finally, the syslog service has to be started with the -r option, which allows the server to receive messages from the network and record them in its logs. For this, the script for launching the service has to be changed on the server. As you should remember, all scripts are stored in the /etc/rc.d/init.d directory; the script for the syslogd service is stored in the syslog file. Listing 12.3 shows the main contents of this file.

Listing 12.3: The contents of the /etc/rc.d/init.d/syslog file

 #!/bin/bash . /etc/init.d/functions [ -f /sbin/syslogd ]  exit 0 [ -f /sbin/klogd ]  exit 0 # Source config # Loading the configuration file if [ -f /etc/sysconfig/syslog ] ; then        . /etc/sysconfig/syslog else        SYSLOGD_OPTIONS  =  "-m 0"        KLOGD_OPTIONS  =  "-2" fi RETVAL=0 umask 077 start() {        echo -n $"Starting system logger: "        daemon syslogd $SYSLOGD_OPTIONS        RETVAL  =  $?        echo        echo -n $"Starting kernel logger: "        daemon klogd $KLOGD_OPTIONS        echo        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/syslog        return $RETVAL } stop() { # Commands to stop the service } rhstatus() { # Commands to output the status } restart() {        stop        start } ... ...

The most interesting information is in the following lines:

 if [ -f /etc/sysconfig/syslog ] ; then       . /etc/sysconfig/syslog else       SYSLOGD_OPTIONS  =  "-m 0"       KLOGD_OPTIONS  =  "-2" fi

Here, in the if block, a check is performed on whether the /etc/sysconfig/syslog file exists. If so, the loading parameters are taken from this file; otherwise, they are specified explicitly in the else block:

 SYSLOGD_OPTIONS  =  "-m 0"

The parameters are specified within the quotation marks. The -r option is added, modifying the directive as follows:

 SYSLOGD_OPTIONS  =  "-m 0 -r"

If the /etc/sysconfig/syslog file exists, it will contain the SYSLOGD_OPTIONS="-m 0" entry and you will only have to modify this entry, without having to tamper with the /etc/rc.d/init.d/syslog launch script.

Using a dedicated server is a two-edged sword: Although it enhances system security by providing backup copies of logs, it endangers the same security by exposing the logs to the public. The problem is that log messages are sent over the network in plaintext. This presents no problem if your network is protected from the Internet with a firewall and hackers cannot penetrate it. But if hackers manage to compromise at least one computer in the network, they can install a sniffing program and intercept all log messages.

This problem is easily solved by encrypting the message traffic, sending it through an SSL tunnel. The simplest way of doing this is as follows:

In the configuration file of the server sending log messages, specify the local computer as the one, to which all messages should be sent as follows:
```
 *.*   @localhost 
```
This will result in all messages being sent to the local computer to UDP port 514. The server must not be operating in the message receiving mode; that is, the -r option must not be specified in the launch script. Otherwise, port 514 will be busy, which is not what you need.
Start the stunnel client on port 514 of the local computer as follows:
```
 stunnel -c -d 127.0.0.1: 514 -r logserver:1050 
```
All messages received at this port will be encrypted and sent to port 1050 of the logserver computer, where logserver is the address of your log message receiving server.
On the logserver server, create an stunnel service as follows:
```
 stunnel -d 1050 -r 127.0.0.1:514 
```

This will result in the stunnel service on the logserver server receiving encrypted data on port 1050 and redirecting them in plaintext to port 514. The syslogd service on the logserver server must be started with the -r option to receive log messages at port 514.

Now, all log messages will be sent over the network encrypted.

12.5.7. The Logrotate Utility

To keep log files from bloating, Linux uses the Logrotate utility. I will demonstrate its operation using the /var/log/messages log as an example.

When the size of the /var/log/messages file exceeds the maximum size, or when a certain period lapses, the contents of the current log are transferred to the /var/log/messages.1 file, and the /var/log/messages file is cleaned and reused to log messages.
The next time one of the threshold values is reached, the contents of the /var/log/messages.1 file are transferred to the /var/log/messages.2 file and the contents of the /var/log/messages file are transferred to the /var/log/messages.1 file. The process is repeated every time one of the critical values is reached.

In this way, log messages are stored in separate files, each of which does not exceed a certain size, making logs convenient to view.

The configuration file of the Logrotate utility (/etc/logrotate.conf) is shown in Listing 12.4.

Listing 12.4: The configuration file of the Logrotate utility (/etc/logrotate.conf)

 # See "man logrotate" for details. # Rotate log files weekly. weekly # Keep 4 weeks worth of backlogs. rotate 4 # Create new (empty) log files after rotating old ones. create # Uncomment this if you want your log files compressed. #compress # RPM packages drop log rotation information into this directory. include /etc/logrotate.d # No packages own wtmp -- we'll rotate them here /var/log/wtmp {     monthly     create 0664 root utmp     rotate 1 } # System-specific logs may be also be configured here.

The following parameters can be specified in this file:

weekly Specifies that log files are to be rotated weekly. If the server's workload is light, this value can be changed to monthly .
rotate Indicates the number of files for storing the logs. In this case it is four, meaning that there will be four numbered log files /etc/log/name. 1 through /etc/log/name.4 in addition to /etc/log/name.
create Notes that a new log file is to be created immediately after the rotation.
compress Specifies to compress the old versions of the log files. This option is useful for servers with heavy request workloads, generating bulky logs. Because logs contain text information, compressing them reduces their size by 70% and more.

The default values are described at the beginning of the file. Afterwards, necessary values for particular logs are specified. In this configuration file, specific parameters are set for the /var/log/wtmp log file:

 /var/log/wtmp {     monthly     create 0664 root utmp     rotate 1 }

Here, the maximum log file size is not specified; this, however, can be done using the size parameter. For instance, in the following example the maximum size of the log file is set to 100 KB:

 /var/log/wtmp {     monthly     size  =  100k     create 0664 root utmp     rotate 1 }

Now the log file will be rotated whenever one of the following two events occurs:

Monthly
When the file size reaches 100 KB

The file-rotation capability presents both conveniences and shortcomings. For example, the hackers can wipe out their tracks after the attack even if they have no direct access to the log files. All they have to do is to flood the log with trash messages, causing it to exceed the maximum size and be deleted by the system.

Attempting to protect the log file by increasing its size is useless, because hackers will not generate messages manually but can use a simple Perl program or even a script containing shell commands. The latter program is really simple. All it has to do is loop through the logger command (which logs messages), as follows:

 logger -p kern.alert "Message_text"

Running this command in an endless loop will make the system to destroy the log file.

To prevent log data from being destroyed , you can specify a script to send the log to the administrator's email address as follows:

 /var/log/wtmp {     monthly     size  =  100k     create 0664 root utmp     postrotate       The script command for mailing the name.1 journal     endscript     rotate 1 }

In this case, after the log file is rotated and the main log file is renamed to name.l, the script is executed to send the latter file to the administrator's email address.

When emailing log files, make sure that your mailbox is large enough. For example, if the maximum size of the log file is 10 MB but your mailbox can only take 5 MB of messages, you will never receive this file because it will be deleted by the system.

12.5.8. User Logs

All commands executed by users are logged in the .bash_history file (when the /bin/bash command interpreter is used), which is located in the user's home directory. If you know which user account hackers used to break in to the system, you can trace all their actions with the help of this log.

You will be able to tell what commands or programs were run, and this information may be helpful in determining how the hackers penetrated the system and what they may have changed in it. If the hackers added a user or modified some important system file, you will be able to see this, return everything back to normal, and close the holes in the system used by the hackers to get in.

Professional hackers who earn their living breaking in to computer systems know this and do everything possible to hide their tracks and regularly clean up this file. Extraneous changes in the .bash_history file may serve as an indication that this account was used by the hackers to break in to the system.

You should also regularly check and clean user logs yourself. Users, including yourself, may make a mistake when issuing a command and include your password with it. Hackers can discover this password when analyzing the .bash_history file and use it for nefarious purposes.

If you specified the root password when entering a command, take your time to delete the corresponding entry from the user log. Leaving it there may cost you dearly.

Administrator passwords can also be entered in the command line when working with MySQL. For example, if you executed the /usr/bin/mysql -uroot -ppassword command, it will be recorded in the log. If hackers obtain access to the bash command log, they will have an opportunity to discover the password entered with the command. If they do learn this password, they will be able to use the MySQL database with root rights. This will be the best case scenario. The worst case will be if you use the same root password for MySQL as for the system: This will give the bad guys complete control over your server.

Note

Never enter a password as a parameter when executing commands from the command line. If for some reason you do this, then delete the corresponding entry from the bash log. With MySQL, only the /usr/bin/mysql -uroot command should be entered. The server will reply, requesting the password. Entering the password will not record it in the log; only the /usr/bin/mysg1 -uroot command will be recorded.

If you work with a MySQL server, there will be the .mysql_history file in addition to the .bash_history file in your home directory. This file stores all commands executed in the MySQL configuration program. If any passwords were entered during the MySQL configuration process, their corresponding records must also be cleaned from the mysql log file. Even though a database is not the system, it can serve as a beachhead for breaking in to the system; moreover, databases can contain confidential data, such as passwords to restricted sections of Web sites.

12.5.9. Things To Heed

In this section, I want to consider what things must be paid attention to in security logs. These are not only the entries recording unauthorized system access. When you see that there was unauthorized access to restricted information, the break-in has already taken place. The idea is to use the information in the logs to detect attacks in the inception stage, and to prevent them.

If logs started suddenly growing in size, it means that there is some irregular activity going on. Perhaps a DoS attack is being perpetrated. You should immediately react to and investigate the causes of this increased activity without waiting for the situation to develop to the point where the server will be overwhelmed and cease servicing visitors . Moreover, if logs fill the disk, the system may crash, so make sure you have enough disk space for logs.

The computer should not reboot on its own. If this happens, check the logs to find out why and when it rebooted. You can use the uptime command to find out how long the system has been running.

Keep an eye on repeating entries, especially on those related to authorization. Too many failed authorization entries are an indication of a possible password-picking attempt.

If you notice some of the suspicious activities just listed, you should establish where the potential threat comes from, namely, the IP address and the location of the attacker's network. To prevent further actions by the hackers, you can change the firewall policy by adding a rule prohibiting any connections from the attacking host address.

When analyzing logs, pay attention to every little thing. For example, to pick a password, hackers can use different camouflaging tricks, such as making false log entries.

If hackers try to pick a password using the enumeration method, there will be many unsuccessful entry attempts by a certain user in the log. An unsuccessful system login attempt generates the following entry in the /var/log/messages log:

 Feb 12 17:31:37 FlenovM login(pam_unix)[1238]: authentication failure; logname=LOGIN uid=0 euid=0 tty=ttyl ruser= rhost= user=root

The login (pam_unix) parameter indicates that the hacker was only trying to login. If the hacker was already in the system but used the su command unsuccessfully, the logname field will contain the user name, under which the hacker entered the system, and the login (pam_unix) string will be replaced with su (pam_unix) .

By this entry, you can easily determine that it was generated by hackers, and easily locate them. But hackers can also easily insert fake entries in the log pointing to another user, making it difficult to pinpoint the real ones. For example, executing the following command, hackers can add an entry to the log identical to an authentication failure entry:

 logger -p kern.alert -t 'su(pam_unix)' "authentication failure ; logname=robert uid=0 euid=0 tty=tty1 ruser= rhost= user=root"

The preceding command will create an entry similar to the following in the log:

 Feb 12 17:31:37 FlenovM login(pam_unix)[1238]: authentication failure; logname=robert uid=0 euid=0 tty=tty1 ruser= rhost= user=root

Now imagine that hackers executed many such commands but with a different logname field in each. It will be practically impossible to determine, which log entries are real and which are fakes .

A less experienced hacker may mess up and not use the -t option when using the logger program, instead entering the command as follows:

 logger -p kern.alert "authentication failure ; logname=robert uid=0 euid=0 tty=tty1 ruser= rhost= user=root"

This will generate the following entry in the log:

 Feb 12 17:31:37 FlenovM logger: authentication failure; logname=robert uid=0 euid=0 tty=tty1 ruser= rhost= user=root

The keyword logger before the error message tells you that the entry was generated by the logger program and is, most likely, a fake.

But even if the hackers have no access to the logger program, they can use a program of their own to place fake entries into the log. To prevent this, only the root administrator should have write rights for the log files.