Section 10.4. Syslog

10.4. Syslog

All Unix variants record significant system and application events to text files called syslog files. Syslog is also the name of the Unix daemon (the equivalent of a Windows service) that performs the logging function. Syslogging isn't restricted to Unix servers: some network devices (from companies like Cisco) also implement the syslog feature.

10.4.1. Sending Syslog Events to MOM

Syslog supports redirection of the events to syslog files on other computers. This is how you get the syslog events into MOM. Controlling the behavior of syslog is done through the entries in the syslog.conf file, typically located in the /root/etc directory on a Unix system. Below is the default syslog.conf file from a Linux variant.

 1 # /etc/syslog.conf - Configuration file for syslogd(8) 2 # 3 # For info about the format of this file, see "man syslog.conf". 4 # 5 6 # 7 # 8 # print most on tty10 and on the xconsole pipe 9 # 10 kern.warning;*.err;authpriv.none   /dev/tty10 11 kern.warning;*.err;authpriv.none  |/dev/xconsole 12 *.emerg                  * 13 14 # enable this, if you want that root is informed 15 # immediately, e.g. of logins 16 #*.alert                 root 17 18 19 # 20 # all email-messages in one file 21 # 22 mail.*                   -/var/log/mail 23 mail.info                -/var/log/mail.info 24 mail.warning             -/var/log/mail.warn 25 mail.err                  /var/log/mail.err 26 27 # 28 # all news-messages 29 # 30 # these files are rotated and examined by "news.daily" 31 news.crit          -/var/log/news/news.crit 32 news.err           -/var/log/news/news.err 33 news.notice        -/var/log/news/news.notice 34 # enable this, if you want to keep all news messages 35 # in one file 36 #news.*                  -/var/log/news.all 37 38 # 39 # Warnings in one file 40 # 41 *.=warning;*.=err       -/var/log/warn 42 *.crit                   /var/log/warn 43 44 # 45 # save the rest in one file 46 # 47 *.*;mail.none;news.none      -/var/log/messages 48 49 # 50 # enable this, if you want to keep all messages 51 # in one file 52 #*.*                   -/var/log/allmessages 53 54 # 55 # Some foreign boot scripts require local7 56 # 57 local0,local1.*              -/var/log/localmessages 58 local2,local3.*              -/var/log/localmessages 59 local4,local5.*              -/var/log/localmessages 60 local6,local7.*              -/var/log/localmessages 

If Unix systems are not where you spend the majority of your time, notice the text in line 3, man syslog.conf. Entering this at a Unix command prompt opens the help file for syslog.conf. The help files are very detailed, so plan on spending some time going through it if you want to know more. The pertinent information has been extracted and it will give you what you need to perform the syslog.conf editing here.

Next, look at lines 21 and 22. Line 21 consists of only a # sign; this is the comment symbol for the syslog file that tells the syslog daemon to ignore the text on this line. Line 22, lacking the # sign will be interpreted and starts with mail.*. In syslog syntax, this means all events from the mail system. If you wanted error events, you would change this to mail.err as shown on line 25. Line 22 continues with a tab, which is ignored by the interpreter, and then -/var/log/mail. This is the default location that all mail events will be written to. Hopefully, the rest of the file now makes more sense.

To get all of the syslog events into MOM, replace all of the default locations (-/var/log/mail) with the IP address of the MOM management server you will be collecting these at, proceeded by a @ sign. So, you would change -/var/log/mail to @10.0.0.64. After editing, the syslog.conf file looks like this:

 # /etc/syslog.conf - Configuration file for syslogd(8) # # For info about the format of this file, see "man syslog.conf". # # # #print most on tty10 and on the xconsole pipe # kern.warning;*.err;authpriv.none   @10.0.0.64 kern.warning;*.err;authpriv.none   @10.0.0.64 *.emerg                  * # enable this, if you want that root is informed # immediately, e.g. of logins #*.alert                 @10.0.0.64 # # all email-messages in one file # mail.*                  @10.0.0.64 mail.info               @10.0.0.64 mail.warning            @10.0.0.64 mail.err                @10.0.0.64 # # all news-messages # # these files are rotated and examined by "news.daily" news.crit          -@10.0.0.64 news.err           -@10.0.0.64 news.notice        -@10.0.0.64 # enable this, if you want to keep all news messages # in one file news.*             -@10.0.0.64 # # Warnings in one file # *.=warning;*.=err       -@10.0.0.64 *.crit                   @10.0.0.64 # # save the rest in one file # *.*;mail.none;news.none        -@10.0.0.64 # # enable this, if you want to keep all messages # in one file #*.*                   -@10.0.0.64 # # Some foreign boot scripts require local7 # local0,local1.*              -@10.0.0.64 local2,local3.*              -@10.0.0.64 local4,local5.*              -@10.0.0.64 local6,local7.*              -@10.0.0.64 

Save the file and reboot the Unix server (which restarts the syslog daemon). All of the syslog events will flow into MOM.

10.4.2. Configuring MOM for Syslogs

Preparing MOM to receive syslog events is much simpler than preparing MOM to receive SNMP because there is nothing in the OS to configure; all configuration is done in MOM.

The rule that will generate alerts from syslog messages must have a data provider to monitor, so the first step is to create that provider. In the Administrator console open the Management Packs Providers container, bring up the context menu for the Providers container and select to create a new provider. This opens the Select Data Provider Type page, as shown in Figure 10-18.

Figure 10-18. Creating a syslog port provider in MOM

Select the Application Log data provider type and click Next. Since syslogs are just plain text files, it make sense that MOM would treat them the same as any other application log, such as the IIS logs. This opens the Application Log Provider Properties page (see Figure 10-19).

Syslog runs over UDP port 541, and MOM has a Provider log type already created for this. Select the "Syslog port" Provider type, give the provider a name, and click Finish.

Now you can create a syslog rule group, associate it with a computer group and make a new event rule just as you did for the SNMP rule. In this case though, for duplicate alert suppression, select computer, domain, and Alert Description as the qualifying fields. Don't forget to commit your configuration change.

When MOM starts receiving syslog messages, you will see entries for these computers in the Administrator console, in the All Computers container (see Figure 10-20).Figure 10-21 shows what a syslog message generated alert looks like in the Operator console. Because MOM is able to classify the domain as Unix Systems and give each alert a unique nameits IP addressthe alerts generated by these messages are shown in the Operator console as coming from those machines. This happens despite the fact that all of the syslog-to-alert alerts are generated on the management server (point 1 in Figure 10-21).

Figure 10-19. Selecting the application log type

Figure 10-20. Unix systems in the Administrator console

Figure 10-21. A syslog message generated alert

Point 2 in Figure 10-21 shows that the syslog message has the severity of Error and is from a system daemon named windbindd. Point 3 shows a repeat count of 2 for this alert. By itself, this is not remarkable, but if you look in the results pane (under point 1 in Figure 10-21) you can see that there is a previous alert from the 10.0.0.51 system. This proves that duplicate alert suppression is working via the repeat count being incremented to 2 and that alerts that differ based on Alert Description are not being suppressed.