Once the monitoring systems are in place, all captured data should be logged to a file or database. Like monitoring, logging should include a layered approach of in-band and out-of-band solutions. Log files on the honeypot system should be used, but because they can be manipulated by the hacker, they should not be relied on unless collaborated by external logs.
Together, all the different in-band and out-of-band methods of monitoring and logging allow a honeypot system to meet the basic goal of data capturing. In-band methods for real and virtual machine honeypots include using the Windows event log, auditing, a keystroke logger, and Syslog. For emulated honeypots, in-band methods include using the console log, screen log, packet capturing, log files, and Syslog. For out-of-band monitoring and logging for all types of honeypots, you can use packet capture, an IDS, and network device logging.
Although writing events to a log file may seem simple, security logging is an art form. So many people do it badly. You know the ones (it could be you)—they have dozens to hundreds of log files being collected across their networks, but they don’t read them. Why? Because the unuseful information (noise) filling the logs far outweighs the useful information. Security administrators are already overworked, so they certainly don’t have the time to look through hundreds and thousands (and hundreds of thousands) of security events that really don’t have anything to do with active vulnerabilities and attacks. Good security logging involves the following parts:
Security event logging
Centralized data collection
Useful information extraction
All logging parts must be planned, coordinated, and tested in order for logging to be successful and useful.
It is important that all computers involved in the honeypot system be time synchronized, including the logging and monitoring workstations. Make sure the time, date, and time zone settings are identical. Unsynchronized systems can make event correlation much harder than it needs to be. Secondarily, guaranteed accurate time synchronization will make hacking evidence hold up in court better.
I also recommend that you set the time formats on security logging devices to Coordinated Universal Time (UTC). UTC is the international time atomic clock standard successor to Greenwich Mean Time (GMT). Local time zones are measured as plus or minus hours as compared to UTC or GMT. For instance, UTC minus 5 hours equates to the Eastern time zone of the United States.
Hackers frequently cross international borders and time zones. When sending log information to foreign third parties or trying to coordinate monitoring efforts, using UTC references will give everyone a common starting point. The Network Time Protocol (NTP) uses UTC time increments sent with UDP packets to coordinate time for computers reachable over the Internet. Centrally contactable time servers updated by very accurate atomic clocks are used to keep NTP hosts time synchronized. The NTP Project (http://www.ntp.org) is a great source of information about automated computer time synchronization.
Modern Windows computers use the Windows Time Service and a protocol called Simple Network Time Protocol (SNTP, documented in IETF RFC 2030). The Windows Time Service can be fed its time reading from an internal computer clock (the default) or use an external time source, like an Internet NTP time server. Window workstations participating in aWindows 2000 or above domain and using the default Kerberos authentication must be time synchronized within five minutes of the authenticating domain controller to complete a successful login. In aWindows domain environment, the domain controller that fulfills the PDC Emulator role is the centralized time sync server for the domain. The PDC Emulator computer should use a very accurate internal PC clock or be configured to get its time from an external NTP server source (http://support.microsoft.com/kb/216734/EN-US). There are several free NTP clients for honeypot systems that do not have an NTP-compatible application. NetTime (http://nettime.sourceforge.net) is a free NTP client for Windows 9x and NT systems.
Where should logging be done? The short answer is anywhere it can. On honeypots with a real Windows OS, start with the Windows event logs, and use other third-party tools as desired. Capture all network traffic into and out of the honeypot system to an IDS and/or network traffic sniffer (as covered in Chapter 9). All network devices in the path of data communications headed into and out of your honeypot system should have logging enabled, and detailed logging when possible. This means logs from OS, firewalls, IDS, routers, switches, gateways, antivirus software, and anything else that may track packets or network communications.
Although the main Windows logging system writes events to the Security log, almost every major Windows service and process (Routing and Remote Access, IIS, bootup, and so on) has its own logs. Don’t miss the opportunity to enable and use them when possible.
Windows event logging doesn’t get enough respect. Although Microsoft’s event log messages may be convoluted at times, if you configure audit policy correctly and research the resulting messages, you’ll find a fairly detailed log. Microsoft event log files are perhaps the best logs of any OS. Once, when I was interviewing a hacker, I asked why his rootkit didn’t bother to clear the event logs. He responded that nobody even checked them, so why waste the time and CPU cycles. Point taken.
You should enable auditing on all real Windows honeypots. By default, a honeypot system should not have any activity above what was recorded in the baseline. So any activity, whether it is successful or fails, should be logged. Here are Windows main auditing categories:
Audit Account Logon Events: If enabled, these events are logged on the authorizing domain controller, no matter which computer is logged in to or authenticated to. It tracks domain controller authentication events and login behavior. Account logins are tracked when users log in to computer or access a resource (such as file or printer share). Audit Account Logon Events was introduced in Windows 2000. On Windows NT, Audit Logon Events was used. This necessitated visiting each computer in a network when trying to track a hacker. With Audit Account Logon Events, you just need to go to every domain controller computer in a domain, at the most.
Logon events are really logon and authentication events, not just logins. Anytime a user or computer accesses an object (in Windows nearly everything is an object with permissions), an authentication event has to happen before the user is granted access to the object.
Audit Account Management Enabling: This tracks when a user account or group is created, changed, or deleted. It also tracks when a password is set or changed.
Audit Directory Service Access: This records the event of a user accessing an Active Directory object. It can track hacker activities during a compromise.
Audit Logon Events: This audits logins and authentication events at the computer where the resource is accessed. It tracks domain account logins on the computer where a logon event is happening. It can also track local account logins. The Security log will contain two events if a user logs in to a local account and Audit Account Logon Events is enabled, too.
Audit Object Access: This tracks success or failure of a security principal (user, group, service, or computer account) trying to access a file, folder, Registry key, or printer. After the category is enabled, auditing must be turned on by selecting the individual object and security principal to be tracked.
Audit Policy Change: This tracks any change to user rights assignment, audit, or trust policies. It might catch a hacker attempting to elevate privileges.
Audit Privilege Use: This tracks use or attempted use of user rights assignment privileges, which are listed in any Group Policy Object. Privileges allow a user, group, computer, or service to do something not explicitly given by a regular security permission (such as logging in as service or batch job).
Audit Process Tracking: This tracks detailed tracking information for events such as program activation, process exit, handle duplication, and indirect object access. For a honeypot, this audit event will reveal every program and command the hacker tries or uses.
Audit System Events: This tracks when a user restarts or shuts down the computer, or when an event occurs that affects either the system security or the Security log.
Most auditing is recorded to the Security log. Account Logon events are logged on the domain controller (or perhaps on the local computer).
You should track the success and failure of all audit events on all Windows honeypots. When all audit categories are enabled, the information auditing provides is extremely valuable. Chapter 11 discusses the critical auditing events you should monitor.
Logs should be collected and rotated frequently enough that data is not overwritten. It is important that logs be rotated on a regular enough basis to find balance between accuracy and performance. Logs can quickly become large. A compromised honeypot can have daily logs hundreds of megabytes big. If the log files are not ended and new logs started—that is, they are not rotated—frequently enough, the log files will grow too large to handle easily during the data analysis portion.
In one of the examples you’ll see in Chapter 11, the log files were over 200MB. Every analysis query took 2 to 10 minutes each. This might not seem like a lot of time, but it at least doubled the analysis time and made it hard to concentrate on the task at hand. Smaller logs offer better performance. The offsetting problem is in losing accuracy because similar events are spread across two or more logs. In a simple example, a port scan could be spread across two or more log files, resulting in missed identification of the port-scan activity.
Logging of security events data should be done to a permanent write-once, read-many media source if there is a chance it will be used in court. Courts often support the “best evidence” doctrine, which states that nearly anything (except evidence known as hearsay) can be used for evidence in a court of law. But evidence that is professionally collected and resistant to tampering will fair better in a court of law than evidence without the same protections. If you can prove that your data was collected in a time-synchronized environment, tracked through its chain of custody, and difficult to manipulate after collection, you have a pretty good evidence trail. Thoughtful decision making must accompany log file rotation and archiving.
It is the rare honeypot system or network with only one log. Most systems have dozens or hundreds of logs. Individually checking each log manually almost guarantees that they won’t be checked routinely. Smart administrators collect as many security events to a centralized location as they can. It is nearly impossible for one centralized data collection system to collect all events, but the more you can collect centrally, the better off you will be.
Microsoft has several free tools administrators can use to collect security event logs to a central location, although none are particularly perfect or elegant. There are also dozens of third-party applications available from different vendors to collect and prioritize Windows log files.
If you have Windows 2000 or later, you can use the Event Viewer Microsoft Management Console (MMC) snap-in to view event logs on the local machine and/or one or more remote machines. You can create an Event Viewer console that contains multiple computers’ event logs in one location, as shown in Figure 10-7. Although the Event Viewer console allows you to view multiple computers’ event logs in a central location, each log and its events are still separated.
Figure 10-7: Event Viewer snap-in console monitoring several computers
In order to view remote events with Event Viewer, you must have local administrator rights, plus have the Remote Registry and the Server services enabled on the remote machine.
Microsoft provides several ways to remotely collect multiple security event logs into a centralized database where they can be viewed, sorted, and prioritized at once. The oldest among these tools is EventCombMT (http://support.microsoft.com/default.aspx?scid=kb;en-us;824209&Product=winsvr2003). It allows you to query multiple computer event logs and get the results in a common file. The file can be imported into SQL Server, Microsoft Excel, or another tool for analysis.
EventCombMT works on all versions of Windows NT and above. Although EventCombMT works well, it is not a real-time utility. Each time you want to collect data, you must initiate the EventCombMT query. When running against multiple machines, or even on a single machine with tens of thousands of records, it can take a long time for each query.
Microsoft released a new tool called Log Parser in the IIS 6 Resource Kit (http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=8cde4028-e247-45be-bab9-ac851fc166a4) that, although a rudimentary command-line tool, can extract data and events from a wide range of log sources. It does not require IIS 6 to work.
Log Parser uses SQL-like statements to query log file sources. Queries can be basic and extract all events or pull just specific records. You can save data to SQL databases, CSV files, and many other formats. If you aren’t used to SQL queries, the complexity and exacting syntax can be a hurdle. And like EventCombMT, Log Parser is a batch process. You can learn more about LogParser at http://www.logparser.com.
Microsoft’s newest addition to the log collection family is the Microsoft Audit Collection System (or MACS). MACS is a real-time event log collection system that works with only Microsoft security event logs. Multiple computers’ security log files can be collected to a centralized SQL database. Each participating client runs a MACS-client service that communicates with the centralized server. MACS is Microsoft’s future of security event log collection. It is still in limited beta and not available to the general public. You can read more about MACS at www.windowsboston.com/downloads/doc/MACS_beta_Overview.doc.
Microsoft also supports event log collection using SMS (http://www.microsoft.com/smserver/default.asp) and Microsoft Operations Manager (http://www.microsoft.com/mom/default.mspx), known as MOM.
A popular commercial choice is GFI LANguard Security Event Log Monitor (http://www.gfi.com/downloads/downloads.asp?pid=6&lid=1) from GFI. It is a user-friendly, GUI-driven event log message collector. Although not as feature rich as some of its more expensive competitors, GFI may do the job for most network administrators (and certainly, it’s enough for a smaller honeypot system).
The industry standard for collecting log files is called Syslog (for system log daemon), although it is much more of a standard in the Unix and Linux world. A Syslog application is not included in Windows by Microsoft. Fortunately, many third-party utilities can read Windows event log files and send them to servers running Syslog daemons.
Syslog-collected logs hold a lot of promise for a centralized collection system. First, nearly every security device in the world supports or writes to Syslog files natively. Second, those that don’t can usually be modified to support Syslog in a client or server capacity.
The disadvantages of Syslog are that although it is an international standard, it is implemented differently in different systems, and the data it collects is raw.
Figure 10-8 shows an excellent commercial product known as Kiwi Syslog (http://www.kiwisyslog.com). It allows you to collect Syslog files from any Syslog-enabled security device to acommon Windows-based database file, where it can be queried and analyzed. Kiwi Enterprises offers more than a dozen Syslog-based tools and utilities, many of them free.
Figure 10-8: Kiwi Syslog collecting events from a honeypot system
An open-source utility called the Eventlog to Syslog Utility (https://engineering.purdue.edu/ECN/Resources/Documents/UNIX/evtsys) will copy Windows event log messages to remote Syslog servers. This is nice because, even though most security logging utilities can write to aSyslog server, Windows Event Viewer isn’t one of them. The open-source utility allows security logs to be consolidated in one centralized location. EventReporter (http://www.eventreporter.com) is a commercial alterative. You can even centralize Syslog events to aWindows computer (http://www.winsyslog.com).
In a real Windows honeypot environment, there are two basic log collection approaches. The first is to use a tool that retrieves and analyzes native Windows event logs. While this works great for Windows security logs, it’s not a good solution for most network devices. The second approach is to use Syslog and convert your Windows honeypot and other honeypot collection systems log events to Syslog. This way, at least you get the most commonality possible.
Unfortunately, neither approach addresses how to capture logs that don’t fit into these two categories. Windows includes dozens of other log files that might be useful to forensic analysis. Most are plain ASCII text files, but in different formats; others are in proprietary formats or binary representations. The best you can hope for is to analyze all the log files you plan to collect in your honeypot system and choose the method that collects the most logs to one single location, and then come up with alternate solutions for your miscellaneous log files.
There are several log collection programs able to collect ASCII-based logs, although you’ll still need to work out the different file formats when collecting them to a centralized database. SecurIT’s (http://iquebec.ifrance.com/securit/) LogAgent program is a real-time log monitoring tool that works with Event Viewer files and nearly any ASCII log file. Microsoft’s LogParser (http://www.logparser.com) is also capable of querying nearly any log file format.
Once all the data is collected (on a real-time ongoing basis), it needs to be sorted and prioritized. High-priority items need to be brought to the administrator’s attention, and low-priority messages should be filtered out and saved to a log file.
The trick is how to determine what is and isn’t high priority. Certainly, any initial contact of a honeypot is notable and should be considered high priority. This is so you can be around when the honeypot is first contacted in its first live test, and to make sure all systems, monitoring mechanisms, and data controls are working. Thereafter, it is up to the administrator what constitutes a high-priority item. A single successful login might be deemed high priority. A high-volume worm attack may be another. A system file modification is probably something critical the honeypot administrator wants to know about.
The key is varying levels of priority should be developed, and all the different collected events should be evaluated and triaged into the different priority levels. Low-priority events should be simply recorded. Never delete any security event log messages, as something innocuous might end up being important information at a later date. Sometimes, what isn’t happening at a particular point in time is as important as what did happen then. The log message collection system needs to prioritize events according to predefined rules.
Many data collection systems also contain data filtering abilities. All of Microsoft’s event log collection tools contain query features that can be used to sort collected events. Even the Event Viewer utility has simple query features. For example, Figure 10-9 shows the Event Viewer application filtering out all events (tens of thousands) except Event 682 (Successful Re-Connection to a Winstation), which shows successful RDP reconnections, including the login name used. A user could construct a simple filter querying for a handful of high-priority events.
Figure 10-9: Event Viewer filtering successful logins
Most of the third-party software programs mentioned as Windows event log collectors also have a query feature. A common query that could be constructed in most event log data filters is collecting all Warning and Failure audit events. An administrator should define which events should be considered high priority and create the appropriate query.
When reviewing Windows security audit files on a honeypot, every event log message not documented as baseline activity is important (because all activity is unauthorized), but the following are of particular interest:
Failure audit events, which show that someone tried to do something he or she did not have the permissions or privilege to do
Success audit events of critical file and Registry areas, like the autorun areas
Failed logins, which show login names attempted
Successful logins, which show which accounts were used for successful logins
Successful or failed policy changes, which show changes in audit policy
Successful or failed privilege changes, which offer evidence of a privilege escalation attack
Successful or failed system events
Although the centralized collection of data may seem like a challenge, it is really the easy part. The hard part is a result of no two OSs or security devices collecting and defining data in the same way, leading to aTower of Babel situation. For instance, competing antivirus scanners are notorious for giving the same malware two different names. So, if you were collecting Syslog entries from two different antivirus platforms protecting your honeypot, one may report the SQL Slammer worm as SQL.Slammer.worm.A and another as SQL.Worm.32.
Antivirus vendors have spent a decade trying to establish common names for identical viruses and worms. This failed effort was called the Computer Antivirus Researcher’s Organization (CARO) naming convention.
There is no easy solution for this problem, other than the administrator getting familiar with the formats and naming conventions passed by each security device. There are several competing standards trying to establish a common security descriptor language, including a few that involve XML. The Security Assertion Markup Language (SAML) (http://www.oasis-open.org/specs/index.php#samlv1.1) holds promise, but it is only one step toward a common convention for security log files.
In Windows 2000, event IDs 680 and 681 document successful and unsuccessful NTLM authentication events, respectively. In Windows 2003, event ID 681 is used for both successes and failures. Kerberos authentication events are tracked with different event IDs. Chapter 11 will cover the critical event IDs you should monitor on a real Windows honeypot system.
Other log files can be just as critical. For instance, any activity blocked by IIS is interesting. The IIS logs (located at %windir%\system32\logfiles\W3CSvc) will show you the commands attempted against your web server. Often, those logs are full of buffer overflow and directory transversal attacks, like these:
http://example.com/index.asp?something=..\..\..\..\WINNT\system32\cmd.exe?DIR+e:\WINNT\*.txt http://example.com/scripts/root.exe?/c+dir+c:\ http://example.com/scripts/..%c0%af../winnt/system32/cmd.exe?+/c+dir+c:\ 192.168.1.37 - - [18/May/2004:07:12:04 +0500] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 3141
IIS6 has a log called HTTPERR that tracks invalid and potentially malicious HTTP requests and connections.
If you didn’t know what these IIS exploit lines do, you could search for them using any Internet search engine. You’ll find information that explains their motives.
Data correlation is the collection of data into useful sets of information, instead of relying on one data point. A simple example is a port scan. When aWindows PC is port scanned, if the Windows Firewall (or the earlier version, Internet Connection Firewall) is turned on and the logging option enabled, the log will report drop packets at different TCP/IP ports. The Windows Firewall doesn’t have enough intelligence coded in to detect the multiple port probes at different ports as a port-scan attack. One dropped UDP port could be a mistake. Two hundred UDP probes is an attack-gathering tool. Also, a port probe against one computer is not as important as knowing that a hacker probed multiple hosts on your network.
Data correlation is also needed to mark common events with a common name. At what point do port scans become a port probe? Different logging systems determine port probes differently. One log system may not identify port scans as such at all, as is the case with the Windows Firewall. Another system may classify a port probe as any four sequential ports scanned in five seconds from a common source IP address. Still another may say a port probe is any six ports, no matter what the sequence, in ten seconds.
You want to use a data correlation tool that can make intelligent observations about the data. Unfortunately, Microsoft does not have any tools with this kind of intelligence, although I’m sure good programmers could write their own programs to interrogate the centralized collected logs. Data correlation tools are just starting to make their debut. Tools like the Honeynet Security Console and Security Event Managers are first-generation attempts.
Activeworx, Inc. (http://www.activeworx.com) offers some of the most exciting data collection tools available for the honeypot administrator. Activeworx’s commercial product, Activeworx Security Center (ASC), is a GUI security event manager that collects and correlates events and activity from honeypots, firewalls, VPNs, vulnerability assessment tools, Windows event logs, Syslog, Sebek, routers, tcpdump tools, Snort, IDSs, and web servers (Apache and IIS). ASC offers an intuitive interface for event management and forensic analysis. It has detailed built-in reporting, session analyzers, packet decoders, an IRC traffic decoder, a keystroke viewer, and event correlation. Information can be analyzed within ASC or extracted to another database or program.
Activeworx offers dumb-downed versions of ASC in their free offerings (http://www.activeworx.org): Honeynet Security Console (HSC), IDS Policy Manager, and a few other useful utilities. HSC has the same beautiful GUI as ASC, but is limited to a single device for each event type (for example, IDS vs. firewall vs. Windows system log). It doesn’t have as many pretty reports, diagrams, or graphs and is limited to MySQL databases (ASC supports Microsoft SQL Server, too). HSC also does not have a rules engine or an alerting module, and does not allow event detail viewing. Still, I highly recommend honeypot administrators give HSC a try, and if you like what you see, download a trial version of ASC.
All of this careful log planning should end in only the useful, critical, information being extracted and presented to the administrator. Critical events should be alerted to the administrator via some messaging mechanism, and the rest simply stored awaiting further analysis. Every event, no matter how trivial it seems, should be stored. A successful honeypot compromise will require hours, if not days, of careful forensic analysis.
In order for security events to be useful, they must be relevant to your environment. Relevancy is how likely a particular exploit attack will be successful in your environment. My web server honeypots are attacked with exploits destined for Apache servers, even when I run only IIS. Would you care if a hacker was trying Unix exploits against your Windows Server 2003 machines? Some system administrators would, and some would not. The last piece in the puzzle in event log management is to make sure that only relevant critical events get passed to the administrator. Events with a low relevancy should be noted and logged, but not passed in an alert.
Into this security log nightmare has evolved a new microcosm of vendors specializing in Security Event Management (SEM) or Security Incident Management (SIM). SEM/SIM vendors perform the event log management tasks as described in the preceding sections, and even merge vulnerability ratings, patch management, and risk exposure into the cycle. Vendors such as ArcSight (http://www.arcsight.com), netForensics (http://www.netforensics.com), Computer Associates (http://www.ca.com), and IBM (http://www.ibm.com) are heavily investing resources into their software, hardware, and managed service SEM/SIM products. See InfoWorld magazine’s excellent summary article on SIM/SEM at http://www.infoworld.com/infoworld/article/04/10/29/44FEbigsecure_1.html for more details.
While contracting a SEM/SIM to manage your honeypot system is probably overkill, you should include your honeypot system in any bid of similar managed services. A honeypot, if appropriately configured and placed, is as valuable as your perimeter firewall. It should be managed and addressed as you would any other professional security tool in your defense-in-depth strategy.
Logs must be secured against unauthorized access. In Windows, this means explicitly denying access to log files to nonadministrator accounts. Normally, Windows will let a regular end user read the log files (but not security logs on newer Windows platforms). But in a honeypot environment, you want to hide the logs from the hacker. You can change the default location of Microsoft’s security log (see http://www.microsoft.com/technet/security/topics/issues/w2kccadm/auditman/w2kadm26.mspx).
You may want to consider creating a fake log file that sits in the normal expected place. Place the real log file in another inconspicuous location. With any luck, the hacker will clear or delete the fake log, if he even cares enough to do that, and leave the real log file alone. The only downside is the fake log file would then need to be kept up-to-date in order to avoid suspicion.