Detection Methods

These are a few of the methods used to detect rootkits:

• Compare a file list to a similar list generated by a known clean environment.

• Compare a file list to its corresponding NTFS Master File Table.

• Compare a registry scan to a similar scan generated by a known clean environment.

• Compare the registry to its corresponding disk images.

• Compare process and service lists using different system functions.

• Compare process and service lists using rootkit injection technology.

• Detect the use of alternate data streams.

• Compare the kernel system call table to its corresponding disk image.

• Confirm kernel system call table entries are within kernel boundaries.

• Compare kernel functions to their corresponding disk images.

• Compare INT 2E functionality to Windows NT functionality.

• Compare SwapContext processes to the process link list.

• Compare kernel memory to known rootkit signatures.

Several of the preceding comparisons are between known good and suspected bad environments. A known good environment is any operating system (not necessarily Windows) that does not share software components from the suspect operating system. This can be a fresh operating system booted from a CD or diskette, a partitioned operating system located on the same disk drive as the suspect operating system, or a remotely connected operating system. As long as no components are shared between the test system and the suspect operating system, list comparisons can reveal concealment techniques.

File comparison can detect file hiding by comparing a file list from a suspect operating system with a similar list generated by a known good operating system. This is a great way to catch file-hiding rootkits. The initial file list generated by the host operating system will not show files hidden by a rootkit, while the same list generated by a clean system will show all files. File hiding can also be detected by comparing file lists generated by different system calls, or by comparing a file list to its associated NTFS Master File Table. Any discrepancy between two such file lists would tend to indicate the presence of a rootkit.

Unfortunately, the vast number of files and directories required to run a modern operating system make file hiding unnecessary. Rootkit authors have at their disposal many techniques to load rootkits from many locations. With so many files, directories, registry keys, and loading techniques, and the ability to obfuscate using system service naming conventions, there is no real need to hide files, especially now that file-hiding detectors are freely available everywhere; so don’t expect to find any of the newer rootkits with file-hiding detectors.

Registry key hiding can also be detected by comparing a registry scan from a suspect operating system with a similar scan generated by a known good operating system. In addition, registry tampering can be detected by comparing the registry to its corresponding disk files. All that is needed for this comparison is an understanding of the registry file format. Parsing registry files (hives) to determine what should be in the registry and comparing the expected results to an actual registry scan will uncover simple registry key hiding techniques.

Unfortunately, as with files, there are simply too many places in the registry to hide. Some obfuscation may be required (I wouldn’t put the location of a device driver under HKLM\Software\MyRootkit) but the proliferation of registry hiding detectors has greatly curtailed the use of registry key hiding techniques in modern rootkits; so again, don’t expect to find any of the newer rootkits with registry key hiding detectors.

Detecting hidden processes and services is yet another way to uncover the presence of a rootkit. Some rootkit detectors actually employ rootkit technology to detect rootkits by injecting the kernel to restore the ability to detect processes and services. Comparing initial process and service lists to similar lists generated after kernel injection can reveal process and service hiding. Rootkit detectors can also create process and service lists using different operating system calls or create lists from both suspect operating systems and known good operating systems. In all cases, a discrepancy between two lists would tend to indicate the presence of a rootkit.

Unfortunately, some forms of process hiding can remain undetected, and many rootkits do not employ services, so these forms of rootkit detection will not uncover a growing number of modern rootkits.

The use of alternate data streams does not by itself guarantee the presence of a rootkit, but there are logical and statistical associations between rootkits and ADS. This makes ADS detection of questionable value by itself, but the indication of a possible rootkit is likely to spawn an investigation that will eventually reveal the reason for the use of ADS.

A file-tracking system might choose to tag tracked files with ADS. A video game manufacturer might choose to add copy prevention information to executables using ADS. A source-control system might keep track of versioning information using ADS. There can be valid uses for the technology, but statistically, the use of ADS is likely to be rootkit related.

Unfortunately, ADS detectors currently only test for data streams linked to file objects, leaving other forms of ADS linking open. In addition, any investigation into the use of an ADS can be thwarted by simple encryption or the addition of thousands of small alternate data streams throughout the operating system.

Kernel call table hooking can be detected by comparing the kernel system call table to the corresponding image on disk. The system file ntoskrnl.exe is used to create the system call table. All that is needed for this comparison is an understanding of how ntoskrnl.exe is parsed during system boot. Kernel call table hooking can also be detected by comparing system call table entries to boundary conditions. This can be accomplished by calling the exported ntdll.dll function ZwQuerySystemInformation and requesting SystemModuleInformation for ntoskrnl.exe. The data structure returned by this call contains the base address and size of the kernel module. Any system call table entry outside this boundary would tend to indicate the presence of a rootkit.

Unfortunately, as shown in Chapter 4, hooking can be placed in the kernel functions themselves using a trampoline hooking technique instead of modifying the kernel system call table. Furthermore, a growing number of applications hook the system call table for viable reasons. Determining which hooks are needed and which hooks should be removed can be a difficult task.

Kernel function (trampoline) hooking can be detected by searching for unexpected jumps at or near the beginning of kernel functions. This method requires the rootkit detector to first rebuild the system call table and then follow each call, checking for immediate, unconditional jumps. The kernel functions in memory can also be compared to the code on disk, provided relocateable addresses are skipped.

Unfortunately, trampoline jumps don’t have to be immediate or unconditional, and there is nothing to prevent a rootkit from modifying the kernel image, ntoskrnl.exe, seen by calling processes. There are too many operating system versions, too many service packs, and too many patches to easily detect image tampering, although, just like file, registry, process, and system call table tampering, image tampering will one day become a viable rootkit detection technique.

Using a gated interrupt to perform system operations can bypass the hooks inserted by some rootkits. This is a great way to detect user mode hooking, as Windows NT functions are completely bypassed using the 2E interrupt. File lists, directory lists, registry keys, and registry values can all be scanned with and without the use of Windows NT functions. Because both methods call the same kernel functions, there should be no difference between the two scanning techniques. Any discrepancy would tend to indicate the presence of a rootkit.

Unfortunately, user mode hooking can be conditional based upon the name of the process loading the image, purposely not hooking known detectors while hooking everything else. This avoidance technique can easily leave a false sense of security. In addition, user mode hooking of Windows NT functions rarely covers all possible rootkit concealment requirements. This creates a compelling reason for rootkits to use some other concealment technique.

Comparing the process blocks passed through SwapContext to the process blocks in the doubly linked process list can detect the type of process hiding used in this book. SwapContext is called whenever an old process is swapped out and a new process is swapped in. Following the KTHREAD of the swapped-in process (located in the EDI register) will lead to the EPROCESS block for the new process. That process block should also be somewhere in the doubly linked process list. Any discrepancy here is most definitely a rootkit. Figure 13-1 shows the SwapContext Process Hiding Detection.

Figure 13-1

Unfortunately, SwapContext can be hooked to check for the process block of the rootkit. When the rootkit’s process is being swapped in, the hook can temporarily link the rootkit process entry into the process list before calling the original function, and then unlink the process entry from the process list after calling the original function. This will give any rootkit detection software the impression that the rest of the operating system can see the hidden process.

Scanning kernel memory can also be used to detect known rootkits. This is a complicated task, requiring the scanner to either switch to the context of the process being scanned or perform virtual-to-physical address translations. Once accomplished, however, existing rootkits can be detected by comparing scanned memory to known rootkit signatures.

Unfortunately, memory scanning will not detect newer rootkits; and without a means to circumvent process hiding, memory scanning might not even detect known rootkits with established signatures.

By now, you’ve probably noticed the pattern here: a paragraph defining a rootkit detection technique, followed by a paragraph detailing ways to avoid detection by that technique. This list has grown considerably in recent years, but the format has not. For every new detection technique, there is (or will soon be) a corresponding avoidance technique. This progressive evolution of rootkit and anti-rootkit technology is expected to continue indefinitely.

One by-product of this struggle for dominance is that rootkits will become old faster. By “old” I mean “easily detectable by readily available anti-rootkit software.” This is not to say there will ever be a signature-based rootkit prevention system like those employed to prevent viruses, but a detector-based rootkit prevention system is probable. Like Professional Rootkits, Professional Rootkit Detectors, created by teams of professionals, will dominate the anti-rootkit market in the near future. This will ensure that new rootkit hiding techniques will be countered quickly and that the developed detectors will be distributed immediately to a vast demographic.

Unfortunately, the underlying purpose of a rootkit is stealth, whereas the underlying purpose of a virus is propagation. This makes rootkit detection more difficult than virus detection, greatly increasing the time between the use of a new rootkit hiding technique and eventual rootkit detection.

Another by-product of the struggle for dominance between rootkits and rootkit detectors, this time in favor of the rootkit designer, is the creation of a broad spectrum of readily available tools that can be used to validate rootkit implementation. By testing a rootkit against many rootkit detectors, the rootkit designer not only ensures initial stealth, but also future stealth because existing rootkit detectors will often implement new functionality already found in other rootkit detectors.