12.13 Detecting Unix Debuggers

12.13.1 Problem

You need to prevent someone from debugging a Unix binary.

12.13.2 Solution

Single-stepping through code in a Unix environment causes a SIGTRAP to be sent to the process. The debugger captures this signal and allows the user to examine the state of the process before continuing execution. By installing a SIGTRAP handler and sending itself a SIGTRAP, the process can determine whether it is being debugged.

12.13.3 Discussion

The spc_trap_detect( ) function is used to install a signal handler to catch trap signals sent to the target, then issue a trap signal. The SPC_DEBUGGER_PRESENT macro checks the num_traps counter managed by the trap signal handler; if the counter is zero, a debugger is capturing the trap signals and is not sending them to the process.

#include <stdio.h> #include <signal.h>     #define SPC_DEBUGGER_PRESENT (num_traps =  = 0) static int num_traps = 0;     static void dbg_trap(int signo) {   num_traps++; }   int spc_trap_detect(void) {   if (signal(SIGTRAP, dbg_trap) =  = SIG_ERR) return 0;   raise(SIGTRAP);   return 1; }

The following example demonstrates the use of spc_trap_detect( ) to initialize the debugger detection, and SPC_DEBUGGER_PRESENT to check for the presence of a debugger:

int main(int argc, char *argv[  ]) {   int x;       spc_trap_detect(  );   for (x = 0; x < 10; x++) {     if (SPC_DEBUGGER_PRESENT) printf("being debugged!\n");     else printf("y\n");   }         return(0); }

This detection method is not particularly effective because most Unix debuggers allow the trap signal to be sent through to the process; however, tools that automatically single step through their targets (to record system calls, data access, etc.) will be detected using this method.

Most Unix debuggers are based on the ptrace system service, which is an interface to process control services in the kernel. ptrace-based debuggers were designed with source code debugging in mind, so they are incapable of dealing with hostile code. Detecting a ptrace debugger is simple, and the technique is well-known: ptrace prevents a process that is currently being traced from tracing itself or another process, so an attempt to ptrace another process will always fail if the current process is being traced. The following code demonstrates how to detect a ptrace-based debugger by creating a child process and attempting to attach to it.

#include <sys/types.h> #include <errno.h> #include <signal.h> #include <stdlib.h> #include <unistd.h> #include <sys/ptrace.h> #include <sys/wait.h> int spc_detect_ptrace(void) {   int   status, waitrc;   pid_t child, parent;   parent = getpid();   if (!(child = fork())) {     /* this is the child process */     if (ptrace(PT_ATTACH, parent, 0, 0)) exit(1);     do {       waitrc = waitpid(parent, &status, 0);     } while (waitrc == -1 && errno == EINTR);     ptrace(PT_DETACH, parent, (caddr_t)1, SIGCONT);     exit(0);   }   if (child == -1) return -1;   do {     waitrc = waitpid(child, &status, 0);   } while (waitrc == -1 && errno == EINTR);   return WEXITSTATUS(status); }

The state of the art in anti-debugging on Unix is not very advanced, because all widely used Unix debuggers are based on ptrace and do not require any special tricks to detect; generally speaking, any method that detects or counters ptrace should succeed. It is important to realize, however, that calls to ptrace( ) can be replaced with nop instructions in the binary to defeat the debugger detection, so take care to disguise them. For example, by using the system call interface instead of the C interface, the ptrace( ) system call can also be hooked at the kernel level to force a successful return.

12.13.4 See Also

"Linux Anti-Debugging Techniques" by Silvio Cesare (the techniques listed here were published in that 1999 paper, http://vx.netlux.org/lib/vsc04.html.