Understanding System Calls | Hacking Ubuntu: Serious Hacks Mods and Customizations (ExtremeTech)

We write shellcode because we want the target program to function in a manner other than what was intended by the designer. One way to manipulate the program is to force it to make a system call or syscall . Syscalls are an extremely powerful set of functions that will allow you to access operating system-specific functions such as getting input, producing output, exiting a process, and executing a binary file. Syscalls allow you to directly access the kernel, which gives you access to lower-level functions. Syscalls are the interface between protected kernel mode and user mode. Implementing a protected kernel mode, in theory, keeps user applications from interfering with or compromising the OS. When a user mode program attempts to access kernel memory space, an access exception is generated, preventing the user mode program from directly accessing kernel memory space. Because some operating-specific services are required in order for programs to function, syscalls were implemented as an interface between regular user mode and kernel mode.

There are two common methods of executing a syscall in Linux. You can use either the C library wrapper, libc, which works indirectly, or execute the syscall directly with assembly by loading the appropriate arguments into registers and then calling a software interrupt. Libc wrappers were created so that programs can continue to function normally if a syscall is changed and to provide some very useful functions (such as our friend malloc). That said, most libc syscalls are very close representations of actual kernel system calls.

System calls in Linux are accomplished via software interrupts and are called with the int 0x80 instruction. When int 0x80 is executed by a user mode program, the CPU switches into kernel mode and executes the syscall function. Linux differs from other Unix syscall calling methods in that it features a fastcall convention for system calls, which makes use of registers for higher performance. The process works as follows :

The specific syscall function is loaded into EAX .
Arguments to the syscall function are placed in other registers.
The instruction int 0x80 is executed.
The CPU switches to kernel mode.
The syscall function is executed.

A specific integer value is associated with each syscall; this value must be placed into EAX . Each syscall can have a maximum of six arguments, which are inserted into EBX , ECX , EDX , ESI , EDI , and EPB , respectively. If more than the stock six arguments are required for the syscall, the arguments are passed via a data structure to the first argument.

Now that you are familiar with how a syscall works from an assembly level, let's follow the steps, make a syscall in C, disassemble the compiled program, and see what the actual assembly instructions are.

The most basic syscall is exit() . As expected, it terminates the current process. To create a simple C program that only starts up then exits, use the following code:

 main() {         exit(0); }

Compile this program using the static option with gccthis prevents dynamic linking, which will preserve our exit syscall.

 gcc static o exit exit.c

Next, disassemble the binary.

 [slap@0day root] gdb exit GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you  are welcome to change it and/or distribute copies of it under certain  conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.  Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (gdb) disas _exit Dump of assembler code for function _exit: 0x0804d9bc <_exit+0>:   mov    0x4(%esp,1),%ebx 0x0804d9c0 <_exit+4>:   mov  [slap@0day root] gdb exit GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you  are welcome to change it and/or distribute copies of it under certain  conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.  Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (gdb) disas _exit Dump of assembler code for function _exit: 0x0804d9bc <_exit+0>:   mov    0x4(%esp,1),%ebx 0x0804d9c0 <_exit+4>:   mov    $0xfc,%eax 0x0804d9c5 <_exit+9>:   int    $0x80 0x0804d9c7 <_exit+11>:  mov    $0x1,%eax 0x0804d9cc <_exit+16>:  int    $0x80 0x0804d9ce <_exit+18>:  hlt 0x0804d9cf <_exit+19>:  nop End of assembler dump. 
 xfc,%eax 0x0804d9c5 <_exit+9>:   int  [slap@0day root] gdb exit GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you  are welcome to change it and/or distribute copies of it under certain  conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.  Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (gdb) disas _exit Dump of assembler code for function _exit: 0x0804d9bc <_exit+0>:   mov    0x4(%esp,1),%ebx 0x0804d9c0 <_exit+4>:   mov    $0xfc,%eax 0x0804d9c5 <_exit+9>:   int    $0x80 0x0804d9c7 <_exit+11>:  mov    $0x1,%eax 0x0804d9cc <_exit+16>:  int    $0x80 0x0804d9ce <_exit+18>:  hlt 0x0804d9cf <_exit+19>:  nop End of assembler dump. 
 x80 0x0804d9c7 <_exit+11>:  mov  [slap@0day root] gdb exit GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you  are welcome to change it and/or distribute copies of it under certain  conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.  Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (gdb) disas _exit Dump of assembler code for function _exit: 0x0804d9bc <_exit+0>:   mov    0x4(%esp,1),%ebx 0x0804d9c0 <_exit+4>:   mov    $0xfc,%eax 0x0804d9c5 <_exit+9>:   int    $0x80 0x0804d9c7 <_exit+11>:  mov    $0x1,%eax 0x0804d9cc <_exit+16>:  int    $0x80 0x0804d9ce <_exit+18>:  hlt 0x0804d9cf <_exit+19>:  nop End of assembler dump. 
 x1,%eax 0x0804d9cc <_exit+16>:  int  [slap@0day root] gdb exit GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you  are welcome to change it and/or distribute copies of it under certain  conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.  Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (gdb) disas _exit Dump of assembler code for function _exit: 0x0804d9bc <_exit+0>:   mov    0x4(%esp,1),%ebx 0x0804d9c0 <_exit+4>:   mov    $0xfc,%eax 0x0804d9c5 <_exit+9>:   int    $0x80 0x0804d9c7 <_exit+11>:  mov    $0x1,%eax 0x0804d9cc <_exit+16>:  int    $0x80 0x0804d9ce <_exit+18>:  hlt 0x0804d9cf <_exit+19>:  nop End of assembler dump. 
 x80 0x0804d9ce <_exit+18>:  hlt 0x0804d9cf <_exit+19>:  nop End of assembler dump.

If you look at the disassembly for exit , you can see that we have two syscalls. The value of the syscall to be called is stored in EAX in lines exit+4 and exit+11 .

 0x0804d9c0 <_exit+4>:   mov  0x0804d9c0 <_exit+4>:   mov    $0xfc,%eax  0x0804d9c7 <_exit+11>:  mov    $0x1,%eax 
 xfc,%eax  0x0804d9c7 <_exit+11>:  mov  0x0804d9c0 <_exit+4>:   mov    $0xfc,%eax  0x0804d9c7 <_exit+11>:  mov    $0x1,%eax 
 x1,%eax

These correspond to syscall 252, exit_group() , and syscall 1, exit() . We also have an instruction that loads the argument to our exit syscall into EBX . This argument was pushed onto the stack previously, and has a value of zero.

 0x0804d9bc <_exit+0>:   mov    0x4(%esp,1),%ebx

Finally, we have the two int 0x80 instructions, which switch the CPU over to kernel mode and make our syscalls happen.

 0x0804d9c5 <_exit+9>:   int  0x0804d9c5 <_exit+9>:   int    $0x80 0x0804d9cc <_exit+16>:  int    $0x80 
 x80 0x0804d9cc <_exit+16>:  int  0x0804d9c5 <_exit+9>:   int    $0x80 0x0804d9cc <_exit+16>:  int    $0x80 
 x80

There you have it, the assembly instructions that correspond to a simple syscall, exit() .