13.6 Integer Overflows

The term integer overflow is often misleading. An integer overflow is simply a delivery mechanism for a stack, heap, or static overflow to occur (depending on where the integer ends up in memory).

Arithmetic calculations are often performed on integers to calculate many things, such as the amount of data to be received from the network, the size of a buffer, etc. Some calculations are vital to the logic of a program, and if they result in erroneous values, the program's logic may be severely corrupted or hijacked completely.

Calculations can sometimes be made to give incorrect results because the result is simply too big to be stored in the variable to which it is assigned. When this happens, the lowest part of the result is stored, and the rest (which doesn't fit in the variable) is simply discarded, as demonstrated here:

int a = 0xffffffff; int b = 1; int r = a + b;

After this code has executed, r should contain the value 0x100000000. However, this value is too big to hold as a 32-bit integer, so only the lowest 32 bits are kept and r is assigned the value 0.

This section concentrates on situations in which these incorrect calculations can be made to occur and some ways they can be used to bypass security. Usually the number provided is either too large, negative, or both.

13.6.1 Heap Wrap-Around Attacks

Programs often dynamically allocate buffers in which to store user-supplied data, especially if the amount of data sent varies. For example, a user sends a 2-KB file to a server, which allocates a 2-KB buffer and reads from the network into the buffer. Sometimes, the user will tell the program how much data she is going to send, so the program calculates the size of the buffer needed. Example 13-8 contains a function that allocates enough room for an array on the heap (of length len integers).

Example 13-8. Code containing an integer overflow bug

int myfunction(int *array, int len) {     int *myarray, i;     myarray = malloc(len * sizeof(int));     if(myarray == NULL)     {         return -1;     }     for(i = 0; i < len; i++)     {         myarray[i] = array[i];     }     return myarray; }

The calculation to find the size of len is the number of integers to be copied, multiplied by the length of an integer. This code is vulnerable to an integer overflow, which can cause the size of the buffer allocated to be much smaller than is needed. If the len parameter is very large (for example 0x40000001), the following calculation will happen:

length to allocate = len * sizeof(int)                    = 0x40000000 * 4                    = 0x100000004

0x100000004 is too big to store as a 32-bit integer, so the lowest 32 bits are used, truncating it to 0x00000004. This means that malloc( ) will allocate only a 4-byte buffer, and the loop to copy data into the newly allocated array will write way past the end of this allocated buffer. This results in a heap overflow (which can be exploited in a number of ways, depending on the heap implementation).

A real-life example of an integer overflow is the challenge-response integer overflow in OpenSSH 3.3 (CVE-2002-0639). Example 13-9 shows the code that is executed when a user requests challenge-response authentication.

Example 13-9. The vulnerable OpenSSH 3.3 code

nresp = packet_get_int( ); if (nresp > 0) {     response = xmalloc(nresp * sizeof(char*));     for (i = 0; i < nresp; i++)         response[i] = packet_get_string(NULL); }

packet_get_int( ) returns an integer read from the client, and packet_get_string( ) returns a pointer to a buffer on the heap containing a string read from the client. The user can set nresp to be any value, effectively allowing the user to completely control the size of the buffer allocated for response, and thus overflow it.

In this case a heap overflow occurs, resulting in a function pointer being overwritten. By carefully choosing the size of the buffer, an attacker can allocate it at a memory address below a useful function pointer. After overwriting the function pointer with the address of the shellcode, it is executed when the pointer is used.

13.6.2 Negative-Size Bugs

Sometimes an application needs to copy data into a fixed-size buffer, so it checks the length of the data to avoid a buffer overflow. This type of check ensures secure operation of the application, so bypassing such a check can have severe consequences. Example 13-10 shows a function that is vulnerable to a negative-size attack.

Example 13-10. A negative-size bug in C

int a_function(char *src, int len) {     char dst[80];     if(len > sizeof(buf))     {         printf("That's too long\n");             return 1;     }     memcpy(dst, src, len);     return 0; }

A quick look suggests that this function is indeed secure: if the input data is too large to fit in the buffer, it refuses to copy the data and returns immediately. However, if the len parameter is negative, the size check will pass (because any negative value is less than 80), and the copy operation will take place. When memcpy( ) is told to copy, for example, -200 bytes, it interprets the number -200 as an unsigned value, which by definition can't be negative.

The hexadecimal representation of -200 is 0xffffff38, so memcpy( ) copies 4,294,967,096 bytes of data (0xffffff38 in decimal) from src into dst, resulting in a buffer overflow and inevitable program crash.

Some implementations of memcpy( ) allow you to pass negative values for the length to be copied and still not copy so much data that the program dies before you can do something useful. The memcpy( ) supplied with BSD-derived systems can be abused in this manner, because you can force it to copy the last 3 bytes of the buffer before copying the rest of the buffer. It does this because copying whole words (4 bytes) onto whole word boundaries can be done very quickly, but copying onto nonword-aligned addresses (i.e., addresses that aren't multiples of 4) is comparatively slow. It therefore makes sense to copy any odd bytes first, so that the remainder of the buffer is word-aligned and can be copied quickly.

A problem arises, however, because after copying the odd bytes, the length to copy is reread from the stack and used to copy the rest of the buffer. If you can overwrite part of this length value with your first 3 bytes, you can trick memcpy( ) into copying a much smaller amount of data and not induce a crash.

Negative-size bugs are often difficult to exploit because they relying on peripheral issues (such as memcpy( ) use in BSD-derived systems) for successful exploitation, as opposed to a program crash. For further technical details of integer overflows and exploitation methods, please see the following papers:

http://www.phrack.org/phrack/60/p60-0x0a.txt

http://fakehalo.deadpig.org/IAO-paper.txt

http://www.fort-knox.org/thesis.pdf