15.4 Some Potential Attacks

JVM security is often discussed in terms of the Java language. The Java compiler rejects programs that attempt some kinds of attacks, such as reading private variables or casting an object reference to an invalid type. However, some attackers may try to bypass the compiler errors by writing in Oolong instead of Java or by writing bytecodes directly. In this section, we show how the Java virtual machine thwarts some potential attacks on system security.

15.4.1 Implementation Details

Examples in this section frequently involve private methods and fields of standard classes. Private methods and fields are not necessarily consistent from JVM implementation to JVM implementation; the private keyword denotes parts of the code that are implementation dependent.

Security must not depend on the fact that the attacker does not know the implementation details of the system. An attacker may guess what private methods are being used, since the attacker can easily obtain a copy of the same virtual machine implementation the victim is using. If the attack is through an applet, then the attacker can reasonably assume that the victim is using one of the popular Java-supporting web browsers and may use attacks using private implementation details of these browsers.

Fortunately, the JVM provides a strong base for making the implementation secure, even if the attacker knows how the system was implemented. This section shows how some potential attacks are thwarted, either by the verification algorithm or by the runtime system, which depends on the verifier.

15.4.2 Protecting the Security Manager

The system has exactly one security manager, which is kept by the java.lang.System class. It is retrieved with getSecurityManager, a static method in java. lang.System.

Initially, there is no security manager, and getSecurityManager returns null. The security manager can be set with setSecurityManager:

 AppletSecurityManager asm = new AppletSecurityManager(); System.setSecurityManager(asm); 

The first time setSecurityManager is called, it sets the security manager. After that, all calls to setSecurityManager will throw a SecurityException without setting the manager. This prevents anybody else, such as a malicious applet, from altering the security manager. It is the job of the browser to set the security manager before loading any applets.

Each JVM implementation can implement the System class differently. In subsequent sections, we assume that it stores the security manager in a private field called security:

 package java.lang; public final class System {    private static SecurityManager security;    public static SecurityManager getSecurityManager()    {       return security;    }    public static setSecurityManager(SecurityManager sm)    {       if(security == null)          throw new SecurityException             ("SecurityManager already set");       security = sm;    } } 

The integrity of the security field is critical, since an applet that could set that field could control the security policy of the system. This makes security a likely target for attack.

15.4.3 Bypassing Java Security

The most straightforward attack would be to try to set the security field. The attacker's applet might be written in Java:

 public class NastyApplet extends Applet {    void attack()    {       System.security = null;       // The security manager is null, so everything is       // permitted       // Put code to wreak havoc here    } } 

When this class is compiled, the Java compiler will notice that the security field is private to the System class and refuse to compile the file. This would not stop a determined attacker, who would proceed to rewrite the applet in Oolong:

 .class public NastyApplet .super java/applet/Applet .method attack()V aconst_null               ; Install null as the                           ; security manager putstatic java/lang/System/security Ljava/lang/SecurityManager; ;; Wreak havoc .end method 

This is a valid Oolong program. Once you have downloaded the applet into your web browser, the attacker's applet will try to use the attack method to cause trouble.

The JVM prevents this attack from succeeding by running the verification algorithm on the class when it is loaded. All the information needed about what fields are accessed is present in the code. That is, it isn't necessary to run the applet to find out which fields it may try to access. The verification algorithm finds that the security field is private, so this code cannot access it. The verification algorithm causes the system to refuse to load this class, thwarting the attack before the applet starts to run.

The following method is perfectly safe because the potentially dangerous code can never actually be executed:

 .method attack()V return ; Just kidding. There's no way to execute the next line, since ; the method always returns before it gets here aconst_null               ; Install null as the                           ; security manager putstatic java/lang/System/security Ljava/lang/SecurityManager; return .end method 

The verification algorithm rejects this applet just because of the presence of this security-violating instruction, even though the dangerous code is unreachable.

There is no way to bypass the verification step, since it occurs automatically when the resolveClass method is invoked in the ClassLoader that loads the class. The class is unusable until it has been resolved. When it is resolved, the verification algorithm catches the attack and refuses to load the class. If it is never loaded, then it can never be executed, so the attack is avoided.

15.4.4 Using Unconstructed Objects

Some classes depend on their constructors to control who may create instances. For example, the setSecurityManager method in java.lang.System is public, which permits anybody to set the security manager.

An applet might try to install its own security manager:

 class WimpySecurityManager extends SecurityManager {    // Override the methods so that they don't do anything } class MaliciousApplet extends Applet {    SecurityManager mySecurityManager = new       WimpySecurityManager();    System.setSecurityManager(mySecurityManager);    // Now I can do anything I want! } 

This attack won't work, even though setSecurityManager is public. The reason is that you're not allowed to create your own SecurityManager once a security manager has been installed. The SecurityManager's constructor has code to check if there already is a security manager installed:

 public class SecurityManager {    protected SecurityManager()    {       if (System.getSecurityManager() != null)           throw new SecurityException(              "security manager already installed.");       // Continue constructing the security manager    } } 

The verification algorithm requires that each constructor call a superclass constructor. Since this is the only constructor for SecurityManager, the WimpySecurityManager must call it. In the example, the Java compiler creates a default constructor for WimpySecurityManager, which calls the constructor for SecurityManager. That constructor finds that a SecurityManager has already been created and throws a SecurityException. Since the system won't permit use of an unconstructed object, the attacker can never install the WimpySecurityManager as the security manager of the system.

Suppose, however, that the attacker wrote a constructor for WimpySecurityManager that didn't call the constructor:

 .class WimpySecurityManager .super java/lang/SecurityManager .method public <init>()V return                ; Don't call the superclass constructor! .end method 

The JVM verification algorithm rejects WimpySecurityManager, since one of the things the verification algorithm checks is that each constructor calls a superclass constructor somewhere in the body of the constructor.

The attacker might try to fool the verification algorithm by hiding the call:

 .method public <init>()V goto end                  ; Skip over the superclass constructor aload_0 invokespecial java/lang/SecurityManager/<init> ()V end: return .end method 

The verification algorithm discovers that the superclass <init> method is not always called, and it rejects this class. Another attempt might involve hiding the call behind an if instruction:

 .method public <init>(I)V .limit locals 2 iload_1 ifeq end              ; Skip the superclass constructor if aload_0               ; the argument is nonzero invokespecial java/lang/SecurityManager/<init> ()V end: return .end method 

This constructor calls the superclass constructor if the argument is nonzero; otherwise, it skips it. However, any path through the method must call the superclass constructor, or the verification algorithm rejects it. The verification algorithm does not have to run the code to determine whether the constructor will be called. All the verification algorithm has to do is to prove that there is a way through the method that does not involve the superclass constructor call, to provide sufficient grounds to reject the entire class.

This must be true of all constructors. Even if you provide a valid constructor as well as an invalid one, the verification algorithm still rejects the class. The goal is that the verification algorithm must be able to prove that the superclass constructor is called no matter which constructor is called and no matter which arguments are provided.

15.4.5 Invalid Casts

An attacker would gain a lot of information if the JVM permitted invalid casts, especially with reference types. An attacker may assume, often correctly, that a reference is represented internally by a pointer into memory. The attacker may be able to guess how the object is stored. If the system could be fooled into thinking that the object had a different class, it might permit a program to read or write fields on an object where access would otherwise be denied.

For example, a web browser might keep a copy of the user's private key for signing messages. The Message class is public to permit programs to sign messages, but the private key should not be revealed, even to those programs allowed to do the signing.

 public class Message {    private PrivateKey private_key;    /** Sign the data, but don't reveal the key    public byte[] sign(byte[] data); } 

An attacker might try to get the private key by creating a class like Message but with different permissions:

 public class SneakyMessage {    public PrivateKey private_key; } 

This class is likely to have the same layout in memory as Message. If the attacker could cast a Message into a SneakyMessage, then the value of private_key would appear to be public.

Fortunately, this can't happen. This attack, written in Java, might look like this:

 void attack(Message m) {    SneakyMessage hack = (SneakyMessage) m;    // Read the private_key field from hack } 

The Java compiler rejects this class, pointing out that the cast from Message to SneakyMessage will never succeed. The attacker might try to bypass the compiler like this:

 .class SneakyApplet .super java/applet/Applet .method attack(LMessage;)V aload_1               ; Variable 1 contains a Message                       ; Try to get the private key out getfield SneakyMessage/private_key Ljava/security/PrivateKey; ;; Use the private key .end method 

The verification algorithm rejects this code, even though the private_key in SneakyMessage is public, since it knows that a SneakyMessage is not a Message and vice versa. Since variable 1 contains a Message, not a SneakyMessage, the getfield instruction is invalid.

Some people are surprised that the verification algorithm makes this determination, since it is not always possible to prove that variable 1 contains a Message. This is possible because it is not the verification algorithm's job to prove that variable 1 contains a nonsneaky Message before rejecting this class. Rather, the verification algorithm tries to prove that variable 1 does contain a SneakyMessage in order to accept the getfield instruction. This is not true, since variable 1 is initialized to a Message and the variable is never altered, so the verification algorithm rejects the code.

It is easy to create code leading up to the getfield where it is impossible to be sure what the top of the stack is without actually running the code:

 .method attack(LMessage;LSneakyMessage;I) iload_3               ; Push the number ifeq attack           ; If it is 0, then try the attack    aload_2            ; Push the sneaky message    goto continue      ; Continue with the attack attack:    aload_1            ; Push the message, which we try to continue:             ; treat as a sneaky message getfield SneakyMessage/private_key Ljava/security/PrivateKey; 

At the last instruction, the value on top of the stack may or may not be SneakyMessage, depending on the value of the third argument to the method. The verification algorithm will unify two stack pictures at continue: One with a SneakMessage and one with a Message. The unification is Object. This makes the getfield instruction invalid, so the class is rejected.

15.4.6 Changing the Class of a Reference

The checkcast instruction changes the verification algorithm's perception of the class of an object. The attack method could be rewritten using checkcast:

 .method attack(LMessage;)V aload_1                    ; Variable 1 contains a Message checkcast SneakyMessage    ; Ask the verifier to believe it is                            ; a SneakyMessage                            ; Try to get the private key out getfield SneakyMessage/private_key Ljava/security/PrivateKey; ;; Use the private key .end method 

The verification algorithm approves this code, and the applet gets a chance to run. However, the attack is still ineffective, because the checkcast instruction checks the actual type of the value on top of the stack when the program is running. Since the value on top of the stack is really a Message, and a Message is not a SneakyMessage, the checkcast instruction throws an exception. The exception transfers control away from the getfield instruction, so the information is not compromised.

The checkcast instruction does not affect the underlying object. It only changes the verification algorithm's perception of the object as the class is being loaded. The verification algorithm attempts to prove that the program is safe. The checkcast instruction tells the verification algorithm, "You cannot prove that this is safe, but if you check it out as the program runs, you will find that the cast will succeed."

As the program runs, each time checkcast is encountered the JVM checks to ensure that the class of the argument is really a SneakyMessage. If it fails, the JVM throws a ClassCastException. Because no Message can be a SneakyMessage, this code always causes the exception to be thrown.

There is one way for the code to get past the checkcast at runtime. If the Message is null, then the checkcast allows the program to proceed. However, this doesn't help the attacker, since any attempt to read fields from a null reference will be met with a NullPointerException.

15.4.7 Reading Uninitialized Fields

When an object is first created, it may be assigned a place in memory where something important used to be. For example, the web browser may have the user's password to some web site stored in a Connection object. The Connection object and the password string live somewhere in the system's memory, as in Figure 15.1.

Figure 15.1. A Connection object points to your password

Later, the garbage collector may move the Connection object. Things don't really move in a computer's memory; instead, they are copied to a new location and the old location is forgotten. Suppose a user creates a new object that is assigned to the memory space that the Connection object used to occupy. An attacker might hope that the memory looks like Figure 15.2. The first field in MyObject, field1, happens to fall in the same place as the password field of the Connection used to. Before field1 is initialized, it appears to contain a reference to the value that used to be the password field of the Connection object. Although it is unlikely that this would happen, it is not impossible, and the applet code could use this information in a malicious fashion.

Figure 15.2. Connection moved, and a new object in the same memory

Fortunately, the virtual machine prevents this attack by requiring that all fields be initialized before they are read. If the field does not have an explicit initializer, it is implicitly initialized to 0 for numeric fields or null for reference fields. This happens immediately after the object is created, before it can be used. Therefore, in reality the picture looks like Figure 15.3. Even though the new object occupies the same memory space as the Connection used to, it is unable to access the values that used to be there because field1 is initialized to null before the code has a chance to read the old value.

Figure 15.3. field1 is initialized immediately

15.4.8 Array Bounds Checks

Another kind of memory-based attack is to try to read past the ends of an array. To extend an earlier example, suppose that the user created an array of ten bytes that just happened to be placed next to an important place in memory (Figure 15.4). The first ten elements of the array are safe to be read and written. The location that would hold the eleventh byte, however, is the beginning of the password string. If an applet were to read beyond the end of the array, the password would be revealed.

Figure 15.4. Password object in memory immediately after an array

The only way to read those bytes is to use the baload instruction:

 bipush 10 newarray byte               ; Create an array of ten bytes bipush 10                   ; Try to read the eleventh byte baload 

Both the size of the array and the array element to be accessed are determined by operands on the operand stack, not by instruction arguments. This means that there is no way for the virtual machine to be certain what these values will be without running the program.

Therefore, this sort of error cannot be caught by the verification algorithm. Instead, this attack is stopped by the baload instruction. Each time the instruction is executed, the length of the array is checked to ensure that the element does not fall out of bounds. If it does, an ArrayIndexOutOfBoundsException is thrown.

The bounds check operation is part of the baload instruction and all the other array load instructions. The virtual machine does not depend on the program to check array bounds before accessing elements. This removes some responsibility from the programmer, while still ensuring that array bounds are respected.

15.4.9 Catching Exceptions

An attacker might try to catch exceptions that are intended to prevent harmful things from happening to the system. The attack would try to ignore the exceptions.

For example, it was shown earlier that the SecurityManager initializer does not permit a second SecurityManager to be created once one has been installed. If the initializer detects the existence of another security manager, it throws a SecurityException, which prevents the new SecurityManager from being used.

Suppose that the attacker catches the exception from the SecurityManager constructor within the constructor of WimpySecurityManager:

 .class WimpySecurityManager .method <init>()V .catch java/lang/SecurityException from begin to end using handler begin:    aload_0           ; Call the superclass constructor    invokevirtual java/lang/SecurityManager/<init>()V end:    return            ; If I'm allowed to invoke the                      ; constructor, then something is wrong    handler:          ; Since a SecurityException is thrown,                      ; control continues here       return         ; See if I can return with the object                      ; only partially constructed .end method 

The JVM refuses to load this code because it fails to pass the verification algorithm. One of the verification algorithm's goals in tracing through a constructor is that every return is on a path through a call to a superclass constructor. Because all the lines between begin and end are covered by an exception handler, any of them might throw an exception.

This means that there is a path through the method that does not invoke the constructor. The verification algorithm judges the method to be invalid, which means that the entire class is invalid. This class is rejected, and the system is safe.

15.4.10 Hidden Code

An attacker might try to hide bytecodes inside other bytecodes. Consider an innocuous-looking piece of Oolong code like this:

 sipush -19712 ineg 

This code could actually contain a potential attack hidden in the bytecodes. When assembled, the above code produces the following bytecodes:

 Location      Value      Meaning 0050          11         sipush 0051          b3               0xb300=-19712 0052          00 0053          74         ineg 

An attacker could generate a class file in which the next bytes were

 0054          a7         goto 0055          ff               0xfffd=-3 0056          fd 

The goto points to location 51, which is in the middle of the sipush instruction. It is not possible to write this code in Oolong, since the Oolong assembler will permit a label only between instructions, not in the middle of an instruction.

The attacker hopes that when the code reaches the instruction at location 54, it will attempt to go back three bytes to the byte at 51. If we interpret the bytes at location 51 as bytecodes, we get

 0051          b3         putstatic 0052          00                    0x0074=116 0054          74 

This is an interpretation of the bytes' attempts to read the field contained in the constant 116.

The attacker might make the constant 116 a Fieldref to a private field like security in java.lang.System, hoping to wipe out the security manager for the system. The attacker hopes that the JVM will not attempt to verify the bytecodes interpreted in this fashion. The attacker hopes that the system will assume that it has checked the code statically, and won't try to check whether or not the field is private at runtime.

There is nothing wrong with the bytes at locations 51 through 53, as long as they are interpreted properly. The problem is the code at 54, which tries to jump into the middle of an instruction. The JVM discovers that the code is attempting to branch into the middle of an instruction, and it will reject the class.

Of course, not all code like this is necessarily harmful. This code may just come from an overly clever programmer who attempted to shrink the code by reinterpreting it this way, and the code may be perfectly safe to execute. However, the verification algorithm never promised that it would accept all safe programs. It only promises to reject all that do not meet certain criteria, which includes all unsafe programs. Because this program does not follow the rules, it is rejected.