10.1 Math Utilities

11.2.2 File Streams

Java provides two specialized streams for reading from and writing to files in the filesystem: FileInputStream and FileOutputStream . These streams provide the basic InputStream and OutputStream functionality applied to reading and writing files. They can be combined with the filter streams described earlier to work with files in the same way we do other stream communications.

Because FileInputStream is a subclass of InputStream , it inherits all standard InputStream functionality for reading a file. FileInputStream provides only a low-level interface to reading data, however, so you'll typically wrap it with another stream, such as a DataInputStream .

You can create a FileInputStream from a String pathname or a File object:

FileInputStream in = new FileInputStream("/etc/passwd");

When you create a FileInputStream , the Java runtime system attempts to open the specified file. Thus, the FileInputStream constructors can throw a FileNotFoundException if the specified file doesn't exist, or an IOException if some other I/O error occurs. You must catch these exceptions in your code. When the stream is first created, its available() method and the File object's length() method should return the same value. To save resources, you can call the close() method when you are done with the file.

To read characters from a file, you can wrap an InputStreamReader around a FileInputStream . If you want to use the default character-encoding scheme, you can use the FileReader class instead, which is provided as a convenience. FileReader works just like FileInputStream , except that it reads characters instead of bytes and wraps a Reader instead of an InputStream .

The following class, ListIt , is a small utility that sends the contents of a file or directory to standard output:

//file: ListIt.java
import java.io.*;  
  
class ListIt {  
    public static void main (String args[]) throws Exception {  
        File file =  new File(args[0]);  
  
        if (!file.exists()  !file.canRead()) {  
            System.out.println("Can't read " + file);  
            return;  
        }  
  
        if (file.isDirectory()) {  
            String [] files = file.list();  
            for (int i=0; i< files.length; i++)  
                System.out.println(files[i]);  
        } else  
            try {  
                FileReader fr = new FileReader (file); 
                BufferedReader in = new BufferedReader(fr); 
                String line; 
                while ((line = in.readLine()) != null) 
                System.out.println(line); 
            }  
            catch (FileNotFoundException e) { 
                System.out.println("File Disappeared");  
            }  
    }  
}

ListIt constructs a File object from its first command-line argument and tests the File to see whether it exists and is readable. If the File is a directory, ListIt outputs the names of the files in the directory. Otherwise, ListIt reads and outputs the file.

FileOutputStream is a subclass of OutputStream , so it inherits all the standard OutputStream functionality for writing to a file. Just like FileInputStream though, FileOutputStream provides only a low-level interface to writing data. You'll typically wrap another stream, such as a DataOutputStream or a PrintWriter , around the FileOutputStream to provide higher-level functionality.

You can create a FileOutputStream from a String pathname or a File object. Unlike FileInputStream , however, the FileOutputStream constructors don't throw a FileNotFoundException . If the specified file doesn't exist, the FileOutputStream creates the file. The FileOutputStream constructors can throw an IOException if some other I/O error occurs, so you still need to handle this exception.

If the specified file does exist, the FileOutputStream opens it for writing. When you subsequently call the write() method, the new data overwrites the current contents of the file. If you need to append data to an existing file, you can use a form of the constructor that accepts an append flag:

FileInputStream fooOut = new FileOutputStream(fooFile);  
FileInputStream pwdOut = new FileOutputStream("/etc/passwd", true);

Another way to append data to files is with RandomAccessFile , as we'll discuss shortly.

To write characters (instead of bytes) to a file, you can wrap an OutputStreamWriter around a FileOutputStream . If you want to use the default character-encoding scheme, you can use instead the FileWriter class, which is provided as a convenience. FileWriter works just like FileOutputStream , except that it writes characters instead of bytes and wraps a Writer instead of an OutputStream .

The following example reads a line of data from standard input and writes it to the file /tmp/foo.txt :

String s = new BufferedReader(new InputStreamReader(System.in)).readLine();  
File out = new File("/tmp/foo.txt");  
FileWriter fw = new FileWriter (out);  
PrintWriter pw = new PrintWriter(fw)  
pw.println(s); 
fw.close();

Notice how we wrapped a PrintWriter around the FileWriter to facilitate writing the data. Also, to be a good filesystem citizen, we've called the close() method when we're done with the FileWriter .

11.2.3 The java.io.RandomAccessFile Class

The java.io.RandomAccessFile class provides the ability to read and write data at a specified location in a file. RandomAccessFile implements both the DataInput and DataOutput interfaces, so you can use it to read and write strings and primitive types. In other words, RandomAccessFile defines the same methods for reading and writing data as DataInputStream and DataOutputStream . However, because the class provides random, rather than sequential, access to file data, it's not a subclass of either InputStream or OutputStream .

You can create a RandomAccessFile from a String pathname or a File object. The constructor also takes a second String argument that specifies the mode of the file. Use r for a read-only file or rw for a read-write file. Here's how we would start to create a simple database to keep track of user information:

try {  
    RandomAccessFile users =
        new RandomAccessFile("Users", "rw")
 } catch (IOException e) { ... }

When you create a RandomAccessFile in read-only mode, Java tries to open the specified file. If the file doesn't exist, RandomAccessFile throws an IOException . If, however, you're creating a RandomAccessFile in read-write mode, the object creates the file if it doesn't exist. The constructor can still throw an IOException if another I/O error occurs, so you still need to handle this exception.

After you have created a RandomAccessFile , call any of the normal reading and writing methods, just as you would with a DataInputStream or DataOutputStream . If you try to write to a read-only file, the write method throws an IOException .

What makes a RandomAccessFile special is the seek() method. This method takes a long value and uses it to set the location for reading and writing in the file. You can use the getFilePointer() method to get the current location. If you need to append data to the end of the file, use length() to determine that location, then seek() to it. You can write or seek beyond the end of a file, but you can't read beyond the end of a file. The read() method throws an EOFException if you try to do this.

Here's an example of writing some data to a user database:

users.seek(userNum * RECORDSIZE);  
users.writeUTF(userName);  
users.writeInt(userID);

Of course, in this nave example we assume that the String length for userName , along with any data that comes after it, fits within the specified record size.

11.2.4 Applets and Files

Unless otherwise restricted, a Java application can read and write to the host filesystem with the same level of access as the user running the Java interpreter. For security reasons, untrusted applets and applications are not permitted to read from or write to arbitrary places in the filesystem. The ability of untrusted code to read and write files, as with any kind of system resource, is under the control of the system security policy, through a SecurityManager object. A security policy is set by the application that is running the untrusted code, such as appletviewer or a Java-enabled web browser. All filesystem access must first pass the scrutiny of the SecurityManager .

Some web browsers allow untrusted applets to have access to specific files designated by the user. Netscape Navigator and Internet Explorer currently do not allow untrusted applets any access to the filesystem. However, as we'll see in Chapter 22, signed applets can be given arbitrary access to the filesystem, just like a standalone Java application.

It's not unusual to want an applet to maintain some kind of state information on the system on which it's running. But for a Java applet that is restricted from access to the local filesystem, the only option is to store data over the network on its server (or possibly in a client-side cookie). Applets have at their disposal powerful general means for communicating data over networks. The only limitation is that, by convention, an applet's network communication is restricted to the server that launched it. This limits the options for where the data will reside.

Currently, the only way for a Java program to send data to a server is through a network socket or tools such as RMI, which run over sockets. In Chapter 11 we'll take a detailed look at building networked applications with sockets. With the tools described in that chapter, it's possible to build powerful client/server applications. Sun also has a Java extension called WebNFS, which allows applets and applications to work with files on an NFS server in much the same way as the ordinary File API.

11.2.5 Loading Application Resources

We often package data files and other objects with our applications. Java provides many ways to access these resources. In a standalone application, we can simply open files and read the bytes. In both standalone applications and applets, we can construct URLs to well-known locations. The problem with these methods is that we generally have to know where our application lives in order to find our data. This is not always as easy as it seems. What is needed is a universal way to access resources associated with our application and our application's individual classes. The Class class's getResource() method provides just this.

What does getResource() do for us? To construct a URL to a file, we normally have to figure out a home directory for our code and construct a path relative to that. As we'll see in Chapter 22, in an applet, we could use getCodeBase() or getDocumentBase() to find the base URL and then use that base to create the URL for the resource we want. But these methods don't help a standalone application, and there's no reason that a standalone application and an applet shouldn't be written in the same way anyway. To solve this problem, the getResource() method provides a standard way to get objects relative to a given class file or to the system classpath. getResource() returns a special URL that uses the class's class loader. This means that no matter where the class came from—a web server, the local filesystem, or even a JAR file—we can simply ask for an object, get a URL for the object, and use the URL to access the object.

getResource() takes as an argument a slash-separated pathname for the resource and returns a URL. There are two kinds of paths: absolute and relative. An absolute path begins with a slash, for example, /foo/bar/blah.txt . In this case, the search for the object begins at the top of the classpath. If there is a directory foo/bar in the classpath, getResource() searches that directory for the blah.txt file. A relative URL does not begin with a slash. In this case, the search begins at the location of the class file, whether it is local, on a remote server, or in a JAR file (either local or remote). So if we were calling getResource() on a class loader that loaded a class in the foo.bar package, we could refer to the file as blah.txt . In this case, the class itself would be loaded from the directory foo/bar somewhere on the classpath, and we'd expect to find the file in the same directory.

For example, here's an application that looks up some resources:

//file: FindResources.java
package mypackage; 
import java.net.URL;
import java.io.IOException;
 
public class FindResources { 
  public static void main(String [] args) throws IOException { 
    // absolute from the classpath 
    URL url = FindResources.class.getResource("/mypackage/foo.txt");
    // relative to the class location 
    url = FindResources.class.getResource("foo.txt"); 
    // another relative document 
    url = FindResources.class.getResource("docs/bar.txt"); 
  }
}

The FindResources class belongs to the mypackage package, so its class file will live in a mypackage directory somewhere on the classpath. FindResources locates the document foo.txt using an absolute and then a relative URL. At the end, FindResources uses a relative path to reach a document in the mypackage/docs directory. In each case we refer to the FindResources 's Class object using the static .class notation. Alternatively, if we had an instance of the object, we could use its getClass() method to reach the Class object.

For an applet, the search is similar but occurs on the host from which the applet was loaded. getResource() first checks any JAR files loaded with the applet, and then searches the normal remote applet classpath, constructed relative to the applet's codebase URL.

getResource() returns a URL for whatever type of object you reference. This could be a text file or properties file that you want to read as a stream, or it might be an image or sound file or some other object. If you want the data as a stream, the Class class also provides a getResourceAsStream() method. In the case of an image, you'd probably hand the URL over to the getImage() method of a Swing component for loading.

11.3 Serialization

Using a DataOutputStream , you could write an application that saves the data content of your objects as simple types. However Java provides an even more powerful mechanism called object serialization that does almost all the work for you. In its simplest form, object serialization is an automatic way to save and load the state of an object. However, object serialization has depths that we cannot plumb within the scope of this book, including complete control over the serialization process and interesting conundrums such as class versioning.

Basically, an object of any class that implements the Serializable interface can be saved and restored from a stream. Special stream subclasses, ObjectInputStream and ObjectOutputStream , are used to serialize primitive types and objects. Subclasses of Serializable classes are also serializable. The default serialization mechanism saves the value of an object's nonstatic and nontransient (see the following explanation) member variables.

One of the most important (and tricky) things about serialization is that when an object is serialized, any object references it contains are also serialized. Serialization can capture entire "graphs" of interconnected objects and put them back together on the receiving end (we'll demonstrate this in an upcoming example). The implication is that any object we serialize must contain only references to other Serializable objects. We can take control by marking nonserializable members as transient or overriding the default serialization mechanisms. The transient modifier can be applied to any instance variable to indicate that its contents are not useful outside of the current context and should never be saved.

In the following example, we create a Hashtable and write it to a disk file called h.ser . The Hashtable object is serializable because it implements the Serializable interface.

//file: Save.java
import java.io.*; 
import java.util.*; 
  
public class Save { 
  public static void main(String[] args) { 
    Hashtable h = new Hashtable(); 
    h.put("string", "Gabriel Garcia Marquez"); 
    h.put("int", new Integer(26)); 
    h.put("double", new Double(Math.PI)); 
     
    try { 
      FileOutputStream fileOut = new FileOutputStream("h.ser"); 
      ObjectOutputStream out = new ObjectOutputStream(fileOut); 
      out.writeObject(h); 
    } 
    catch (Exception e) { 
      System.out.println(e); 
    } 
  } 
}

First we construct a Hashtable with a few elements in it. Then, in the three lines of code inside the try block, we write the Hashtable to a file called h.ser , using the writeObject() method of ObjectOutputStream . The ObjectOutputStream class is a lot like the DataOutputStream class, except that it includes the powerful writeObject() method.

The Hashtable we created has internal references to the items it contains. Thus, these components are automatically serialized along with the Hashtable . We'll see this in the next example when we deserialize the Hashtable .

//file: Load.java
import java.io.*; 
import java.util.*; 
  
public class Load { 
  public static void main(String[] args) { 
    try { 
      FileInputStream fileIn = new FileInputStream("h.ser"); 
      ObjectInputStream in = new ObjectInputStream(fileIn); 
      Hashtable h = (Hashtable)in.readObject(); 
      System.out.println(h.toString()); 
    } 
    catch (Exception e) { 
      System.out.println(e); 
    } 
  } 
}

In this example, we read the Hashtable from the h.ser file, using the readObject() method of ObjectInputStream . The ObjectInputStream class is a lot like DataInputStream , except that it includes the readObject() method. The return type of readObject() is Object , so we need to cast it to a Hashtable . Finally, we print out the contents of the Hashtable using its toString() method.

11.3.1 Initialization with readObject()

Often simple deserialization alone is not enough to reconstruct the full state of an object. For example, the object may have had transient fields representing state that could not be serialized, such as network connections, event registration, or decoded image data. Objects have an opportunity to do their own setup after deserialization by implementing a special method named readObject() .

Not to be confused with the readObject() method of the ObjectInputStream , this method is implemented by the serializable object itself. The readObject() method must have a specific signature, and it must be private. The following snippet is taken from an animated JavaBean that we'll talk about in Chapter 21:

private void readObject(ObjectInputStream s)
    throws IOException, ClassNotFoundException 
{
    s.defaultReadObject();
    initialize();
    if (isRunning)
        start();
}

When the readObject() method with this signature exists in an object it is called during the deserialization process. The argument to the method is the ObjectInputStream doing the object construction. We delegate to its defaultReadObject() method to do the normal deserialization and then do our custom setup. In this case we call one of our methods, named initialize() , and optionally a method called start() .

We'll talk more about serialization in Chapter 21 when we discuss JavaBeans. There we'll see that it is even possible to serialize a graphical GUI component in mid-use and bring it back to life later.

11.4 Data Compression

The java.util.zip package contains classes you can use for data compression. In this section, we'll talk about how to use these classes. We'll also present two useful example programs that build on what you have just learned about streams and files. The classes in the java.util.zip package support two widespread compression formats: GZIP and ZIP.

11.4.1 Compressing Data

The java.util.zip class provides two FilterOutputStream subclasses to write compressed data to a stream. To write compressed data in the GZIP format, simply wrap a GZIPOutputStream around an underlying stream and write to it. The following is a complete example that shows how to compress a file using the GZIP format.

//file: GZip.java
import java.io.*; 
import java.util.zip.*; 
  
public class GZip { 
  public static int sChunk = 8192; 
  
  public static void main(String[] args) { 
    if (args.length != 1) { 
      System.out.println("Usage: GZip source"); 
      return; 
    } 
    // create output stream
    String zipname = args[0] + ".gz"; 
    GZIPOutputStream zipout; 
    try { 
      FileOutputStream out = new FileOutputStream(zipname); 
      zipout = new GZIPOutputStream(out); 
    } 
    catch (IOException e) { 
      System.out.println("Couldn't create " + zipname + "."); 
      return; 
    } 
    byte[] buffer = new byte[sChunk]; 
    // compress the file
    try { 
      FileInputStream in = new FileInputStream(args[0]); 
      int length; 
      while ((length = in.read(buffer, 0, sChunk)) != -1) 
        zipout.write(buffer, 0, length); 
      in.close(); 
    } 
    catch (IOException e) { 
      System.out.println("Couldn't compress " + args[0] + "."); 
    } 
    try { zipout.close(); } 
    catch (IOException e) {} 
  } 
}

First we check to make sure we have a command-line argument representing a filename. Then we construct a GZIPOutputStream wrapped around a FileOutputStream representing the given filename, with the .gz suffix appended. With this in place, we open the source file. We read chunks of data and write them into the GZIPOutputStream . Finally, we clean up by closing our open streams.

Writing data to a ZIP archive file is a little more involved but still quite manageable. While a GZIP file contains only one compressed file, a ZIP file is actually a collection of files, some (or all) of which may be compressed. Each item in the ZIP file is represented by a ZipEntry object. When writing to a ZipOutputStream , you'll need to call putNextEntry() before writing the data for each item. The following example shows how to create a ZipOutputStream . You'll notice it's just like creating a GZIPOutputStream :

ZipOutputStream zipout; 
try { 
  FileOutputStream out = new FileOutputStream("archive.zip"); 
  zipout = new ZipOutputStream(out); 
} 
catch (IOException e) {}

Let's say we have two files we want to write into this archive. Before we begin writing, we need to call putNextEntry() . We'll create a simple entry with just a name. There are other fields in ZipEntry that you can set, but most of the time you won't need to bother with them.

try { 
  ZipEntry entry = new ZipEntry("First"); 
  zipout.putNextEntry(entry); 
  ZipEntry entry = new ZipEntry("Second"); 
  zipout.putNextEntry(entry); 
  . . .
} 
catch (IOException e) {}

11.4.2 Decompressing Data

To decompress data, you can use one of the two FilterInputStream subclasses provided in java.util.zip . To decompress data in the GZIP format, simply wrap a GZIPInputStream around an underlying FileInputStream and read from it. The following is a complete example that shows how to decompress a GZIP file:

//file: GUnzip.java
import java.io.*; 
import java.util.zip.*; 
  
public class GUnzip { 
  public static int sChunk = 8192; 
  public static void main(String[] args) { 
    if (args.length != 1) { 
      System.out.println("Usage: GUnzip source"); 
      return; 
    } 
    // create input stream
    String zipname, source; 
    if (args[0].endsWith(".gz")) { 
      zipname = args[0]; 
      source = args[0].substring(0, args[0].length() - 3); 
    } 
    else { 
      zipname = args[0] + ".gz"; 
      source = args[0]; 
    } 
    GZIPInputStream zipin; 
    try { 
      FileInputStream in = new FileInputStream(zipname); 
      zipin = new GZIPInputStream(in); 
    } 
    catch (IOException e) { 
      System.out.println("Couldn't open " + zipname + "."); 
      return; 
    } 
    byte[] buffer = new byte[sChunk]; 
    // decompress the file
    try { 
      FileOutputStream out = new FileOutputStream(source); 
      int length; 
      while ((length = zipin.read(buffer, 0, sChunk)) != -1) 
        out.write(buffer, 0, length); 
      out.close(); 
    } 
    catch (IOException e) { 
      System.out.println("Couldn't decompress " + args[0] + ".");
    } 
    try { zipin.close(); } 
    catch (IOException e) {} 
  } 
}

First we check to make sure we have a command-line argument representing a filename. If the argument ends with .gz , we figure out what the filename for the uncompressed file should be. Otherwise, we use the given argument and assume the compressed file has the .gz suffix. Then we construct a GZIPInputStream wrapped around a FileInputStream , representing the compressed file. With this in place, we open the target file. We read chunks of data from the GZIPInputStream and write them into the target file. Finally, we clean up by closing our open streams.

Again, the ZIP archive presents a little more complexity than the GZIP file. When reading from a ZipInputStream , you should call getNextEntry() before reading each item. When getNextEntry() returns null , there are no more items to read. The following example shows how to create a ZipInputStream . You'll notice it's just like creating a GZIPInputStream :

ZipInputStream zipin; 
try { 
  FileInputStream in = new FileInputStream("archive.zip"); 
  zipin = new ZipInputStream(in); 
} 
catch (IOException e) {}

Suppose we want to read two files from this archive. Before we begin reading, we need to call getNextEntry() . At the least, the entry will give us a name of the item we are reading from the archive:

try { 
  ZipEntry first = zipin.getNextEntry(); 
} 
catch (IOException e) {}

At this point, you can read the contents of the first item in the archive. When you come to the end of the item, the read() method will return -1 . Now you can call getNextEntry() again to read the second item from the archive:

try { 
  ZipEntry second = zipin.getNextEntry(); 
} 
catch (IOException e) {}

If you call getNextEntry(), and it returns null , there are no more items, and you have reached the end of the archive.

11.5 The NIO Package

The java.nio package is a major new addition in Java 1.4. The name NIO stands for "new I/O," which may seem to imply that it is to be a replacement for the java.io package. In fact, much of the NIO functionality does overlap with existing APIs. NIO was added primarily to address specific issues of scalability for large systems, especially in networked applications. That said, NIO also provides several new features Java lacked in basic I/O, so there are some tools here that you'll want to look at even if you aren't planning to write any large or high-performance services. The primary features of NIO are outlined in the following sections.

11.5.1 Asynchronous I/O

Most of the need for the NIO package was driven by the desire to add nonblocking and selectable I/O to Java. Prior to NIO, most read and write operations in Java were bound to threads that were forced to block for unpredictable amounts of time. Although certain APIs such as Sockets (which we'll see in Chapter 12) provided specific means to limit how long an I/O call could take, this was a workaround to compensate for the lack of a more general mechanism. Prior to the introduction of threads, in many languages I/O could still be done efficiently by setting I/O streams to a nonblocking mode and testing them for their readiness to send or receive data. In a nonblocking mode, a read or write does only as much work as can be done immediately—filling or emptying a buffer and then returning. Combined with the ability to test for readiness, this allows a single thread to continuously service many channels efficiently . The main thread "selects" a stream that is ready and works with it until it blocks, then moves to another. On a single processor system, this is fundamentally equivalent to using multiple threads. Even now, this style of processing has advantages when using a pool of threads (rather than just one). We'll discuss this in detail in Chapter 12 when we discuss networking and building servers that can handle many clients simultaneously .

In addition to nonblocking and selectable I/O, the NIO package enables closing and interrupting I/O operations asynchronously. As discussed in Chapter 8, prior to NIO there was no reliable way to stop or wake up a thread blocked in an I/O operation. With NIO, threads blocked in I/O operations always wake up when interrupted or when the channel is closed by anyone. Additionally, if you interrupt a thread while it is blocked in an NIO operation, its channel is automatically closed. (Closing the channel because the thread is interrupted might seem too strong, but usually it's the right thing to do.)

11.5.2 Performance

Channel I/O is designed around the concept of buffers , which are a more sophisticated form of array, tailored to working with communications. The NIO package supports the concept of direct buffers , buffers that maintain their memory outside the Java virtual machine, in the native host operating system. Since all real I/O operations ultimately have to work with the host OS, by maintaining the buffer space there, some operations can be made much more efficient. Data can be transferred without first copying it into Java and back out.

11.5.3 Mapped and Locked Files

NIO provides two general-purpose file-related features—memory-mapped files and file locking. We'll discuss memory-mapped files later, but suffice it to say that they allow you to work with file data as if it were all magically resident in memory. File locking supports the concept of shared and exclusive locks on regions of files—useful for concurrent access by multiple applications.

11.5.4 Channels

While java.io deals with streams, java.nio works with channels. A channel is an endpoint for communication. Although in practice channels are similar to streams, the underlying notion of a channel is a bit more abstract and primitive. Whereas streams in java.io are defined in terms of input or output with methods to read and write bytes, the basic channel interface says nothing about how communications happen. It simply defines whether the channel is open or closed, via the methods isOpen() and close() . Implementations of channels for files, network sockets, or arbitrary devices then add their own methods for operations such as reading, writing, or transferring data. The following channels are provided by NIO:

• FileChannel

• Pipe.SinkChannel , Pipe.SourceChannel

• SocketChannel , ServerSocketChannel , DatagramChannel

We'll cover FileChannel in this chapter. The Pipe channels are simply the channel equivalents of the java.io Pipe facilities. We'll talk about Socket and Datagram channels in Chapter 12.

All these basic channels implement the ByteChannel interface, designed for channels that have read and write methods such as I/O streams. ByteChannel s read and write ByteBuffer s, however, not byte arrays.

In addition to these native channels, you can bridge to channels from java.io I/O streams and readers and writers for interoperability. Know that, if you mix these features, you may not get the full benefits of performance and asynchronous I/O.

11.5.5 Buffers

Most of the utilities of the java.io and java.net packages operate on byte arrays. The corresponding tools of the NIO package are built around ByteBuffer s (with another type of buffer, CharBuffer , serving as a bridge to the text world). Byte arrays are simple, so why are buffers necessary? They serve several purposes.

• They formalize the usage patterns for buffered data and they provide for things like read-only buffers and keep track of read/write positions and limits within a large buffer space. They also provide a mark/reset facility like that of BufferedInputStream .

• They provide additional APIs for working with raw data representing primitive types. You can create buffers that "view" your byte data as a series of larger primitives such as short s, int s, or float s. The most general type of data buffer, ByteBuffer , includes methods that let you read and write all primitive types like DataOutputStream does for streams.

• They abstract the underlying storage of the data, allowing for special optimizations by Java. Specifically, buffers may be allocated as direct buffers that use native buffers of the host operating system instead of arrays in Java's memory. The NIO Channel facilities that work with buffers can recognize direct buffers automatically and try to optimize I/O to use them. For example, a read from a file channel into a Java byte array normally requires Java to copy the data for the read from the host operating system into Java's memory. But with a direct buffer the data can remain outside Java's normal memory space, in the host operating system.

11.5.5.1 Buffer operations

Buffer is a subclass of java.nio.Buffer object. The base Buffer is something like an array with state. The base Buffer class does not specify what type of elements it holds (that is for subtypes to decide), but it does define functionality common to all data buffers. A Buffer has a fixed size called its capacity . Although all the standard Buffer s provide "random access" to their contents, a Buffer expects to be read and written sequentially, so Buffer s maintain the notion of a position where the next element is read or written. In addition to the position a Buffer can maintain two other pieces of state information: a limit , which is a position that is a "soft" limit to the extent of a read or write, and a mark , which can be used to remember an earlier position for future recall.

Implementations of Buffer add specific, typed get and put methods that read and write the buffer contents. For example, ByteBuffer is a buffer of bytes and it has get() and put() methods that read and write bytes and arrays of bytes (along with many other useful methods we'll discuss later). Getting from and putting to the Buffer changes the position marker, so the Buffer keeps track of its contents somewhat like a stream. Attempting to read or write past the limit marker generates a BufferUnderflowException or BufferOverflowException , respectively.

The mark, position, limit, and capacity values always obey the formula:

mark position limit capacity

The position for reading and writing the Buffer is always greater than the mark, which serves as a lower bound, and the limit, which serves as an upper bound. The capacity represents the physical extent of the buffer space.

You can set the position and limit markers explicitly with the position() and limit() methods. But several convenience methods are provided for the common usage patterns. The reset() method sets the position back to the mark. If no mark has been set, an InvalidMarkException is thrown. The clear() method resets the position to zero and makes the limit the capacity, readying the buffer for new data (the mark is discarded). Note that the clear() method does not actually do anything to the data in the buffer; it simply changes the position markers.

The flip() method is used for the common pattern of writing data into the buffer and then reading it back out. flip makes the current position the limit and then resets the current position to zero (any mark is thrown away). This saves having to keep track of how much data was read. Another method, rewind() , simply resets the position to zero, leaving the limit alone. You might use it to write the same data again. Here is a snippet of code that uses these methods to read data from a channel and writes it to two channels:

ByteBuffer buff = ...
while (inChannel.read(buff) > 0) { // position = ?  
    buff.flip();    // limit = position; position = 0;
    outChannel.write(buff); 
    buff.rewind();  // position = 0 
    outChannel2.write(buff);
    buff.clear();   // position = 0; limit = capacity
}

This might be confusing the first time you look at it because here the read from the Channel is actually a write to the Buffer and vice versa. Because this example writes all the available data up to the limit, either flip() or rewind() have the same effect in this case.

11.5.5.2 Buffer types

As stated earlier, various buffer types add get and put methods for reading and writing specific data types. There is a buffer type for each of the Java primitive types: ByteBuffer , CharBuffer , ShortBuffer , IntBuffer , LongBuffer , FloatBuffer and DoubleBuffer . Each provides get and put methods for reading and writing its type and arrays of its type. Of these, ByteBuffer is the most flexible. Because it has the "finest grain" of all the buffers, it has been given a full complement of get and put methods for reading and writing all the other data types, as well as byte . Here are some ByteBuffer methods:

byte get()
char getChar()
short getShort()
int getInt()
long getLong()
float getFloat()
double getDouble()
  
void put(byte b)
void put(ByteBuffer src)
void put(byte[] src, int offset, int length) 
void put(byte[] src)
void putChar(char value)
void putShort(short value)
void putInt(int value)
void putLong(long value)
void putFloat(float value)
void putDouble(double value)

As we said, all the standard buffers also support random access. So for each of the aforementioned methods of ByteBuffer , there is an additional form that takes an index:

getLong(int index)
putLong(int index, long value)

But that's not all. ByteBuffer can also provide "views" of itself as any of the larger grained types. For example, you can fetch a ShortBuffer view of a ByteBuffer with the asShortBuffer() method. The ShortBuffer view is backed by the ByteBuffer , which means that they work on the same data, and changes to either one affect the other. The view buffer's extent starts at the ByteBuffer 's current position, and its capacity is a function of the remaining number of bytes, divided by the new type's size. (For example, short s and float s consume two bytes each, long s and double s four.) View buffers are convenient for reading and writing large blocks of a contiguous type within a ByteBuffer .

CharBuffer s are interesting as well, primarily because of their integration with String s. Both CharBuffer s and String s implement the java.lang.CharSequence interface. This is the interface that provides the standard charAt() and length() methods. Because of this, newer APIs (such as the java.util.regex package) allow you to use a CharBuffer or a String interchangeably. In this case, the buffer acts like a String with user-configurable start and end positions.

11.5.5.3 Byte order

Now, since we're talking about reading and writing types larger than a byte here, the question arises: in what order do the bytes of multibyte values (e.g. short s, int s) get written? There are two camps in this world: "big endian" and "little endian." ^[1] Big endian means that the most significant bytes come first; little endian is the reverse. If you're writing binary data for consumption by some native application, this is important. Intel-compatible computers use little endian, and many workstations that run Unix use big endian. The ByteOrder class encapsulates the choice. You can specify the byte order to use with the ByteArray order() method, using the identifiers ByteOrder.BIG_ENDIAN and ByteOrder.LITTLE_ENDIAN like so:

byteArray.order(ByteOrder.BIG_ENDIAN);

You can retrieve the native ordering for your platform using the static ByteOrder.nativeOrder() method.

11.5.5.4 Allocating buffers

You can create a buffer either by allocating it explicitly using allocate() or by wrapping an existing array type. Each buffer type has a static allocate() method that takes a capacity (size) and also a wrap() method that takes an existing array:

CharBuffer cbuf = CharByffer.allocate(64*1024);

A direct buffer is allocated in the same way, with the allocateDirect() method:

ByteBuffer bbuf = ByteByffer.allocateDirect(64*1024);

As we described earlier, direct buffers can use native host operating-system memory structures that are optimized for use with some kinds of I/O operations. The tradeoff is that allocating a direct buffer is a little slower than a plain buffer, so you should try to use them for longer term buffers. (For example, on a 400-MHz Sparc Ultra 60, it took about 10 milliseconds to allocate a 1-MB direct buffer versus 2 milliseconds for a plain buffer of the same size.)

11.5.6 Character Encoders and Decoders

Character encoders and decoders turn characters into raw bytes and vice versa, mapping from the Unicode standard to particular encoding schemes. Encoders and decoders have always existed in Java for use by Reader and Writer streams and in the methods of the String class that work with byte arrays. However, prior to Java 1.4, there was no API for working with encoding explicitly; you simply referred to encoders and decoders wherever necessary by name as a String . The java.nio.charset package formalizes the idea of a Unicode character set with the Charset class.

The Charset class is a factory for Charset instances, which know how to encode character buffers to byte buffers and decode byte buffers to character buffers. You can look up a character set by name with the static Charset.forName() method and use it in conversions:

Charset charset = Charset.forName("US-ASCII");
CharBuffer charBuff = charset.decode(byteBuff);  // to ascii
ByteBuffer byteBuff = charset.encode(charBuff);  // and back

You can also test to see if an encoding is available with the static Charset.isSupported() method.

The following character sets are guaranteed to be supplied:

• US-ASCII

• ISO-8859-1

• UTF-8

• UTF-16BE

• UTF-16LE

• UTF-16

You can list all the encoders available on your platform using the static availableCharsets() method:

Map map = Charset.availableCharsets();
Iterator it = map.keySet().iterator();
while (it.hasNext()) 
    System.out.println(it.next());

The result of availableCharsets() is a map because character sets may have "aliases" and appear under more than one name.

In addition to the buffer-oriented classes of the java.nio package, the InputStreamReader and OutputStreamWriter bridge classes of the java.io package have been updated to work with Charset as well. You can specify the encoding as a Charset object or by name.

11.5.6.1 CharsetEncoder and CharsetDecoder

You can get more control over the encoding and decoding process by creating an instance of CharsetEncoder or CharsetDecoder (codec) with the Charset newEncoder() and newDecoder() methods. In our earlier example, we assumed that all the data was available in a single buffer. More often, however, we might have to process data as it arrives in chunks. The encoder/decoder API allows for this by providing more general encode() and decode() methods that take a flag specifying whether more data is expected. The codec needs to know this because it might have been left hanging in the middle of a multibyte character conversion when the data ran out. If it knows that more data is coming, it will not throw an error on this incomplete conversion. In the following snippet, we use a decoder to read from a ByteBuffer bbuff and accumulate character data into a CharBuffer cbuff :

CharsetDecoder decoder = Charset.forName("US-ASCII").newDecoder();
  
boolean done = false;
while (!done) {
    bbuff.clear();
    done = (in.read(bbuff) == -1);
    bbuff.flip();
    decoder.decode(bbuff, cbuff, done);
}
cbuff.flip();
// use cbuff. . .

Here we look for the end of input condition on the in channel to set the flag done . The encode() and decode() methods also return a special result object, CoderResult , that can determine the progress of encoding. The methods isError() , isUnderflow() , and isOverflow() on the CoderResult specify why encoding stopped : for an error, a lack of bytes on the input buffer, or a full output buffer, respectively.

11.5.7 FileChannel

Now that we've covered the basics of channels and buffers, it's time to look at a real channel type. The FileChannel is the NIO equivalent of the java.io.RandomAccessFile , but it provides several basic new features, in addition to some performance optimizations. You will want to use a FileChannel in place of a plain java.io file stream if you wish to use file locking, memory mapped file access, or perform highly optimized data transfer between files or between file and network channels.

A FileChannel is constructed from a FileInputStream , FileOutputStream , or RandomAccessFile :

FileChannel readOnlyFc = new FileInputStream("file.txt").getChannel();
FileChannel readWriteFc = 
    new RandomAccessFile("file.txt", "rw").getChannel();

FileChannel s for file input and output streams are read-only or write-only, respectively. To get a read-write FileChannel you must construct a RandomAccessFile with read-write permissions, as in the previous example.

Using a FileChannel is just like a RandomAccessFile , but it works with ByteBuffer instead of byte arrays:

bbuf.clear();
readOnlyFc.position(index);
readOnlyFc.read(bbuf);
bbuf.flip();
readWriteFc.write(bbuf);

You can control how much data is read and written either by setting buffer position and limit markers or using another form of read/write that takes a buffer starting position and length. You can also read and write to a random position using:

readWriteFc.read(bbuf, index)
readWriteFc.write(bbuf, index2);

In each case, the actual number of bytes read or written depends on several factors. The operation tries to read or write to the limit of the buffer and the vast majority of the time that is what happens with local file access. But the operation is only guaranteed to block until at least one byte has been processed. Whatever happens, the number of bytes processed is returned, and the buffer position is updated accordingly . This is one of the things that is convenient about buffers; they can manage the count for you. Like standard streams, the channel read() method returns -1 upon reaching the end of input.

The size of the file is always available with the size() method. It can change if you write past the end of the file. Conversely, you can truncate the file to a specified length with the truncate() method.

11.5.7.1 Concurrent access

FileChannel s are safe for use by multiple threads and guarantee that data " viewed " by them is consistent across channels in the same VM. However no guarantees are made about how quickly writes are propagated to the storage mechanism. If you need to be sure that data is safe before moving on, you can use the force() method to flush changes to disk. The force() method takes a boolean argument indicating whether or not file metadata, including timestamp and permissions, must be written. Some systems keep track of reads on files as well as writes, so you can save a lot of updates if you set the flag to false , which indicates that you don't care about syncing that data immediately.

As with all Channel s, a FileChannel may be closed by any thread. Once closed all its read/write and position-related methods throw a ClosedChannelException .

11.5.7.2 File locking

FileChannel s support exclusive and shared locks on regions of files through the lock() method:

FileLock fileLock = fileChannel.lock();
int start = 0, len = fileChannel2.size();
FileLock readLock = fileChannel2.lock(start, len, true);

Locks may be either shared or exclusive. An exclusive lock prevents others from acquiring a lock of any kind on the specified file or file region. A shared lock allows others to acquire overlapping shared locks but not exclusive locks. These are useful as write locks and read locks respectively. When you are writing, you don't want others to be able to write until you're done, but when reading, you need only to block others from writing, not reading concurrently.

The simple lock() method in the previous example attempts to acquire an exclusive lock for the whole file. The second form accepts a starting and length parameter, as well as a flag indicating whether the lock should be shared (or exclusive). The FileLock object returned by the lock() method can be used to release the lock:

fileLock.release();

Note that file locks are a cooperative API; they do not necessarily prevent anyone from reading or writing to the locked file contents. In general the only way to guarantee that locks are obeyed is for both parties to attempt to acquire the lock and use it. Also, shared locks are not implemented on some systems, in which case all requested locks are exclusive. You can test if a lock is shared with the isShared() method.

11.5.7.3 Memory mapped files

One of the most interesting new features offered through FileChannel is the ability to map a file into memory. When a file is memory mapped , like magic it becomes accessible through a single ByteBuffer —just as if the entire file was read into memory at once. The implementation of this is extremely efficient, generally among the fastest ways to access the data. In fact, for working with large files, memory mapping can save a lot of resources and time.

This may seem counterintuitive; we're getting a conceptually easier way to access our data and it's also faster and more efficient? What's the catch? There really is no catch. The reason for this is that all modern operating systems are based on the idea of virtual memory. In a nutshell , that means the operating system makes disk space act like memory by continually paging (swapping 4K blocks called "pages") between memory and disk, transparent to the applications. Operating systems are very good at this; they efficiently cache the data the application is using and let go of what is not. So memory mapping a file is really just taking advantage of what the OS is doing internally.

A good example of where a memory-mapped file would be useful is in a database. Imagine a 100-MB file containing records indexed at various positions. By mapping the file we can work with a standard ByteBuffer , reading and writing data at arbitrary positions and let the native operating system read and write the underlying data in fine grained pages, as necessary. We could emulate this behavior with RandomAccessFile or FileChannel , but we would have to explicitly read and write data into buffers first, and the implementation would almost certainly not be as efficient.

A mapping is created with the FileChannel map() method. For example:

FileChannel fc = new RandomAccessFile("index.db", "rw").getChannel();
MappedByteBuffer mappedBuff = 
    fc.map(FileChannel.MapMode.READ_WRITE, 0, fc.size());

The map() method returns a MappedByteBuffer , which is simply the standard ByteBuffer with a few additional methods relating to the mapping. The most important is force() , which ensures that any data written to the buffer is flushed out to permanent storage on the disk. The READ_ONLY and READ_WRITE constant identifiers of the FileChannel.MapMode static inner class specify the type of access. Read-write access is available only when mapping a read-write file channel. Data read through the buffer is always consistent within the same Java VM. It can also be consistent across applications on the same host machine, but this is not guaranteed.

Again, a MappedByteBuffer acts just like a ByteBuffer . Continuing with the previous example, we could decode the buffer with a character decoder and search for a pattern like so:

CharBuffer cbuff = Charset.forName("US-ASCII").decode(mappedBuff);
Matcher matcher = Pattern.compile("abc*").matcher(cbuff);
while (matcher.find())
        print(matcher.start()+": "+matcher.group(0));

Here we have effectively implemented the Unix grep command in about five lines of code (thanks to the fact that the Regular Expression API can work with our CharBuffer as a CharSequence ). Of course in this example, the CharBuffer allocated by the decode() method is as large as the mapped file and must be held in memory. More generally, we can use the CharsetDecoder shown earlier to iterate through a large mapped space.

11.5.7.4 Direct transfer

The final feature of FileChannel that we'll look at is performance optimization. FileChannel supports two highly optimized data transfer methods: transferFrom() and transferTo() , which move data between the file channel and another channel. These methods can take advantage of direct buffers internally to move data between the channels as fast as possible, often without copying the bytes into Java's memory space at all. The following example is currently the fastest way to implement a file copy in Java:

import java.io.*;
import java.nio.*;
import java.nio.channels.*;
  
public class CopyFile {
    public static void main(String [] args) throws Exception
    {
        String fromFileName = args[0];
        String toFileName = args[1];
        FileChannel in = new FileInputStream(fromFileName).getChannel();
        FileChannel out = new FileOutputStream(toFileName).getChannel();
        in.transferTo(0, (int)in.size(), out);
        in.close();
        out.close();
    }
}

11.5.8 Scaleable I/O with NIO

We've laid the groundwork for using the NIO package in this chapter but left out some of the important pieces. In the next chapter, we'll see more of the real motivation for java.nio when we talk about nonblocking and selectable I/O. In addition to the performance optimizations that can be made through direct buffers, these capabilities make possible a design for network servers that uses fewer threads and can scale well to large systems. We'll also look at the other significant Channel types: SocketChannel , ServerSocketChannel , and DatagramChannel .

[1]  The terms "big endian" and "little endian" come from Jonathan Swift's novel Gulliver's Travels , where it denoted two camps of Lilliputians: those who eat their eggs from the big end and those who eat them from the little end.

CONTENTS