New IO

New I/O

JDK 1.4 contains a number of features for improved input/output processing, collectively called the "new I/O," in the java.nio package. (Of course, the "new" moniker is somewhat regrettable because, a few years down the road, the package won't be new any longer.)

The package includes support for the following features:

Memory-mapped files
File locking
Character set encoders and decoders
Nonblocking I/O

We already introduced character sets on page 634. In this section, we discuss only the first two features. Nonblocking I/O requires the use of threads, which are covered in Volume 2.

Memory-Mapped Files

Most operating systems can take advantage of the virtual memory implementation to "map" a file, or a region of a file, into memory. Then the file can be accessed as if it were an in-memory array, which is much faster than the traditional file operations.

At the end of this section, you can find a program that computes the CRC32 checksum of a file, using traditional file input and a memory-mapped file. On one machine, we got the timing data shown in Table 12-7 when computing the checksum of the 37-Mbyte file rt.jar in the jre/lib directory of the JDK.

Table 12-7. Timing Data for File Operations
Method	Time
Plain Input Stream	110 seconds
Buffered Input Stream	9.9 seconds
Random Access File	162 seconds
Memory Mapped file	7.2 seconds

As you can see, on this particular machine, memory mapping is a bit faster than using buffered sequential input and dramatically faster than using a RandomAccessFile.

Of course, the exact values will differ greatly from one machine to another, but it is obvious that the performance gain can be substantial if you need to use random access. For sequential reading of files of moderate size, on the other hand, there is no reason to use memory mapping.

The java.nio package makes memory mapping quite simple. Here is what you do.

First, get a channel from the file. A channel is an abstraction for disk files that lets you access operating system features such as memory mapping, file locking, and fast data transfers between files. You get a channel by calling the getChannel method that has been added to the FileInputStream, FileOutputStream, and RandomAccessFile class.

 FileInputStream in = new FileInputStream(. . .); FileChannel channel = in.getChannel();

Then you get a MappedByteBuffer from the channel by calling the map method of the FileChannel class. You specify the area of the file that you want to map and a mapping mode. Three modes are supported:

FileChannel.MapMode.READ_ONLY: The resulting buffer is read-only. Any attempt to write to the buffer results in a ReadOnlyBufferException.
FileChannel.MapMode.READ_WRITE: The resulting buffer is writable, and the changes will be written back to the file at some time. Note that other programs that have mapped the same file may not see those changes immediately. The exact behavior of simultaneous file mapping by multiple programs is operating-system dependent.
FileChannel.MapMode.PRIVATE: The resulting buffer is writable, but any changes are private to this buffer and are not propagated to the file.

Once you have the buffer, you can read and write data, using the methods of the ByteBuffer class and the Buffer superclass.

Buffers support both sequential and random data access. A buffer has a position that is advanced by get and put operations. For example, you can sequentially traverse all bytes in the buffer as

 while (buffer.hasRemaining()) {    byte b = buffer.get();    . . . }

Alternatively, you can use random access:

 for (int i = 0; i < buffer.limit(); i++) {    byte b = buffer.get(i);    . . . }

You can also read and write arrays of bytes with the methods

 get(byte[] bytes) get(byte[], int offset, int length)

Finally, there are methods

 getInt getLong getShort getChar getFloat getDouble

to read primitive type values that are stored as binary values in the file. As we already mentioned, Java uses big-endian ordering for binary data. However, if you need to process a file containing binary numbers in little-endian order, simply call

 buffer.order(ByteOrder.LITTLE_ENDIAN);

To find out the current byte order of a buffer, call

 ByteOrder b = buffer.order()

CAUTION

This pair of methods does not use the set/get naming convention.

To write numbers to a buffer, use one of the methods

 putInt putLong putShort putChar putFloat putDouble

Example 12-8 computes the 32-bit cyclic redundancy checksum (CRC32) of a file. That quantity is a checksum that is often used to determine whether a file has been corrupted. Corruption of a file makes it very likely that the checksum has changed. The java.util.zip package contains a class CRC32 that computes the checksum of a sequence of bytes, using the following loop:

CRC32 crc = new CRC32();
while (more bytes)
crc.update(next byte)
long checksum = crc.getValue();

NOTE

For a nice explanation of the CRC algorithm, see http://www.relisoft.com/Science/CrcMath.html.

The details of the CRC computation are not important. We just use it as an example of a useful file operation.

Run the program as

java NIOTest filename

Example 12-8. NIOTest.java

   1. import java.io.*;   2. import java.nio.*;   3. import java.nio.channels.*;   4. import java.util.zip.*;   5.   6. /**   7.    This program computes the CRC checksum of a file.   8.    Usage: java NIOTest filename   9. */  10. public class NIOTest  11. {  12.    public static long checksumInputStream(String filename)  13.       throws IOException  14.    {  15.       InputStream in = new FileInputStream(filename);  16.       CRC32 crc = new CRC32();  17.  18.       int c;  19.       while((c = in.read()) != -1)  20.          crc.update(c);  21.       return crc.getValue();  22.    }  23.  24.    public static long checksumBufferedInputStream(String filename)  25.       throws IOException  26.    {  27.       InputStream in = new BufferedInputStream(new FileInputStream(filename));  28.       CRC32 crc = new CRC32();  29.  30.       int c;  31.       while((c = in.read()) != -1)  32.          crc.update(c);  33.       return crc.getValue();  34.    }  35.  36.    public static long checksumRandomAccessFile(String filename)  37.       throws IOException  38.    {  39.       RandomAccessFile file = new RandomAccessFile(filename, "r");  40.       long length = file.length();  41.       CRC32 crc = new CRC32();  42.  43.       for (long p = 0; p < length; p++)  44.       {  45.          file.seek(p);  46.          int c = file.readByte();  47.          crc.update(c);  48.       }  49.       return crc.getValue();  50.    }  51.  52.    public static long checksumMappedFile(String filename)  53.       throws IOException  54.    {  55.       FileInputStream in = new FileInputStream(filename);  56.       FileChannel channel = in.getChannel();  57.  58.       CRC32 crc = new CRC32();  59.       int length = (int) channel.size();  60.       MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, length);  61.  62.       for (int p = 0; p < length; p++)  63.       {  64.          int c = buffer.get(p);  65.          crc.update(c);  66.       }  67.       return crc.getValue();  68.    }  69.  70.    public static void main(String[] args)  71.       throws IOException  72.    {  73.       System.out.println("Input Stream:");  74.       long start = System.currentTimeMillis();  75.       long crcValue = checksumInputStream(args[0]);  76.       long end = System.currentTimeMillis();  77.       System.out.println(Long.toHexString(crcValue));  78.       System.out.println((end - start) + " milliseconds");  79.  80.       System.out.println("Buffered Input Stream:");  81.       start = System.currentTimeMillis();  82.       crcValue = checksumBufferedInputStream(args[0]);  83.       end = System.currentTimeMillis();  84.       System.out.println(Long.toHexString(crcValue));  85.       System.out.println((end - start) + " milliseconds");  86.  87.       System.out.println("Random Access File:");  88.       start = System.currentTimeMillis();  89.       crcValue = checksumRandomAccessFile(args[0]);  90.       end = System.currentTimeMillis();  91.       System.out.println(Long.toHexString(crcValue));  92.       System.out.println((end - start) + " milliseconds");  93.  94.       System.out.println("Mapped File:");  95.       start = System.currentTimeMillis();  96.       crcValue = checksumMappedFile(args[0]);  97.       end = System.currentTimeMillis();  98.       System.out.println(Long.toHexString(crcValue));  99.       System.out.println((end - start) + " milliseconds"); 100.    } 101. }

 java.io.FileInputStream 1.0

FileChannel getChannel()3 1.4
returns a channel for accessing this stream.

 java.io.FileOutputStream 1.0

FileChannel getChannel() 1.4
returns a channel for accessing this stream.

 java.io.RandomAccessFile 1.0

FileChannel getChannel() 1.4
returns a channel for accessing this file.

 java.nio.channels.FileChannel 1.4

MappedByteBuffer map(FileChannel.MapMode mode, long position, long size)
maps a region of the file to memory.
Parameters:
mode
One of the constants READ_ONLY, READ_WRITE, or PRIVATE in the FileChannel.MapMode class

position
The start of the mapped region

size
The size of the mapped region

 java.nio.Buffer 1.4

boolean hasRemaining()
returns TRue if the current buffer position has not yet reached the buffer's limit position.
int limit()
returns the limit position of the buffer, that is, the first position at which no more values are available.

 java.nio.ByteBuffer 1.4

byte get()
gets a byte from the current position and advances the current position to the next byte.
byte get(int index)
gets a byte from the specified index.
ByteBuffer put(byte b)
puts a byte to the current position and advances the current position to the next byte. Returns a reference to this buffer.
ByteBuffer put(int index, byte b)
puts a byte at the specified index. Returns a reference to this buffer.
ByteBuffer get(byte[] destination)
ByteBuffer get(byte[] destination, int offset, int length)
fill a byte array, or a region of a byte array, with bytes from the buffer, and advance the current position by the number of bytes read. If not enough bytes remain in the buffer, then no bytes are read, and a BufferUnderflowException is thrown. Return a reference to this buffer.
Parameters:
destination
The byte array to be filled

offset
The offset of the region to be filled

length
The length of the region to be filled
ByteBuffer put(byte[] source)
ByteBuffer put(byte[] source, int offset, int length)
put all bytes from a byte array, or the bytes from a region of a byte array, into the buffer, and advance the current position by the number of bytes read. If not enough bytes remain in the buffer, then no bytes are written, and a BufferOverflowException is thrown. Returns a reference to this buffer.
Parameters:
source
The byte array to be written

offset
The offset of the region to be written

length
The length of the region to be written
Xxx getXxx()
Xxx getXxx(int index)
ByteBuffer putXxx(xxx value)
ByteBuffer putXxx(int index, xxx value)
are used for relative and absolute reading and writing of binary numbers. Xxx is one of Int, Long, Short, Char, Float, or Double.
ByteBuffer order(ByteOrder order)
ByteOrder order()
set or get the byte order. The value for order is one of the constants BIG_ENDIAN or LITTLE_ENDIAN of the ByteOrder class.

The Buffer Data Structure

When you use memory mapping, you make a single buffer that spans the entire file, or the area of the file in which you are interested. You can also use buffers to read and write more modest chunks of information.

In this section, we briefly describe the basic operations on Buffer objects. A buffer is an array of values of the same type. The Buffer class is an abstract class with concrete subclasses ByteBuffer, CharBuffer, DoubleBuffer, FloatBuffer, IntBuffer, LongBuffer, and ShortBuffer. In practice, you will most commonly use ByteBuffer and CharBuffer. As shown in Figure 12-13, a buffer has

a capacity that never changes
a position at which the next value is read or written
a limit beyond which reading and writing is meaningless
optionally, a mark for repeating a read or write operation

These values fulfill the condition

0	mark	position	limit	capacity

Figure 12-13. A buffer

The principal purpose for a buffer is a "write, then read" cycle. At the outset, the buffer's position is 0 and the limit is the capacity. Keep calling put to add values to the buffer. When you run out of data or you reach the capacity, it is time to switch to reading.

Call flip to set the limit to the current position and the position to 0. Now keep calling get while the remaining method (which returns limit - position) is positive. When you have read all values in the buffer, call clear to prepare the buffer for the next writing cycle. The clear method resets the position to 0 and the limit to the capacity.

If you want to re-read the buffer, use rewind or mark/reset see the API notes for details.

 java.nio.Buffer 1.4

Buffer clear()
prepares this buffer for writing by setting the position to zero and the limit to the capacity; returns this.
Buffer flip()
prepares this buffer for reading by setting the limit to the position and the position to zero; returns this.
Buffer rewind()
prepares this buffer for re-reading the same values by setting the position to zero and leaving the limit unchanged; returns this.
Buffer mark()
sets the mark of this buffer to the position; returns this.
Buffer reset()
sets the position of this buffer to the mark, thus allowing the marked portion to be read or written again; returns this.
int remaining()
returns the remaining number of readable or writable values, that is, the difference between limit and position.
int position()
returns the position of this buffer.
int capacity()
returns the capacity of this buffer.

 java.nio.CharBuffer 1.4

char get()
CharBuffer get(char[] destination)
CharBuffer get(char[] destination, int offset, int length)
gets one char value, or a range of char values, starting at the buffer's position and moving the position past the characters that were read. The last two methods return this.
CharBuffer put(char c)
CharBuffer put(char[] source)
CharBuffer put(char[] source, int offset, int length)
CharBuffer put(String source)
CharBuffer put(CharBuffer source)
puts one char value, or a range of char values, starting at the buffer's position and advancing the position past the characters that were written. When reading from a CharBuffer, all remaining characters are read. All methods return this.
CharBuffer read(CharBuffer destination)
gets char values from this buffer and puts them into the destination until the destination's limit is reached. Returns this.

File Locking

Consider a situation in which multiple simultaneously executing programs need to modify the same file. Clearly, the programs need to communicate in some way, or the file can easily become damaged.

File locks control access to a file or a range of bytes within a file. However, file locking varies greatly among operating systems, which explains why file locking capabilities were absent from prior versions of the JDK.

Frankly, file locking is not all that common in application programs. Many applications use a database for data storage, and the database has mechanisms for resolving concurrent access problems. If you store information in flat files and are worried about concurrent access, you may well find it simpler to start using a database rather than designing complex file locking schemes.

Still, there are situations in which file locking is essential. Suppose your application saves a configuration file with user preferences. If a user invokes two instances of the application, it could happen that both of them want to write the configuration file at the same time. In that situation, the first instance should lock the file. When the second instance finds the file locked, it can decide to wait until the file is unlocked or simply skip the writing process.

To lock a file, call either the lock or tryLock method of the FileChannel class:

 FileLock lock = channel.lock();

 FileLock lock = channel.tryLock();

The first call blocks until the lock becomes available. The second call returns immediately, either with the lock or null if the lock is not available. The file remains locked until the channel is closed or the release method is invoked on the lock.

You can also lock a portion of the file with the call

 FileLock lock(long start, long size, boolean exclusive)

 FileLock tryLock(long start, long size, boolean exclusive)

The exclusive flag is TRue to lock the file for both reading and writing. It is false for a shared lock, which allows multiple processes to read from the file, while preventing any process

from acquiring an exclusive lock. Not all operating systems support shared locks. You may get an exclusive lock even if you just asked for a shared one. Call the isShared method of the FileLock class to find out which kind you have.