Processing a Binary File

Problem

You want to read binary data from a file, or write it to one.

Solution

Since Ruby strings make no distinction between binary and text data, processing a binary file needn't be any different than processing a text file. Just make sure you add "b" to your file mode when you open a binary file on Windows.

This code writes 10 bytes of binary data to a file, then reads it back:

open('binary', 'wb') do |f| (0..100).step(10) { |b| f << b.chr } end s = open('binary', 'rb') { |f| f.read } # => "00 2436(2

If you want to process a binary file one byte at a time, you'll probably enjoy the way each_byte returns each byte of the file as a number, rather than as single-character strings:

	open('binary', 'rb') { |f| f.each_byte { |b| puts b } }
	# 0
	# 10
	# 20
	# …
	# 90
	# 100

 

Discussion

The methods introduced earlier to deal with text files work just as well for binary files, assuming that your binary files are supposed to be processed from beginning to end, the way text files typically are. If you want random access to the contents of a binary file, you can manipulate your file object's "cursor."

Think of the cursor as a pointer to the first unread byte in the open file. The current position of the cursor is accessed by the method IO#pos. When you open the file, it's set to zero, just before the first byte. You can then use IO#read to read a number of bytes starting from the current position of the cursor, incrementing the cursor as a side effect.

	f = open('binary')
	f.pos # => 0
	f.read(1) # => "00"
	f.pos # => 1

 

You can also just set pos to jump to a specific byte in the file:

	f.pos = 4 # => 4
	f.read(2) # => "(2"
	f.pos # => 6

 

You can use IO#seek to move the cursor forward or backward relative to its current position (with File::SEEK_CUR), or to move to a certain distance from the end of a file (with File::SEEK_END). Unlike the iterator methods, which go through the entire file once, you can use seek or set pos to jump anywhere in the file, even to a byte you've already read.

	f.seek(8)
	f.pos # => 8
	
	f.seek(-4, File::SEEK_CUR)
	f.pos # => 4
	f.seek(2, File::SEEK_CUR)
	f.pos # => 6
	
	# Move to the second-to-last byte of the file.
	f.seek(-2, File::SEEK_END)
	f.pos # => 9

 

Attempting to read more bytes than there are in the file returns the rest of the bytes, and set your file's eof? flag to true:

	f.read(500) # => "Zd"
	f.pos # => 11
	f.eof? # => true
	f.close

 

Often you need to read from and write to a binary file simultaneously. You can open any file for simultaneous reading and writing using the "r+" mode (or, in this case, "rb+"):

	f = open('binary', 'rb+')
	f.read # => "00
2436(2 "00
Hello.PZd"
	f << 'Goodbye.'
	f.rewind
	f.read # => "00
Hello.PZdGoodbye."
	
	f.close

 

You can append new data to the end of a file you've opened for read-write access, and you can overwrite existing data byte for byte, but you can't insert new data into the middle of a file. This makes the read-write technique useful for binary files, where exact byte offsets are often important, and less useful for text files, where it might make sense to add an extra line in the middle.

Why do you need to append "b" to the file mode when opening a binary file on Windows? Because otherwise Windows will mangle any newline characters that show up in your binary file. The "b" tells Windows to leave the newlines alone, because they're not really newlines: they're binary data. Since it doesn't hurt anything on Unix to put "b" in the file mode, you can make your code cross-platform by appending "b" to the mode whenever you open a file you plan to treat as binary. Note that "b" by itself is not a valid file mode: you probably want "rb".

An MP3 example

Because every binary format is different, probably the best I can do to help you beyond this point is show you an example. Consider MP3 music files. Many MP3 files have a 128-byte data structure at the end called an ID3 tag. These 128 bytes are literally packed with information about the song: its name, the artist, which album it's from, and so on. You can parse this data structure by opening an MP3 file and doing a series of reads from a pos near the end of the file.

According to the ID3 standard, if you start from the 128th-to-last byte of an MP3 file and read three bytes, you should get the string "TAG". If you don't, there's no ID3 ag for this MP3 file, and nothing to do. If there is an ID3 tag present, then the 30 bytes after "TAG" contain the name of the song, the 30 bytes after that contain the name of the artist, and so on. Here's some code that parses a file's ID3 tag and puts the results into a hash:

	def parse_id3(mp3_file)
	 fields_and_sizes = [[:track_name, 30], [:artist_name, 30],
	 [:album_name, 30], [:year, 4], [:comment, 30],
	 [:genre, 1]]

	 tag = {}
	 open(mp3_file) do |f|
	 f.seek(-128, File::SEEK_END)
	 if f.read(3) == 'TAG' # An ID3 tag is present
	 fields_and_sizes.each do |field, size|
	 # Read the field and strip off anything after the first null
	 # character.
	 data = f.read(size).gsub(/00.*/, '')
	 # Convert the genre string to a number.
	 data = data[0] if field == :genre
	 tag[field] = data
	 end
	 end
	 end
	 return tag
	end

	parse_id3('ID3.mp3')
	# => {:year=>"2005", :artist_name=>"The ID Three",
	# :album_name=>"Binary Brain Death",
	# :comment=>"http://www.example.com/id3/", :genre=>22,
	# :track_name=>"ID 3"}

	parse_id3('Too Indie For ID3 Tags.mp3') # => {}

 

Rather than specifying the genre of the music as a string, the :genre element of the hash is a single byte, an entry into a lookup table shared by all applications that use ID3. In this table, genre number 22 is "Death metal".

It's less code to specify the byte offsets for a binary file is in the format recognized by String#unpack, which can parse the bytes of a string according to a given format. It returns an array containing the results of the parsing.

	#Returns [track, artist, album, year, comment, genre]
	def parse_id3(mp3_file)
	 format = 'Z30Z30Z30Z4Z30C'
	 open(mp3_file) do |f|
	 f.seek(-128, File::SEEK_END)
	 if f.read(3) == "TAG" # An ID3 tag is present
	 return f.read(125).unpack(format)
	 end
	 end
	 return nil
	end
	
	parse_id3('ID3.mp3')
	# => ["ID 3", "The ID Three", " 
Binary Brain Death", "2005", "http://www.example.com/
	id3/", 22]

 

As you can see, the unpack format is obscure but very concise. The string "Z30Z30Z30Z4Z30C" passed into String#unpack completely describes the elements of the ID3 format after the "TAG":

• Three strings of 30 bytes, with null characters stripped ("Z30Z30Z30")
• A string of 4 bytes, with null characters stripped ("Z4")
• One more string of 30 bytes, with null characters stripped ("Z30")
• A single character, represented as an unsigned integer ("C")

It doesn't describe what those elements are supposed to be used for, though.

When writing binary data to a file, you can use Array#pack, the opposite of String#unpack:

	id3 = ["ID 3", "The ID Three", "Binary Brain Death", "2005",
	 "http://www.example.com/id3/", 22]
	id3.pack 'Z30Z30Z30Z4Z30C'
	# => "ID 30000000000…http://www.example.com/id3/00000026"

 

See Also

• The ID3 standard, described at http://en.wikipedia.org/wiki/ID3 along with the table of genres; the code in this recipe parses the original ID3v1 standard, which is much simpler than ID3v2
• ri String#unpack and ri Array#pack