iPac: A Resource File Builder

The case for resource files is an easy one to make; but how do you go about building your own resource files? Every game company I know builds their own tools for this mundane job, but they all share similar features and even data file formats.

For the last few years, all the products I managed used a proprietary tool, iPac, to create and manage resource files. It stores graphics, sounds, and anything else the game needs in a single packed file. In practice the games use multiple iPac files to factor common game data from data that is only used when the game is in a certain state, like when particular game screens are up. Sometimes the best way to learn how to do something is by watching someone else. In this case, if you learn how the guts of iPac work you'll probably come up with some good ideas about your own resource file system.

Overview of iPac

iPac is an MDI MFC application complete with all the drag and drop features you'd expect. The look and feel is very similar to the Windows Explorer as shown in Figure 8.2. The tool's main audience is artists and designers who enjoy a familiar interface. Face it, if you make a strange command line tool that only programmers can understand, they are the only ones who will use it.

Figure 8.2: iPac Application Window.

Best Practice

If artists, sound engineers, and other folks outside of the programming team can use a simple tool to import their work into a game, the quality of their work will rise dramatically, and reduce the workload on the programmers. Imagine how tough it would be if programmers couldn't compile their own programs, and they needed someone else to put their code into the game.

The left pane shows the resource hierarchy, which generally maps one-to-one with the structure of the raw data files on our network. The right pane displays the list of resources in the left pane's selected folder. Unlike the Windows Explorer, iPac orders the contents of each folder and assigns each resource a number. This order dependant scheme can save programmers and the team plenty of time. If you sort animations and sounds with some rationale, the code doesn't have to use the identifier for each and every resource in the file. Imagine a resource folder that holds speech files of an actor counting from zero to one hundred; it would be much easier to access the right resource with an index from the resource "zero."

Each resource has a name and a description. The name is an identifier that is written to a C++ header file, so iPac checks it to make sure the name is a legal one. The description is hidden from the game API, and is a convenience for people building iPac files.

The date is the last date when the resource was imported. iPac is smart enough to detect when resources have changed—they are displayed in a different color. iPac can also lock out resources—a really important feature. As the development team locks down source code late in the development cycle the team should lock down the game data as well. A resource can be set to "final" in the resource dialogs, and it won't be overwritten even if the entire iPac file is refreshed.

The iPac file also monitors the size of each resource and the size of the entire file. It's very easy to let the size of a resource file grow out of control. For example, an artist might mistakenly add a 60fps animation when a 15fps one will do, or a sound engineer might import 44KHz stereo sound when 22KHz mono will work just fine. The size of the file is so important it's displayed in the window title bar.

Best Practice

Don't let developers get carried away. Huge resource files will fill up memory in a heartbeat and cause your game performance to plummet. If your game has to constantly hit the disk to swap resources the game will stutter, pause, and fail to run smoothly. Nothing will frustrate a player more than see a stuttering game on bleeding edge hardware. Make sure that custom tools or a DEBUG build of the game has a monitor that detects obese game data. It should send out some kind of annoying alarm or sound effect. Our game sent a message to stderr for any resource that takes longer than 50ms to load.

Each resource folder stored the source directory for the raw resource files, which is called the working directory. An example is shown in Figure 8.3.

Figure 8.3: Creating a Working Directory.

Folders within folders inherit the working directory by default and add their own name. This mechanism makes it easy to construct an arbitrary folder hierarchy that could mirror the folder hierarchy of the network or point to completely different areas.

Best Practice

Always use drive mappings to locate raw resource files on a network, and never store resources locally unless they are under source control. Network topologies change and servers get reassigned to other projects. For example, if every sound resource was mapped to \\Audio\Sfx and the server disappeared, someone would have a lot of work to do to reassign the working directory. If a raw resource is stored only on a developer's hard disk it can evaporate entirely.

iPac distinguishes graphics from other data. A single JPG or BMP file might contain a huge set of buttons or dialog edges. Bitmap sequences are cut from AVI files generated from hand drawn animations, flash animations, or pre-rendered 3D Studio Max files. Since the raw files can store multiple resources, iPac has to know about the boundaries of each resource and how many frames to assemble.

Any resource tool needs to be able to grab multiple resources from the same file. If the graphic has more than one frame, such as a button with rollover or disabled states, iPac has to know where the button art is located within the larger file. The Resource Properties dialog of a graphics element stores enough data to cut exactly the right pixels from a BMP or JPG, or a multiframe sequence from an AVI. Figure 8.4 shows an example of the Resource Properties dialog.

Figure 8.4: The Resource Properties Dialog.

One of the important selections on the dialog is the compression type. Compressed resources take less space in the game file but they take longer to load. You must be able to choose whether the resource is compressed, and what type of compression algorithm you want to use.

The graphic viewer button on the lower left of the resource dialog launches a viewer window, with the raw resource file in view. The boundary of the desired graphic is outlined with a rectangle or polygon tool. It's a common practice for artists to create user interface components, like buttons and dialogs, in a single bitmap file, laid out in a regular grid. This allows them to make sure everything fits together with no visible seams. If there's animation, such as a button highlight, the art is stored in multiple frames of a single AVI. The animated button is "cut" out of the original animation with a rectangle tool. The resulting button animation contains the exact pixels necessary to draw the animating button.

iPac Data Files

iPac has two data files, one for development and one for the game. The game only needs enough data to locate and read resources. The iPac development file is actually a simple text file, and stores things the game doesn't need such as the source file, name, description, bounding points, and whether the resource has been marked final. A text file is a good format because it's easy to edit by hand or perform global search and replace operations.

The file that ships with the game is called a PAD file. It is a flat, indexed file with each index pointing to the bits of data contained in the file: sounds, art, map levels, whatever. Each indexed component may or may not be compressed. If disk space is critical you can trade some loading time by compressing the bits. It should always be possible for some bits to be compressed, and others to be stored in their raw form. In other words, it should be your choice. PAD files start with a short header as shown in Figure 8.5.

Figure 8.5: The PAD File Header.

The header stores the size of the header and the version number of the file format. These help iPac determine what to do if it opens a file that is either newer or older than the version of the tool.

Best Practice

Any internal tool should be able to detect legacy file formats and either convert them to the latest format or at least inform the user. A tool that doesn't detect file format changes can corrupt game files in development.

The number of folders and the folder offsets are stored next. The folder data is stored because a programmer might use the internal organization of the iPac file to their advantage. This is more flexible than storing the resources in a completely flat file.

Origin Systems used to use a game file format called FLX files, mostly on the last few Ultima games. FLX files were similar to a single folder of the PAD file format. Resources within FLX files were located using just the resource number. There's nothing wrong with this approach—it certainly worked on Ultima VIII and Ultima IX.

Assume for a moment that your game had twenty different characters, each of which had exactly the same kind of data. It might be convenient to store all of the character data in one file, each character organized into their own top level folder. As long as the contents of the folders were organized the same way, the code could access the character data with the folder number and a given resource such as AI with the resource number. Each folder has a short header that stores the number of resources it contains and offsets to the resource data as shown in Figure 8.6.

Figure 8.6: The PAD Folder Header.

Since each folder knows how many resources it contains, the game can use that information to choose a random resource from a set. The Bicycle Casino project used this feature to store speech files. Each folder contained a number of MP3 data streams that said the same thing multiple ways. The game picked one of these MP3 streams at random and kept the speech from becoming monotonous.

The resource data included enough information to load resources based on their type and compression setting (see Figure 8.7). Compressed resources had a different actual size than their stored size, so the uncompressed size was stored rather than calculated.

Figure 8.7: The PAD Resource Header.

Graphic resources had an extra header before the data stream as shown in Figure 8.8.

Figure 8.8: The PAD Graphic Source Header.

Graphic resources can have multiple frames. Each frame is assumed to be the same size or shape and anchor at the same position from frame to frame. As long as you're not talking about 3D textures, 2D sprites can be any shape. In fact, most modern user interfaces are quite organic and there's rarely anything square about them.

The bits that make up the graphics data for each frame immediately follow the resource header.

Generated Header Files

A game's source code includes header files that iPac generates each time a new resource is created or the folder format changes. Each .PAD file has a companion .H file, which contains two enumerations:

 // // iPac generated header file // C:\Projects\Casino_Publish\Jokerz\Bin\Data\GamePacs\Common.pac // #if !defined(IPAC_COMMON_H) #define IPAC_COMMON_H enum CommonFolders {    COMMON_FOLDER                                   = 1,    COMMON_FOLDER_SOUND                             = 2,    COMMON_FOLDER_ART                               = 3,    COMMON_FOLDER_ART_USER_INTERFACE                = 4,    COMMON_FOLDER_ART_USER_INTERFACE_MESSAGE_BOX    = 5, }; enum CommonResources {    COMMON_LISTSTART                                = 0,    COMMON_LISTCOUNT                                = 0,    COMMON_SOUND_LISTSTART                          = 0,    COMMON_SOUND_BUTTONPUSHED                       = 0,    COMMON_SOUND_LISTCOUNT                          = 1,    COMMON_ART_LISTSTART                            = 0,    COMMON_ART_LISTCOUNT                            = 0,    COMMON_ART_USER_INTERFACE_LISTSTART             = 0,    COMMON_ART_USER_INTERFACE_SPOT                  = 0,    COMMON_ART_USER_INTERFACE_LISTCOUNT             = 1,    COMMON_ART_USER_INTERFACE_MESSAGE_BOX_LISTSTART = 0,    COMMON_ART_USER_INTERFACE_MESSAGE_BOX_TL        = 0,    COMMON_ART_USER_INTERFACE_MESSAGE_BOX_TL2       = 1,    COMMON_ART_USER_INTERFACE_MESSAGE_BOX_TC        = 2,    COMMON_ART_USER_INTERFACE_MESSAGE_BOX_TC2       = 3,    COMMON_ART_USER_INTERFACE_MESSAGE_BOX_LISTCOUNT = 4, }; #endif //IPAC_COMMON_H 

The names are concatenations of the various resource and folder names. This is important because name collisions between two similar folders or resources will keep a game from compiling.

Best Practice

If you write a tool that generates header files of any kind (especially for resources), make sure the tool updates the header file only when it absolutely must. Earlier versions of iPac wrote the header file out every time an iPac file was opened, even if no changes were made to the file. This caused much recompiling and gnashing of teeth.

Every resource in a game is uniquely identified by the iPac file name, the folder number, and the resource number as shown here:

 IPacResource buttonPushed = IPacResource(_T("common.pad"),                    // iPac file name                      COMMON_FOLDER_SOUND,         // iPac folder                      COMMON_SOUND_BUTTONPUSHED);  // iPac resource 

Other Features for Managing Resources

There are a couple of other important features any resource builder needs:

• Command line interface: All the major features should be exposed via the command line, so that build tools or batch files can change the resource files.

• Resource finalization: Each resource can be locked from further changes. When a game goes into final lockdown, this will keep unwanted resource changes out of the game.

• Checking/viewing resources: If there's a bad or corrupted resource in the file it can be impossible to find manually. If each resource can be automatically checked it will save a lot of debugging time.

• Resource extraction: Some publishers, like Microsoft, require all compressed or concatenated files to be extracted in full so they can be scanned for viruses or unwanted content. Make sure your resources can be dumped, preferably into standard file formats for graphics or sound data.