As soon as file systems appeared on the first home computers, it became obvious that they would be a powerful tool to better organize content in games. Gone were the days of a single file mixing code and data, making projects virtually impossible to maintain. With a decent file system, one file would contain the game code (the executable file), and several files would contain the associated data. This way chaos would be minimized, and work could be divided between different people who would only care about their specific files. A musician need only access the music data, the sprite artists need only work with bitmaps, and so on. Keep in mind that hardware was evolving quickly, and larger games were difficult to create without some proper organization. However, a side effect of this organization soon surfaced: What would happen if you had a single executable file that used data files laid out in a documented format, so you could easily replace the game's data files? Effectively, this enabled the creation of many games that were identical in their gameplay formula, but different in their content. You could do a scroller set in World War II and another one in outer space without touching a single line of the source code. This may seem naïve by today's standards, but try to imagine the revolution it meant for an industry where each game was an indivisible atom with code and data all mixed up. Being able to create different games by modifying the relevant data files was a transformation. Clearly, this was the way to go for the industry; and soon reusable data-driven game engines became ubiquitous because of their obvious economic advantage over hard-coded games.
In these early games, data files were nothing but memory dumps of the data structures, with little or no structure at all. Soon, developers began creating their own file formats, so no one could steal their artwork by looking into the files. Competition fosters innovation, so better systems were devised. At the same time, developers began thinking about more intelligent ways to organize their work using file systems. One of the best ideas they came up with was to use files not only for data, but also for some behavior information. Choreographed trajectories for a scrolling game's AI system could be stored this way. The choreographed AI that made most scrollers famous could be thought of not only as pure data in the form of waypoints, but also as behaviors. Behaviors can be programmed as patterns of movement that actually define the gameplay. Thus, the leap from data files to behavior systems was made. By redefining graphics, sound, and behaviors, game developers and players could quickly derive new games from old ones. Not only did this add flexibility to the development process, but it also enabled the player community to create vast collections of content. Users creating high-quality content were frequently offered jobs at the companies developing their favorite games.
Defining behaviors in the data files brought some changes to the internal game code. What was initially a totally hard-coded gaming system became a bit like an application loader and operating system, providing some internal functionality and interpreting the remainder from the data files. The internal functionality was publicized in an interface that content developers had to learn, and they then used it to implement subsequent games. Thus, game developers began to think not so much in terms of a specific game, but about generic game systems that played different games depending on the data files. The more flexible the engine was the better. Most companies kept their file formats well hidden, partly because of concerns over intellectual property and piracy.
A prototypical example of this first generation would be the Script Creation Utility for Maniac Mansion (SCUMM) system devised by LucasArts. It was a complete authoring system that helped define the game structure, dialogs, locations, and other data needed to implement many popular adventure games. It was first used in games such as Maniac Mansion and Loom, and was employed (under different versions) in most LucasArts adventures up to Curse of Monkey Island.
By the late 1980s/early 1990s, some engine standards began to surface, and their specs were made public, so users could create new missions and content, and thus lengthen the life of the game. The best-known example of this era is the incredibly popular Doom engine, introduced by id Software in 1993. After enjoying a first hit with Wolfenstein, Doom can be considered the game that defined the first-person shooter genre. Its maps and weapons are still venerated by today's standards, and many games have paid tribute to Doom in some form or another. Internally, Doom was a radical follower of the engine philosophy I have exposed. Both data and behaviors were implemented via external files, so the binary game file was really a level loader. In the case of Doom, all engine data was encapsulated in large files using the .wad extension, and were thus called WAD files.
A WAD file was just a collection of data for the game. Data was organized in large blocks; each containing a sequence of one type of information: the level geometry, monster placement, and so on. Each of these blocks was called a lump. Then, the complete WAD file was composed of three main parts:
The header consisted of a four-byte initialization string, which could be Internal WADs (IWADs) or Patch WADs (PWADs). An IWAD contained all data necessary to play the game, so all entries of the file were full. A PWAD, on the other hand, was just a patch to the core IWAD, so only some of the lumps were present. Logically, all user-created WADs were PWADs, which provided only the lumps required to define a level. Right after the header, four bytes were used to store the number of lumps in the WAD file, and the last four bytes stored a long integer that was the file offset to the beginning of the directory to make file traversal faster.
Then, the directory was just a list of entries; each taking 16 bytes. Here, the first four bytes stored the file offset to the start of the lump. Then, the middle four values stored the lump size in bytes. The last eight bytes were available to store the name of the lump, padded with zeros. To give you an estimate, there are about 2,000 entries in the directory of the main Doom WAD file. But only 10 types of lumps are needed to create a new level in a PWAD.
Now we are reaching the interesting part: the lump definition. We will focus on level creation because there are many other lumps that hold information such as menu texts and character sets. A Doom level needs several types of lumps. The most interesting are
Lumps can define both the contents of the level and gameplay elements as well (for example, the Things lump). This effectively meant that new games could be derived from the base Doom product quickly. In fact, Hexen is nothing but the Doom engine with a new set of data files and some minor additions to the WAD format (such as the capability of some potions to appear only for characters of some special class). Doom had a great influence on developers, who followed the trend and began providing complete documentation and even editing tools to encourage user-created modifications. In fact, the popular game Half-Life began as a modification to the game Quake II, which grew so much that a new, completely different game was born.
Today, the concept of a game engine has evolved a great deal since Doom. The core game code is kept to a minimum, providing only the reusable algorithms and functionality that is essential to the game. This includes all time-critical tasks such as the rendering pipeline, networking and audio APIs, and AI interfaces where external modules can be docked. Usually, part of the higher level AI is implemented through structures and scripts in external data files. Lower level, more time critical, and less variable parts of the AI are often implemented in the engine itself. Many games use a mixture of both approaches.
In addition, extensive toolsets are provided to create content for the game. Some products use their own content creation systems (CCS), whereas others have relied on heavily modified versions of off-the-shelf products.
Gmax (see Figure 1.4), for example, is a content creation system built on top of 3d max's core. It can be bundled with your game, so enthusiastic users will be able to create new game content. Coming from max's lineage, Gmax is fundamentally a modeling and texturing package, so it focuses on level creation, characters, and so on. Games such as Microsoft's Flight Simulator have included Gmax with their release. The end user gets the software for free (or included in the cost of the game), whereas the developer must have a licensing agreement with the tool manufacturer discreet in the case of Gmax.
Figure 1.4. Gmax.
Maya hasn't stood still, either.
Maya Personal Edition has been successfully bundled with popular games such as Unreal Tournament 2003. In this case, the game developers created a set of plug-ins for Maya PE, so users can export their data sets made with Maya to Unreal's own formats. Again, the software comes at no cost for the end user.
But there is more to game engine content creation tools than modelers. All commercial engines must come with full documentation because the level of complexity and flexibility has skyrocketed with each new iteration. Keep in mind that these engines are not only sold on an "as is" basis: Some developers will choose the advanced track and hard code new features into the core toolset. A typical example is Half-Life. Valve, the developers of the game, needed a skeletal animation system that was (at the time) significantly more involved than the key frame-based animation system provided by Quake. Thus, after purchasing the engine license, they had to rewrite the animation system from scratch to support the new feature.