Production

Music

We utilized DirectSound for all of the music in Halo to provide a dynamic musical score that is both constantly changing and reactive to gameplay elements. Each piece of music during gameplay was started and stopped in the scripting at various moments within each level. When the gameplay design for a level was nearly completed, we'd sit down with the designer and "spot" how music would play during that level. When a piece was started, it began with the "in" soundtag and continued into the main looping soundtag. This main looping soundtag contained multiple permutations, which play back randomly to create a constantly changing musical experience.

If you sit in one place in Halo listening to a piece of music, you will notice that it never plays back exactly the same way twice due to the randomization of the permutations. The music was edited so that the "in" soundfile plays seamlessly into any of the main loop permutations, and the main loop permutations can play seamlessly into the "out" soundfile when the piece is stopped by the script. These main loop permutations contain unique elements, so as they are randomly played back, the piece takes on a new flavor each time you hear it.

The script can call upon an alternate track, which is used in reaction to something going on during gameplay. This alternate track (which can have its own unique "out") might have a more or less intense score, depending on the needs of that section of gameplay. For instance, if the player is about to enter an encounter with a number of enemies, the script starts a piece of music beginning with the "in" and proceeds to the "main" looping section. If during the course of battle, the designer throws another onslaught of enemies at the player, the script could trigger an alternate track that, in this case, might be a more intense mix of the same piece of music. Then when all enemies have been vanquished, the script could either stop the piece of music (playing the "out" soundtag) or return to the "main" looping section, indicating to the player that the battle was over. This system means that the player gets a constantly changing musical score that reacts to gameplay elements as they appear.

Another example of this would be scripting an ambient piece of music and then scripting a more rhythmic "alt" track that crossfades when the player runs into enemies. For instance, we might begin a piece of music when the player enters a new area. This music is ambient in nature, without a clearly identifiable rhythm. When the level designer decides to ambush the player with enemies, the alternate track is triggered, which could either begin playing at the end of the current loop or crossfade at the moment it is triggered. In this case, the alternate track could be a more rhythmic version of the ambient piece or depart from the ambient piece entirely. As when the enemies have been vanquished, we can instruct the level designer to script the tag to play the "alt out" or return to the "main" loop to continue playing the ambient music.

Other elements that are useful, especially during ambient pieces of music, are details and multiple tracks. Details are soundtags full of soundfiles that could be triggered anytime during the playback of a piece of music. We can set which group of files to use and a time range in which to randomly trigger them. Additionally, we could layer a second track on top of the first one, and having both "main" loops playing back with multiple permutations of varying lengths would provide an additional layer of variability. These techniques allow a lot of flexibility and give the player variation and dynamic playback while utilizing a traditional form of playback (streaming prerecorded audio files). All music soundfiles were ADPCM compressed 44.1 kHz stereo AIFF files and loaded in 128K chunks from the hard drive (sort of a faux streaming).

SFX

The background stereo ambiences in Halo were created in a similar manner to the music. All ambient soundlooping tags were assembled identically to the music files described above. We had multiple tracks playing in many cases (such as an outdoor ambience plus a wind track), each with multiple permutations in the "main" loop tag, as well as detail sounds that could be randomly triggered and were placed randomly in 3D space.

One technique to highlight (which was used extensively in the music tags as well) is permutation weighting. Permutation weighting is the ability to assign different probabilities to each permutation to control how often a given permutation is played. For example, there are six permutations of a "main" loop in an outdoor ambience with lengths varying from two to eight seconds and 27 seconds of material. In one of those loops, there is a distinctive owl hoot, which would be repeated every 27 seconds if the material was played back in the same order each time.

Given the randomness of our permutation playback, you actually might hear it on average every 27 seconds but sometimes a little less and sometimes a little more frequently. If that distinctive hoot is still heard too frequently, we can use permutation weighting to assign a high skip fraction to that one permutation that tells the audio engine to skip this permutation x-percentage of the time. We adjust this number until we hear the distinctive hoot at a frequency that seems natural.

We use this same technique extensively in the music (for unique musical flourishes) and in SFX like the example above (or for an annoying rattle in a piece of machinery we only want to hear every so often), as well as in dialog permutations that should be heard rarely ("I'd have been your daddy but that dog beat me over the fence!" for example). As with the music, all stereo ambiences were 44.1 kHz ADPCM compressed soundfiles. All other sound effects in the game were 22.05 kHz mono ADPCM soundfiles and played back through the best parts of DirectSound's programming, such as 3D positioning, DSP, including reverb, occlusion, Doppler effect, HRTF, and of course real-time 5.1 Dolby Digital surround sound.

We could attach audio to anything in the game, and because of that, the 3D positioning function was critically important to enable the player to sort out where audio elements originate. Knowing that a Marine "had your back" just over your right shoulder brought a sense of security, just as hearing a Hunter's metal armor behind you would bring a sense of impending doom. There were many audio elements in the game that received 3D positioning, such as weapon sound effects, bullet impacts, speech, mechanical objects, particles such as flying dirt or sparks, and outdoor detail ambiences, such as wildlife, rivers, and waterfalls, to name a few. In essence, everything you hear in Halo that is not music or an ambient bed is 3D positioned. With the Xbox audio card, you can play back a maximum of 64 3D positioned sound effects simultaneously, leaving 192 2D voices free.

We also utilize the DSP power that is built into the Xbox audio chip. The onboard Motorola chip is identical to the chips that are used in some Pro Tools DSP farm cards that give the Xbox DSP processing power unlike any other platform. We can use as many reverbs per level as we wish and can switch between them on the fly without generating a lot of CPU overhead. It doesn't matter if we use a small room reverb or a huge airplane hangar reverb; the cost to the engine is the same.

We use, on average, about six reverbs per level, embedding information about reverb parameters at a given location directly into the level geometry (right alongside the ambience soundtag that should be played). There is not a single area in Halo where the 3D audio is not being played with reverb, and it certainly adds a level of realism to the environment. Standing inside the shaft on The Silent Cartographer level and listening to the echo of reloading your pistol helps underline the sheer size of the geometry that the artists created for Halo.

Although we do not utilize other DSP functions extensively for Halo, we do use a filter for those times in the game when the Master Chief descends underwater or becomes immersed in pools of liquid coolant. The filter simulates hearing sound through water very well. We're looking forward to using this and other functions of the DSP in the future.

All 3D audio is subject to occlusion, obstruction, Doppler, and HRTF. Anytime a solid piece of geometry gets between the player and a sound source, it is occluded and obstructed. A good example of this is in the hangar bay of the alien ship where there is a dropship with the engine running. As the Master Chief steps behind a column, the sound of the engine is occluded, rolling off both the gain and the high end of the sound. Other geometry that occludes sound in Halo includes walls and cliffs. A grenade explosion sounds muted when you are around a corner as opposed to having it in an unobstructed field of view.

Sound sources emanating from objects that travel through space in Halo, such as the Sentinels or vehicles, were subject to a Doppler shift. A moment during Halo's development that stands out is when we were first testing our looping sounds as they related to Doppler and hooked up the sound of a hot rod engine to the rocket launcher.

Figure 17-1: A rocket launcher in action.

We hooked up the hot rod loop to the tag for the launcher's projectile, walked the Master Chief across a big field away from the camera, and fired it back toward the camera. What ensued was of great amusement as the rocket flew past, generating the sound of a hot rod going about 100 miles an hour and exploding in the cliff behind us. It worked like a charm, and we left it hooked up that way for a week or so for everyone to enjoy. The Doppler shift in DirectSound worked perfectly and added a lot of realism to sound sources that move around.

HRTF (Head-Related Transfer Function) also add to the audio experience by filtering sounds, depending on which direction the character's head is facing. You can hear it affect the dialog in the cinematics as well as in the sounds that play during combat. Probably the best way to hear this effect, however, is listening to your own footsteps. As you move the Master Chief around in Halo, listen to the sound of his footsteps on various surfaces. Then run through the same areas looking down toward his feet rather than straight ahead and listen to those same footsteps; the difference is stunning.

Occlusion, obstruction, Doppler, and HRTF are all aspects of DirectSound that highlight its capabilities to utilize the same source filtered appropriately to add a sense of realism without having to author content specifically to achieve the same effect.

Surround Sound

One built-in feature to the Xbox chip that is absolutely amazing is its ability to generate real-time 5.1 Dolby Digital output. We encode and decode Dolby Surround on the fly from dynamic game audio so that a player gets a full surround audio experience throughout the game. This is a first in console gaming — and a great first at that!

There are many advantages to implementing 5.1 audio, from alerting the player to the presence of enemies to surrounding the player with ambience and music. While we did not premix the music or ambience in 5.1, we were able to surround the player with sound. Stereo music and ambience were played 100 percent out of the front speakers and 50 percent out of the rear speakers and also sent to the LFE. Non-positional dialog was sent 100 percent to the center speaker and 50 percent to each of the front speakers. All 3D positioned sound was played back in its appropriate position among the four surround speakers and also sent to the LFE. The Halo audio engine queries the Xbox at startup to see whether it is set to stereo or Dolby Digital and plays back the audio appropriately to avoid doubling up audio signals in the stereo fold down.

Attaching Sounds to Animations

In Halo, there were thousands of animations created for several dozen characters. The animator provided us with movies of a group of animations that needed sound effects, and we would then sound design in Pro Tools, making multiple permutations for each one as necessary. For animations such as characters getting in and out of vehicles, characters moving around, or the sound of the Flood Infection Forms skittering about, we generated an appropriate sound effect and attached the soundtag directly to the animation tag for that character. We could also trigger it at a specific frame of the animation in order to ensure the tightest sync possible.

Figure 17-2: An Infection Form closes in on the Master Chief.

Real-time Tracking of Velocity and Pitch

The engine can also trigger sound effects based on velocity. Sounds such as vehicle or shell casing impacts will play louder or softer, depending on the object's velocity at the moment of impact. If you watch a shell casing hit the ground, it will bounce several times with each repetition softer than the one before. For a variety of different sounds, including vehicle engines, we had the ability to vary pitch according to a scale. In the case of a vehicle engine, the scale would be the RPM of the engine at any given moment. Like most driving games, we used multi-layered looping samples at various areas throughout the RPM spectrum, so the samples were never pitch bent very far.

Figure 17-3: The jeep wins a game of chicken.

We also use the scale for sounds other than engines, such as the Banshee contrail or the human dropship hovering. For the contrail, we raise the pitch on the loop as it thickens, lower the pitch as it thins, and fade it out when you fly straight. For the human dropship, we actually use the scale to modify the gain instead of the pitch so that when the dropship hovers, the player hears a swirling, windy sound. When the downdraft jets are activated, the gain ramps up on the wind and then ramps down as the downdraft jets deactivate. The scale can modify pitch, gain, and skip fraction, depending on the requirements for the sound effect.

Figure 17-4: Dropships on the beach.

Cascading Sound Effects

Cascading sound effects include sounds such as breaking glass or rocks and gravel that explode off cliffs or are kicked up from vehicles. Glass breaking is the easiest example to picture. We attach an initial breaking sound to the moment when a glass surface was broken. After that point, however, there are several factors that would affect how the remainder of the glass breaking should sound. Depending on the size of the original piece of glass (how much glass would fall) and the distance of the glass from a solid surface, we had the ability to trigger glass breaking sounds in "cascades" attached to the impact of each glass particle. Keeping in mind that there are often hundreds of particles involved, we triggered soundtags in the following manner.

When the first particle impacts, we call "glass small," which would be a soundtag of a very small glass particle hitting a surface. We set a number (eight, for example) of repeated calls within a specific time period after which the audio code triggers "glass medium," which would be a soundtag of multiple glass particles breaking. On that tag, we set another number (five, for example) after which the code would trigger "glass large," which is a soundtag of the mother of all glass smashes. After the final soundtag is triggered, we could start the entire cascading sequence over again if there is still more glass breaking. Each of the three levels would, of course, have many permutations so that each time glass breaks, you do not hear the same series of sound effects over and over. This gives us the ability to sync to individual particles in combination with the "chorus" effect achieved by recording a larger, more complex event.

Dialog

The dialog in Halo was one of the areas that helped to give Halo a unique flavor. There are two types of dialog: cinematic dialog that is a traditional linear script and dynamic dialog. As you play through Halo, your Marines will dynamically alert you to what is going on around you, warn each other and you of danger, chastise, apologize for allies getting shot or killed by friendly fire, and generally amuse you. This aspect of Halo's audio could not have succeeded without the hard work of AI (artificial intelligence) programmer Chris Butcher who gave us many possible AI states in which a character might say something and controlled how much was said to make it all seem natural.

Figure 17-5: Shotgun-toting Marines are great to have around.

In addition to the seven distinct Marine voices, we also created four unique voices for the alien races. To make the voices even more colorful, we used a different actor for each character and utilized various dialects (that really is an Australian in there). We went out of our way to have as many permutations as possible for each AI state (sometimes over 20) so that you can play the game for a long time and rarely hear a repeated line. We also used the same AI framework for all of the alien speech, some of which was intelligible (grunts in particular) and some that was not. With such a good speech system, we merely had to apply our production values to the recording sessions. We used professional (AFTRA/SAG) talent and produced, recorded, and edited the sessions ourselves to ensure we got exactly what we wanted. Actors were cast out of both Chicago and Seattle and came to the studio multiple times to record initial takes and pickups later on.

Figure 17-6: A Jackal has an unpleasant meeting with the front of the jeep.

The cinematic dialog also works well due to both a great storyline by the Bungie team and excellent writing by Joseph Staten. Scripts for the cut scenes and other interior scenes during gameplay were written and storyboarded so that we could clearly deliver the plot to the player. There is about one hour and 15 minutes of cut scenes in Halo, and that puts this part of Halo's production on par with a short feature film.

The Final Mix

In the commercial, film, and theater industries, the final mix is one of the most important steps in post-production. Those of you who are in the game industry are likely laughing right now, as it is rare that we have the time necessary at the end of a project to do a final mix. In Halo, we got a chance to do some broad stroke mixing but were never able to get down to the minutiae that make up an ideal final mix. It's almost as if Halo is an interactive movie that changes every time you play it, and while we need to be cognizant of that fact, we can still do more to make sure that the mix "sits" nicely in all cases. There are still elements of the mix that annoy us (as there will likely be in every game we do), and all we can do is seek to minimize them as we approach the finish line for each game.