DirectMusic and the Production Process

The Prototype and Pre-production

The first step in the whole production process was to zero in on the musical direction and thematic material of the score. This began with discussions about the style of the music with game designer Craig Hubbard. Next, I was to bring these ideas to realization in the studio in the form of a prototype. Each thematic prototype was created in my MIDI studio using all appropriate synthesizer/sampler modules and sounds available. The idea was to ignore the nonlinear aspects that the music would take on and ignore the technical limitations that the game machine would place on the music. In this way the focus of the prototype was the thematic material itself, the musical style, and an ideal set of production values.

Each prototype was mixed to a stereo wave file and presented to the designer and producer. Some themes were accepted on the first take, some were sent back to the drawing board, and others were rejected outright. By the end of the process, we had five or six themes that we were all happy with, and DirectMusic production could begin.

Composing and Sequencing

The sequences created for the prototypes (using Digital Performer on a Mac) served as a starting point for production in DirectMusic Producer. Some of the sequences were fairly complete, while others required extensive work and additional sections once brought into Producer. From Digital Performer I exported each prototype sequence in the form of a standard MIDI file (.mid). This allowed Producer to import the sequences for editing and arranging.

Due to its latency, interface issues, and some nagging bugs, DirectMusic Producer can be a difficult program to use, clogging the creative flow. This is especially true in terms of creating music sequences. It's slow going, and there's no way around it (for now). That being the case, I did as much sequencing as practical in Digital Performer in conjunction with my samplers. One key piece of advice that I can offer when using this approach is to have the instrument samples from your studio match, as closely as possible, the instrument samples of the DLS banks to be used in Producer. This is becoming easier to do, as the DLS-2 format used by DirectMusic can be easily translated to GigaStudio (.gig) and SoundFont formats. To create the game score for DieHard: Nakatomi Plaza, I had an exactly mirrored set from DLS as the target format to GigaStudio as the production format.

DLS Creation

The DLS Level 1 instruments for No One Lives Forever came from two sources; first, we licensed many instrument collections from Sonic Network (the sounds are called Sonic Implants — www.sonicimplants.com). The other samples were homemade, including some solo cello samples. Sounds are a composer's palette, and having a rich palette, despite memory constraints of the game, was key to making the interactive score convincing.

There are some tricks to creating a rich instrument collection within tough memory requirements.

• Layering sounds and resampling: When creating music in a traditional MIDI studio, layering and stacking instrument patches to create a thick-sounding timbre is commonplace. The drawback to this technique in a game is memory usage and limited polyphony. The cure is to layer your patches and resample them into a single set of samples to be assembled into DLS instruments. For example, I created a brass staccato instrument by stacking about five or six brass patches in unison (including french horns, trombones, and trumpets). One sample from this instrument using one voice sounded like the entire orchestral brass section playing triple forte!

• The use of unique or interesting sounds: One interesting timbre in a piece of music can carry the piece and make it memorable. The low cello glissando in the H.A.R.M. theme is one such example in NOLF. A generic cello patch and the pitch bend wheel would never have cut it. Instead, I brought in cellist Lori Goldston and had her record some short figures and motifs. The ponticello glissando figures were then sampled and pitched down about a perfect fourth. This became the central figure around which the rest of the piece was composed. One "live"-sounding instrument can trick our perception into hearing other parts as performed live.

• Each instrument must sound convincing when soloed: If an instrument sounds weak on its own, it most likely will not add anything to your music. I resampled many instruments with a bit of room or hall reverb on each sample. The real-time reverb in DirectMusic AudioPaths is very useful, but it certainly isn't your Lexicon-quality algorithm. Adding a bit of high-quality processing to the individual samples (be it reverb, compression, or EQ) can go a long way to get that "professional" sound back into your interactive score.

• The samples should match the individual composition: Even within different orchestral arrangements, different sets of samples are called for, depending on the pacing and mood of each piece. The "one size fits all" mentality of General MIDI will fail to give your score anything but a generic quality. Each theme of the NOLF score had some instrument sets that were built specifically for that theme — the vocal "BaDeDum" sample for its theme and the horn "blatts" for the Ambush theme, in addition to the cello samples already mentioned for the H.A.R.M. theme.

Creating looping samples posed a big challenge given the short length of the samples. For many samples, a Mac program called Infinity was used to create internal loops. The program has tools, such as crossfade looping, that help smooth out harmonic thumps and clicks common to short loops. That said, short loops are never perfect, and compromises are always made for the sake of memory constraints.

The DLS Editor

After creating and editing the individual samples, they were brought into DirectMusic Producer for DLS creation. Each NOLF theme was arranged within its own DirectMusic Producer project. Each project has an Instruments folder to help organize the DLS Collections. My tendency is to create several DLS Collections, each containing just a few instruments and waves, rather than making one massive DLS Collection containing all the instruments. This makes it more convenient to share instruments between projects.

When creating a DLS Collection from scratch, a collection was added to the project, and the edited waves were added to the collection's wave folder. Next, an instrument is created and regions assigned across the instrument. The range of a region determines how far a given sample will have to stretch up or down.

For most melodic instruments in NOLF, we found that two regions per octave was adequate. This means that each sample will be stretched up and down about a third in either direction. In crucial or exposed ranges of certain instruments, more regions per octave were necessary for a natural sound. Many instruments had various versions, depending on the needs and constraints of a given theme. For some instruments, extreme region ranges were used to get an unnaturally low, yet purposeful effect. I always experiment with a sample's useful range and often find pleasant surprises, which make their way into my music's arrangements. This is a way to get extra mileage out of your chosen samples.

Good use of instrument articulation can extend the variety of your samples and adds dynamic expression to your instruments. Copying an existing instrument, pasting it into the same collection (as to not duplicate your wave data), and then altering the ADSR articulation is the easiest way to add to your palette. Most common in NOLF is one instrument with a short "attack" (0 to .1 seconds) and its partner instrument with a longer swell attack.

Smart use of the release articulation can help smooth transitions in an interactive score in a way that straight wave files could never approach. When a transition occurs, there are often sustaining notes that must stop or change to accommodate the new Segment. Instrument release acts as a natural fade-out or decay, and each instrument can have its own release time. Being a DirectX 7 game, NOLF used DLS Level 1 as opposed to DLS Level 2 that came about in DirectX 8, meaning instruments didn't use layering or filters.

Styles, Patterns, Bands, and Segments: How They Work Together

Conceptualizing how DirectMusic and its components will work together in your project should be your first task. Having read about DirectMusic's basic functionality, you may have a good understanding of the various DirectMusic components, but different games have different needs, and this is where DirectMusic's open-ended approach is both powerful and confusing in its complexity. The solutions to these different needs are not spoon fed; they must be thought out conceptually first and then applied using DirectMusic's toolset.

For my purposes in NOLF, Styles act as project bins for patterns, Bands, and motifs. Patterns are anywhere from one to eight measures in length, corresponding to the length of logical musical phrases. These patterns are assigned to Segments using groove level assignments. Each pattern is assigned a unique groove level, which a Segment can trigger by calling that number from its own groove track. These Segments are then in turn assigned to LithTech music states.

These layers of abstraction, patterns to Segments to music states, while complex on the surface, give NOLF its nonlinear and adaptive functionality. Picture the approach as such:

• Patterns are generally short musical phrases.

• Segments use multiple or repeating patterns (triggered via groove levels) each commonly one and a half to three minutes in length (exceptions to be discussed further).

• The game's AI ties into music states 1 through 6. One or more Segments are assigned to each music state.

Figure 16-1: The Producer setup for the Ambush theme, showing project structure and Style/pattern associations.

Style and Band Setup

Each project file contained at least one Style. A Style can only have one time signature, so if a theme had changing time signatures, such as the H.A.R.M. and Ambush themes, then a Style was needed for each time signature! If this had been a DirectX 8 game, I may not have used Styles at all but used Pattern Tracks within Segments (which weren't available in DirectX 7) instead, although I do like using styles as a pattern bin, which makes mixing and matching patterns across multiple Segments easier.

Each Style contains a default Band (called band1). This Band is referenced while auditioning patterns and used during most of the creative arranging process. I call this the core Band of the project. For simplicity, I do my best to have one Band per theme and project. This approach works well, as I find it easier to expand pchannels within the Band than to create another Band for the sake of a patch change. The volume and pan info in the core Band served as the relative starting point for most patterns with any dynamic changes done using continuous controllers 7 (volume), 11 (expression), and 10 (pan). There was a continual adjusting of the core Band as an arrangement evolved.

Figure 16-2: The Band setup for the Ambush theme.

Pattern Creation and Organization

When exporting the standard MIDI files from Digital Performer, I did my best to save them out in condensed pattern length files. Duplicate MIDI tracks were deleted and tracks of the same instrument were combined in Digital Performer. Rather than save out one long SMF file, individual SMF files of one to eight measures in length were exported, so they would import more easily as patterns. Once imported, we did have to adjust many of the parts' pchannel assignments. This is because DirectMusic assigns pchannels based on the ordering of the imported SMF tracks — from top to bottom.

Other patterns were created by copying material from existing patterns and then altering it somewhat. Some patterns were created by entering note data manually.

With respect to groove levels, NOLF patterns use a groove range of one, meaning the "lo" and "hi" number are the same. This one-to-one correlation between groove numbers and patterns makes it easy to assign specific patterns to Segments. There are cases where NOLF employs broader groove ranges to create pattern-level variation; if two patterns have overlapping groove ranges or the groove range is the same, DirectMusic randomly chooses one of the patterns to play.

Instrument-level Variation

As you know, each pattern part can have up to 32 variations. Each time a pattern is played, one variation per part is randomly chosen. Multiple variations do not have to be utilized; in fact, many pattern parts in NOLF had one "hard-wired" variation to play. We created this "hard-wired" variation by disabling all other possible variations for a pattern. I found that four or five variations on two or three parts was enough variety for most patterns.

The ambient and sub-ambient patterns tended to get a deeper variation treatment, with most parts having variations. It may be that the moody, atmospheric, and arrhythmic nature of these music states lent themselves to a truly nonlinear treatment.

Variations in the suspense or action music states entailed two or three pattern parts with variation. This allowed the foundation of that music to remain consistent while providing some variety. There are many instances where instrument-level variation is not used. This was mainly due to a limited production schedule. A simple technique for creating subtle variation is to copy the original track (variation 1), paste it into another variation, and then slightly alter it. The new variation could have the same contour with different embellishments. This way, it remains consistent with the intent of the music, yet it adds interest. Another technique for using variations effectively is composing unique melodies for one pattern's variations while the other parts may have subtle variation. This technique creates a "lead" instrument that riffs over a consistent, yet changing bed of music. I ensure that each part has its own rhythmic space to play in. For instance, all of the vibes' variations may occur during measures three and five, while the flute variations are given measures two and six. This ensures that the parts won't step on each other, despite its nonlinear nature.

Composing with the Pattern Editor

You have two choices for composing with the Pattern editor: by hand editing the MIDI data in DirectMusic Producer or by recording with a MIDI controller. The truth of the matter is that there is no easy way to compose music using the Pattern editor. We did the best we could with the tools at our disposal, however.

The first method is laborious but precise — inserting notes one at a time using the mouse or the Enter and arrow keys. After a phrase is inserted, each note's duration has to be dragged or edited to the desired length. Multiple notes can be selected, copied, and pasted when a set of notes repeats. The most awkward aspect of this approach is changing the velocities of individual notes by grabbing and dragging the top edge of the selected notes. More precise changes can be made in the note's properties window. This method is completely unglamorous and slow, but it gets the job done. If you want to utilize DirectMusic's variation abilities, you'll certainly find yourself doing this sort of editing.

Method two for composing content in the Pattern editor involves interfacing a MIDI controller to DirectMusic Producer. This is done using the MIDI/Performance Options tab. Once set up, the keyboard is used to perform part variations in real time. Due to the latency of the software synthesizer (80ms on average), I suggest monitoring each performance with a piano sound from the keyboard controller. Unfortunately, the MIDI timing and sync during real-time performance is not ideal (in DirectX 7 and DirectX 8); therefore most performances need hand editing in the pattern window. At least this method allows for a certain amount of performance spontaneity.

Continuous Controllers

The use of continuous controllers is essential to breathing life into any MIDI score, and a DirectMusic score is no exception. In DirectMusic, each pattern part can have CC tracks inserted beneath its piano roll. In NOLF, controller 7 (volume) is used to change a part's general volume, while controller 11 (expression) creates the dynamic ebb and flow.

Parts without dynamic expression can sound flat and monotonous, no matter how well the part is written. Inserting expression curves (CC 11) helps convey the dramatic intent of a musical phrase. Expression curves are used in the NOLF score as crescendos, as quick swells, as fade-ins/outs, and to emphasize certain phrases of a part.

The interface in DirectMusic Producer for adding and editing CC curves is easy to use conceptually, but actually moving and manipulating curves may quickly tire your mouse hand. The CC curve concept in DirectMusic Producer is much easier to use than placing individual CC data points. I did experiment with "performing" curves using an external controller (the EMU launch pad), but DirectMusic Producer's timing and latency problems prevented the data from being recorded accurately.

This being said, there are two methods for applying CC data to parts: the slow but accurate method of inserting curves by hand (and mouse) or creating the CC data using another sequencer prior to saving out the .mid files. The key in this case is to thin out the CC data. This limits the use of processing power by the DirectMusic engine and makes the data easier to edit once it is in DirectMusic Producer.

Note

Curves that do not reset during a transition can cause CC info to "stick" at an unwanted level. This can occur when the music moves from one Segment to the next during a CC curve. To prevent this, we place CC data points at the beginning of every pattern part for those pchannels that use CC data. This technique initializes the CC data for each pattern, avoiding unwanted CC levels.

Segment Creation

Segments bring together all of the disparate patterns in a project into unified arrangements. The patterns act as building blocks for the Segments. Sometimes the order and arrangements of the Segments were planned prior to their creation. At other times, we would experiment with different orderings of patterns within Segments. This flexibility resulted in some interesting outcomes, and we often got more mileage out of our patterns. Creating short phrase-length patterns gave us this flexibility to work with patterns as if they were Lego blocks.

All NOLF Segments contain the following tracks: Style Track, Tempo Track, Groove Track, Band Track, and Time Signature Track. A few Segments also used Mute Tracks. Segments trigger patterns via the Style and Groove Tracks. The Style is inserted into beat one of the Style Track. This automatically creates a Band Track with that Style's Band inserted. From there, groove numbers are placed along the Groove Track to create the Segment's arrangement. Mute Tracks are used to reveal a pattern gradually within a Segment. For example, a Segment begins with groove level 10 with the Mute Track muting most of the groove/pattern's pchannels. Over the course of 16 measures, the pchannels are individually unmuted, slowly revealing the entire pattern.

Figure 16-3: A typical Segment from NOLF.

Setting up Performance boundaries longer than one measure was tricky for NOLF because it was a DirectX 7 game. In DirectX 8 and beyond, a Marker Track can be used to set up transition points within a Segment, but DirectX 7 did not have that feature. In fact, our work with NOLF may have influenced the addition of the Marker Track by the DirectX team. There were two possible workarounds for longer performance boundaries in NOLF. A music state can play multiple Segments sequentially using the LithTech music script. The Segment performance boundaries were set to "end of Segment." This allowed performance boundaries of varying length within a music state. Another clever solution to the lack of a Marker Track utilized the Time Signature Track. We would set the performance boundary of a music state's Segment to "measure" and place various time signatures along the Segment where transitions were to occur. In other words, the Time Signature Track had no direct correlation to the underlying meter of the music. For example, if we wanted performance boundaries every four measures of music, the time signature would be 16/4. Workarounds have often inspired the creation of features, and features (such as the Marker Track) are not created until there is a known need for them.

Motif Creation

Motifs are created within Styles. There is a folder within each Style for motifs. Motifs are created and edited in the same manner as patterns. The only difference is in functionality; motifs layer over Segments at a Segment's tempo, while patterns are triggered via groove levels from within Segments. Motifs follow the harmonic structure of a Segment (if set up to do so).

To keep things simple, NOLF motifs are short, simple phrases that punctuate and accent the underlying music. As mentioned earlier, NOLF motifs play when a direct hit is made on an enemy AI. Often, one or two staccato brass chords accent the music, or a suspenseful string tremolo rings over the primary Segment. In all cases, NOLF motifs are one measure or less in duration.

The instrument of a motif works best when it is different from the instruments of the primary Segment. This prevents the motifs from stepping on a part that is already playing. Using a different or specific range also helps in this regard. For example, if the primary Segment contains lowand mid-range brass instruments, using a trumpet motif in the upper registers will fit well in the mix rather than make it muddy. Even if the primary Segment contains a midrange trumpet, a solo trumpet motif will be heard because of its distinct timbre.

The performance boundary for motifs was always Grid or Beat, so they would play in sync with Segments, yet respond quickly to the game. This means that a motif plays no more than a half second after a direct hit and its gunshot sound. The timing coordinates perfectly with the sound of the gun, responding to a perfect shot. A performance boundary of Immediate might cause the motif to step on the gun sound and would not play in sync with the primary Segment. Performance boundaries of Measure or Segment would cause the motif to respond too late, losing the desired timing. NOLF motifs utilized DirectMusic's Chord Tracks so they perform in harmony with the underlying music. Motifs are created in a neutral key (C), and the primary Segments' Chord Track transposes the motifs according to its chord symbols. The chord symbols inserted in the Chord Track reflect the key or harmony of the primary Segment's music. In most cases, NOLF Chord Tracks contain one chord at the beginning of a Segment setting its key center, and its patterns are transposed to that key. The motifs, in turn, also perform in the key set by the Chord Track. This use of Chord Tracks differs from the more common technique of placing chords at every harmonic or chordal change within a Segment. The difference is that NOLF uses Chord Tracks to set the overall harmony and key center rather than defining each chord change. This simplified approach works because the motifs function well over any chord within its key.

Creating Transition Segments

Transitions link the six music states to one another musically. It takes a puzzle-like logic to figure out how the transitions should operate. NOLF uses specifically created Segment files for transitions instead of DirectMusic embellishments. This is because DirectMusic embellishment logic is only aware of one Segment/pattern at a time. Its logic cannot take into account both the current music state and the music state to which the music is transitioning. The transition matrix architecture, set up in the script, allows specific Segments to be assigned to each possible transition (between six music states, there are 30 transitions). These transition Segments are one to four measures in length. This duration allows enough time for convincing transitions while being short enough to keep up with the game action.

I ask myself two basic questions when composing a transition Segment: Which music state am I moving from? And to which music state am I transitioning? I write down all the possible transitions and methodically check them off as they are created. I begin composing transition Segments to and from silence — in other words, the intros and ends of each music state (again, these are not DirectMusic embellishments). The music composed for these intros and ends provides the foundation for other transitions. This is because the material written for a music state's intro may become the basis for transitioning to that music state from another music state. Also, a music state's end may also function well when transitioning from that music state to other music states. (This is the puzzle logic that I mentioned!) Sometimes I'll use the end transition of one music state and the intro of another music state as the transition between them. The end material brings the music out of the current music state, and the intro brings the music into the next state. More often, this end/intro transition is the basis for that transition, and further editing and composing is done to make it work well.

Many transition Segments function well in multiple transitions, and this saves production time. For example, the transition Segment built for music states 2 to 3 may also work between music states 2 and 4, etc. I compose one music state's transitions at a time. Thus, when I'm working on music state 2, I sequentially create transitions between states 2 to 1, 2 to 3, 2 to 4, 2 to 5, and 2 to 6. Again, this is because there are bound to be similarities among these transitions that I can reuse.

Simple transitions are often the most effective. If the music needs to stop or transition quickly, a large percussive accent brings the music to a halt. It's as if the music hits a wall, blunt and jarring, and that frequently works well within the game. There are also cases when no transition Segment is needed between music states. In these cases, the music flows directly from one music state to the next, and the release times of the DLS instruments create a natural blending or crossfade between music states.

Sometimes, one transition Segment is not adequate for a particular transition. If a music state contains a variety of musical sections, more than one transition Segment may be needed. For instance, music state 4 has an A section that is 16 measures in length and a B section also 16 measures long. If the instruments used in each section are different or the harmony and tempo vary between them, then a single end Segment may not work from both section A and section B. In a case such as this, two transition Segments are created, one for transitioning from the A section and a second for transitioning from the B section. During gameplay, the transition matrix calls the appropriate transition Segment, depending on the current playback point of the music state (more on this in the section titled "The Transition Matrix").

When creating music states and their transitions, keep their harmonic content in mind. Music states with disparate key centers cause difficult-to-execute transitions because the drastic harmonic modulation of the transition may sound unnatural or awkward. I recommend using key centers that are closely related (i.e., C major to G major) to create convincing transitions. I also use chromatic modulation in NOLF (i.e., B major to C major), and many transitions simply stay in the same key. It is easy to back yourself into a corner harmonically when creating an adaptive score. Be aware of the tonal centers of the music states as you create them, and try to think ahead about how they will transition harmonically.