Chapter 4: Interactive and Adaptive Audio

Scott Selfon

So far, we have covered playing back basic linear audio and adding aspects of variability to that music. The next step is to make the audio dynamic — that is, allow it to respond to events that the user or program creates (i.e., interactive and adaptive audio). For instance, computer games often want to convey a certain emotional level based on what the current game state is. If the player's health is low and many enemies are around, the composer would often like the music to be more frantic. This general concept of an "intensity level" is often difficult to communicate to the programmer and even more challenging to implement convincingly in an application.

Variable Intensity Music and Ambience

DirectMusic provides several potential solutions for creating dynamic music. In this chapter, we cover two in particular — the use of DirectMusic Style files and the use of Segment-based branching.

DirectMusic Style Files

Perhaps the easiest way to implement intensity level is to use DirectMusic Style (.sty) files, which contain numerous musical fragments, each with an assigned intensity level. Audio producers compose one or more patterns (musical loops) within a Style. A pattern is composed of a Pattern Track, those same MIDI-supporting Tracks we've already used in Segments. Each pattern is assigned a groove level range, DirectMusic's term for musical intensity. When the programmer alters the groove level based on application events, a new pattern is chosen automatically based on the new intensity requested.

When a new Style file is created (File>New…, or the Ctrl+N shortcut), it appears in the project tree with several folders.

Figure 4-1: A newly created style file. Remember that design-time file extensions end with p, so this is a .stp file. The run-time version of the file will be a .sty file.

The Bands folder contains one or more Bands (information about per-channel volume, pan, and patch change settings) that can be dragged into a Segment's Band Track. Typically, you will just use one Band per Style, and that Band is automatically inserted when you drop the Style into a Segment. More information on that in a bit.

Motifs provide a subset of the functionality available to secondary Segments (and for history buffs out there, they are actually the predecessor to secondary Segments). Recall that one or more secondary Segments can be played at a time and are typically layered on top of a primary Segment, whereas only a single primary Segment can be playing at a time. While motifs may appear somewhat convenient (all living in a single file), there are several significant drawbacks to motifs:

• They cannot be accessed directly using DirectX Audio Scripting. DirectX Audio scripts can only play and stop Segments, not motifs (which reside in Styles).

• While they can be played directly by a programmer, the composer needs to manually embed a Band within the motif to ensure that its expected instrument patch changes are applied before it is played.

• While motifs support variations, it's up to the game programmer to keep using the same instance of the motif in order to avoid resetting the variation order.

In short, for most cases, secondary Segments are much more useful and convenient than motifs.

So we're typically going to have a Style with a single Band, no motifs, and several patterns. Our default Style has a single pattern. If we double-click it, the Pattern Editor (looking quite similar to the Segment Designer window) opens. MIDI controller editing is identical to the behavior in Pattern Tracks in Segments.

Figure 4-2: The Pattern Editor window.

There is a new Track at the top of the part called the Chords for Composition Track. This Track, covered in more detail in Chapter 6, tells DirectMusic what the original function of the music was so that DirectMusic can make an educated choice when revoicing the music to work over a new chord or key. For now, we leave it at the C major default.

Note

DirectMusic Style files only support Pattern Tracks and the Chords for Composition Track and therefore DLS Instruments. In particular, Styles cannot be used in conjunction with Wave Tracks, so consider Segment-based branching if you are using prerendered and/or streamed music.

Opening the properties page for a pattern gives us access to groove range and other pattern-related functions. This is also where you can adjust the length of this specific pattern.

Figure 4-3: The properties page for a pattern within a Style.

We've discussed groove range earlier. An audio producer can assign a pattern to a single groove level if they wish or a range of appropriate groove levels. Typically, a range of values is used. This allows for two things:

• Pattern variability. If two patterns have overlapping ranges, a requested groove level within the overlap will randomly choose between the two patterns (all other aspects of the patterns being equal — see below for more details on how a pattern is chosen).

• Easy addition of patterns later. Let's say the composer creates three patterns — a low-intensity pattern (assigned to groove range 0–25), a medium-intensity pattern (assigned to groove range 26–75), and a high-intensity pattern (assigned to groove range 76–100). After trying the application, the composer might decide another pattern with intensity between "medium" and "high" is appropriate. He can create a new pattern without the programmer needing to make any changes to his code. By contrast, using a single groove level, rather than a range, for each pattern would probably require the programmer to change how he determines the groove level.

The Destination Range option can only be used when you know what groove level your music is heading toward — that is, generally only when groove levels are authored directly into a Segment via a Groove Track (see the "Using Styles with Segments" section later in this chapter). With destination range, DirectMusic will actually "look ahead" to what groove level is going to be needed next before choosing which pattern should be played. While this could be somewhat useful when "ramping" from one groove level to another, the idea of authoring groove levels into content is typically contrary to the desire for dynamic manipulation of groove levels. For this reason, destination range is rarely used.

The Embellishment area allows you to add even further control to how a pattern is chosen. Embellishments are generally specific patterns that you can call for without changing groove levels. For instance, you might be playing a piece of music at groove level 50 and you want to transition to a new piece of music, also at groove level 50. Rather than temporarily changing the groove level to pick a convincing ending pattern, you could create a pattern still assigned to groove level 50 that is an end embellishment. To further strengthen the flow of the transition, the new Style could have an intro embellishment pattern. Then when the programmer or scripter wants to start playing the second piece of music, he can say to play ending and introductory embellishments, and these specially assigned patterns will be played between the two pieces of music. The naming conventions for embellishments (intro, end, fill, break, and custom) are actually somewhat arbitrary — from a programmer's or scripter's point of view, there's no difference between most of them. The exceptions are that an intro embellishment is always picked from the Style you are going to, and the end embellishment is always picked from the Style that you are coming from. Of course, if you're sticking with the same Style and just wanted a brief pattern change, even these have no difference in function. Note that custom embellishment types (with a range from 100–199 to help keep them from becoming confused with groove ranges) can only be requested programmatically or within content; DirectX Audio Scripting does not support them.

The Chord Rhythm option adds yet another layer of control over how DirectMusic chooses which pattern to play, assuming the Segment uses chords (in Chord Tracks or via ChordMaps). The audio producer might have one pattern that is more ideal to a rapidly changing progression and another that works better under a static chord. DirectMusic will look for a pattern with a chord change rhythm that maps closest to the currently playing Segment's chord rhythm.

Choosing a Pattern from a Style

So after all of these settings, how exactly does DirectMusic choose which pattern to play at a given time? In descending order of priority:

1. Look for the embellishment requested.

2. Look for the groove level requested.

3. Look for pattern matching destination groove level to next groove change. Again, as destination groove level is rarely known ahead of time, this particular rule is rarely used.

4. Longest pattern that fits available space (between now and end of Segment or next groove change).

5. Look for a pattern with an appropriate chord rhythm.

6. In case of a match in all other settings, randomly choose one of these matching patterns to play.

The fourth rule is often the most significant to keep in mind. The argument here is that DirectMusic should pick the musically "strongest" progression, which would mean to prefer to play a single eight-bar pattern rather than a one-bar pattern eight times. But the rule also applies when you have multiple longer patterns to choose between. If you had been anticipating using variable length patterns, it's generally not easy to get around this restriction but it does allow for a simpler composition and musical transition process.

Building Up Styles

The most basic use for Styles is what we like to call additive instrumentation. Each pattern is authored identically to the previous (lower groove level) one but adds another instrument to the mix. This can work quite effectively and is relatively easy to implement. The audio producer creates a low groove level pattern (which could be silence, basic rhythm, or perhaps just subtle ambience). They then create additional patterns, each at a higher groove level and each adding a new instrumental line or additional rhythmic complexity.

Primarily due to the pattern-choosing rules discussed above, it is often easiest to create every basic pattern with the same length. Embellishment patterns of differing lengths can help break up the rhythm somewhat, though all patterns must use the Style's global time signature for their playback length. Individual parts in a pattern can be varying lengths and time signatures to help distract from the feeling of constant time signature.

By the way, groove level and pattern length can be viewed in a somewhat easier-to-read interface via the Style Designer window. The window is opened by double-clicking on the Style itself in the project tree rather than any of its patterns and displays information on all of the patterns (their length, groove ranges, embellishments they are assigned to, and so on).

Figure 4-4: The Style Designer window for a Style with five patterns for varying groove levels. The highlighted pattern is an ending embellishment (as indicated by the E in the Embellishment column).

Using Styles with Segments

Remember that the basic unit of playback (as far as the programmer is concerned) is typically the DirectMusic Segment, not a Style file. So we're going to want to create a Segment that actually "plays" our style. Several Track types come into play when instructing a Segment on how to use one or more Styles:

• Style Track: This Track is the basis for using Styles with Segments. Typically, the Style Track will consist of a single Style placed in the first bar. However, there's nothing wrong with changing Styles as a Segment proceeds.

• Groove Track: This is the Track where the audio producer can set the current groove level for the Segment. This Track also allows for a Segment to manually request an embellishment pattern (for instance, play an intro embellishment at the beginning of the Segment). Inserting a Style into a Style Track automatically makes changes to several other Tracks in a Segment, inserting new Tracks if necessary.

• Time Signature Track: This Track becomes locked to the time signature of the current Style, as you can't override a Style's time signature in a Segment. If additional Styles are inserted later in the Segment, similarly locked time signature events (which appear "grayed out") will appear.

• Tempo Track: The default tempo for the Style (specified in the Style's property page and the Style Designer window) is inserted on the first beat of the bar where the Style was entered.

• Band Track: The Style's default Band is inserted one tick before the bar where the Style begins playback. This ensures that all of the instrument changes and volume/pan settings are applied just before the first notes are heard (assuming no notes begin before beat one — if any notes do occupy the pickup bar, you should manually move the Band even earlier). Note that the Band inserted into the Band Track is a copy of the Style's Band, not a "reference" to it. That is, if you now go back and edit the Style's Band, those changes are not applied to the Band in the Segment. Similarly, if you edit the Band in the Segment, those changes won't be reflected in the original Style's Band. We typically manually drag the Style's Band into the Segment just to make sure we are not using an old version of the Band.

So all of these Tracks are automatically filled in for us. But what about the Groove Track? Because the groove level is something we want to be dynamically changed by the programmer while the application is running, composers don't typically author groove changes into the Groove Track beyond an initial setting (typically the lowest groove level) in the first bar.

Figure 4-5: A typical Style-based Segment. Groove level 1 is specified in the first bar. When the Style was inserted, the other tracks were created and/or modified automatically.

Before we completely dismiss the Groove Track, it is still quite useful for auditioning groove changes within DirectMusic Producer. After composing a bunch of patterns of varying groove levels, you might create a long Segment, drop in the Style and a few groove changes, and listen to how the performance sounds. You could even use variable groove changes (the +/– option in a groove change's property page) or experiment with the embellishment types.

As far as the length of the Segment, you have several options. If you are using patterns of varying lengths, you'll generally want to make the Segment the least common multiple of those lengths. For instance, if one pattern is four bars and another is three, you'll want to make the Segment 12 bars long (and infinitely looping). This just makes sure that, assuming a single groove level (and often a single pattern) is played for a long time, the pattern won't be interrupted or cut short by several bars due to the Segment ending. Worse, remember that the pattern chosen is also influenced by how much space is left before the end of the Segment — so you might get a completely different pattern chosen right at the end if the pattern length did not match the Segment length.

For the specialized (and easier to manage) case where all patterns have exactly the same length, creating an infinitely looping Segment of that same length is generally preferable. This will also give us some added flexibility in how transitions between the groove levels occur, particularly with the ability to successfully use alignment transitions.

Changing the Playing Patterns

If you've experimented with groove changes to audition your patterns in a Segment, you might notice that the new groove level's pattern didn't always take over immediately. The reason behind this is that a groove change tells DirectMusic to choose a more appropriate pattern when the currently playing pattern finishes. Certainly, the application might change groove levels quite a bit, and you wouldn't want new patterns abruptly starting and restarting frequently. However, if you've created a lot of 20-bar patterns, the change to the new groove level could definitely take quite a while and more time than you would hope.

There are several options for making pattern changes occur more frequently. The easiest will involve the programmer or scripter. Every time he adjusts the groove level, he can also restart the currently playing Segment using any of the transition types we've already discussed. For instance, the audio producer could instruct the programmer to restart the Segment at the next bar. This means that a groove change will "take effect" (a new pattern will begin playing) within the next bar. Or, in the above example, the audio producer could structure his 20-bar patterns such that they're actually composed of five four-bar-long sections. The audio producer could then put a Marker Track in the Segment (note that the markers are not in the Style file, as Style patterns don't support Marker Tracks) at the natural break points for the music, and the programmer could restart the Segment on a marker boundary. One difficulty with this solution is that we are restarting the Segment (and thus whatever the appropriate pattern is) every time we touch the groove level. This can be problematic when smaller groove level changes shouldn't really affect the pattern we're playing. For instance, if our "low level" pattern spans groove levels one through 25, it doesn't really matter to us if the groove level changes from one to two. The same piece of music should just keep playing. But in the simplest implementation of the above solution, we're retriggering the Segment — and thus restarting the pattern — every time the groove level changes. Depending on how frequently these changes occur, we could get very intimately acquainted with the first few bars of the pattern and never hear the 20th onward. One way to combat this is for the content creator to instruct the programmer as to when the Segment should really be restarted versus allowing it to continue with the newly adjusted groove level.

Based on the above issue with potentially restarting the playing pattern from the beginning constantly, another commonly used technique is to use alignment to Segment transitions. Remember, in this kind of transition, we jump to the same relative location in the destination Segment as we were at in the previous Segment. So if we were at bar four beat two, we jump to bar four beat two of the new piece of music. In the particular case of adjusting groove levels, both the source and destination Segment are the same piece of music. The retriggering is enough to bump us over to a new pattern more appropriate for the current groove level. But with the use of alignment, we transition over in mid-pattern rather than restarting the Segment. This technique is a bit trickier to manage than the previous one. Remember that if we're starting in the middle of a pattern, any notes that would have started playing prior to that point would not be retriggered — so for instance, if our destination had some nice string pads that had started on the downbeat, we wouldn't hear them if our transition occurred on beat two. An added complexity occurs if our patterns had different lengths, as they now may transition to musically awkward positions in each other. This is another reason why it's often easiest to keep all patterns at the same length.

Figure 4-6: Attempting to transition using Segment alignment can be difficult for patterns of differing lengths. In the above diagram, Pattern 1 is four bars long, and Pattern 2 is five bars long. Below them is a display of the 20 bars of a looping Segment, so we can see how they line up over time. If we transition from Pattern 1 to Pattern 2 at, say, the end of bar eight, we're transitioning at the end of Pattern 1 but into the start of the fourth bar of Pattern 2. If both patterns had been the same length, we would be making a potentially more natural transition from the end of Pattern 1 to the beginning of Pattern 2.

Segment-Based Branching and Layering

An alternative to DirectMusic Style files for creating variable intensity music and ambience is to use multiple Segments. Remember that with strictly Style-based playback, you are restricted to DLS instruments. With Segments, you have the ability to mix and match between DLS instruments and prerendered streaming wave files. There are two primary ways that Segments can be used to shift intensity: branching and layering.

Branching Playback

In the case of branching Segments, we change the Segment we are playing at a musically appropriate boundary, based on some event that changes the emotional level we want to convey. In the simplest implementation, branching just consists of authoring complete pieces of linear music for each emotional intensity, and then picking logical places where one piece could jump to the beginning of the next. Often measures are appropriate boundaries; if not every measure is, the composer can use markers to dictate which are.

Sometimes the above can result in abrupt and unsatisfying transitions, particularly if the two pieces of music were not written with each other in mind. The jolt from playing a slow-moving piece of music to a pulse-poundingly fast one might be appropriate in some circumstances, but in others it can distract more than intended. A potential solution here is to use transition Segments.Aswepreviously discussed, when you tell a new Segment to start playing, you can specify a transition Segment that should be played before this new Segment starts. A common solution to the abruptness issue is to create quick little "bridges" that ramp up or down the tempo, smooth out any instrument entrances and exits, and so on. Of course, this can lead to a bit of extra work on the composer's part — you might find yourself writing transition Segments for every possible configuration (slow to fast, fast to slow, slow piece 1 to slow piece 2, slow piece 2 to slow piece 1, and so on) unless you're clear as to what pieces could possibly jump to other pieces. Building up a roadmap beforehand can aid both in the planning and authoring of transition Segments.

Figure 4-7: A sample diagram for five pieces of music. Each arrow indicates that a piece of music will need to be able to transition to another piece of music. For this scenario, we assume that you have to somehow "engage" enemies in combat, so you'll never jump right to the "fighting" music from the "no enemies" music. Similarly, if you decide to flee the room, you'll disengage from the enemies first, so you don't have to worry about the "fighting" music jumping directly to the hallway "ambient" music.

Yet another alternative use of branching is Segment alignment. Remember that by default, a new Segment will start playing from the beginning. Using alignment, we can make it start at the same position where the last Segment left off. With this technique, we can achieve results similar to the Style-based playback above, but with the benefit of streamed waves. Using this technique, you could author several different versions of the same audio track, with different mixes and increased instrumentation in each version. Again, using markers (in marker tracks) to control what are appropriate boundaries, the music can seamlessly ramp up or down to another wave based on the scenario.

Layered Segments

The other effective Segment-based tool for creating dynamic music is layering. Here we keep playing the same basic piece of music, but we layer other pieces of music over the top. This involves the use of secondary Segments. Remember that only one primary Segment can be playing at a time, and playing a new primary Segment automatically stops any currently playing ones. But you can layer as many secondary Segments on top of a primary Segment as you like. Don't forget that whether a Segment is played as primary or secondary is in the hands of the programmer or scripter, so the composer needs to communicate this information beforehand.

A common use of layered secondary Segments is as motific elements. Indeed, these secondary Segments replace the somewhat outdated Style-based motifs, which were much less flexible and required extra effort from the programmer or scripter in order to use them. As an example, let's say our character's love interest enters the room, and the love theme plays on top of the basic music track. The use of Chord Tracks (described in more detail in Chapter 6) can make this all the more effective. The love theme can be played in the appropriate key, and even repitch itself based on the chord progression of the underlying music.

As a brief example, let's take any old Segment we've already authored and add a Chord Track to it if it doesn't already have one. (Right-click on Segment > Add Track(s)…, and choose Chord Track from the list.) Now let's right-click on the Chord Track (the leftmost part of the track that says "Chords") and bring up its properties. Odds are the track thinks our piece of music is in the key of C because we haven't told it otherwise. Mine happens to be in Eb, so I update the property page like so:

Figure 4-8: Chord Track Properties dialog.

We'd probably also want to adjust that C major 7 chord that appears in measure 1 by default. Let's make it an Eb major 7 instead. (I'm not going to bother entering the full chord progression, but we'll use this particular chord in a moment.) We right-click on the CM7, choose Properties, and modify the property page like so:

Figure 4-9: Our Eb major seventh chord. Adjusting layer 1 (the bottommost chord layer) defaults to automatically editing all four layers, so you would only need to adjust the chord (four-octave keyboard on the left) and the scale (one-octave keyboard on the right) once. Right-click and uncheck Auto-Sync Level 1 to All to change this behavior.

Now let's create our romantic theme. We create a new Segment with a Pattern Track and a Band Track (making sure that the Band and pattern pchannels don't interfere with any from our original piece of music). As far as the actual music, I'll just borrow a recognizable theme from Tchaikovsky's Romeo and Juliet:

Figure 4-10: Our (well, Tchaikovsky's) love theme. Note that we've authored it in the key of C (not having a Chord Track implies this) and named the Segment "LoveTheme." The Pattern Track part uses performance channel 17 (which can be adjusted from the property page), and we disabled all variations except our theme on variation 1. Our band similarly just has an entry for performance channel 17 telling it what instrument to play.

If we played our theme as a primary Segment, it would play in C major. But if we play it as a secondary Segment against our piece of music in the key of Eb, it will transpose to this key. To try it out, open up the Secondary Segment Toolbar (from the View menu, choose Secondary Segment Toolbar), and choose our Segment from one of the drop-downs.

Figure 4-11: The Secondary Segment toolbar.

You can specify the boundary it will audition on by right-clicking the Play button. Now play the primary Segment from the main transport controls (click on the Segment to give it "focus," which will make its name appear next to the primary segment's Play button). As it plays, play our love theme from the secondary Segment toolbar's Play button. The secondary Segment transposes to fit the key of our primary Segment. Similarly, this theme could be played against any of our linear compositions in the proper key just by authoring a Chord Track into those Segments and specifying the proper key and scale in a chord in the track.

Transposition is one thing, but as a more advanced use, we might want our theme to actually follow the chord progression of our primary Segment. Let's add another chord to our primary Segment and start off with a dominant-tonic (V-I) progression. We'll move that Eb chord to bar two (by dragging or copying and pasting) and insert a Bb chord in bar 1.

Figure 4-12: Our new chord.

Figure 4-13: Our primary Segment.

Now when we play our secondary Segment as this primary Segment plays, the notes will conform to the current chord as well as the scale. We can adjust the specifics of how our secondary Segment adjusts itself from its property page, and can even override these settings on a per-note basis if we wish.

Figure 4-14: The Default Play Mode area determines how our secondary Segment responds to chord changes. In the default behavior, chord and scale are used to determine appropriate notes, but there are also settings to only transpose according to scale, to remain fixed to the authored notes (useful for drum tracks), and so on.

Alternatively, secondary Segments can be played as controlling secondary Segments. In this case, rather than taking on the characteristics of the primary Segment, they force the primary Segment to take on their characteristics of tempo, chord progression, and so on — whichever tracks are present in the secondary Segment are used to replace to the primary Segment's. Going back to our previous example, we can add a Chord Track to our secondary Segment telling DirectMusic that our theme was indeed authored in the key of C, overaCmajor chord. If we then played it as a controlling secondary Segment over the primary Segment, the primary Segment would now take on the chord progression of the secondary Segment.

Figure 4-15: Adding a new Chord Track and C major chord to our secondary Segment.

Figure 4-16: Setting up a Play button in the secondary Segment toolbar (by right-clicking on it) to play a controlling secondary Segment. Don't forget that this is an audition setting only; the composer would want to tell the programmer or scripter that the Segment should be played as a controlling secondary Segment.

Remember that Sequence Tracks do not respond to chord changes — they are static, non-pitch-varying tracks, analogous to traditional sequenced MIDI tracks. So the fact that our controlling secondary Segment has switched out the chord progression won't be heard in the primary Segment unless any of its notes are played by Pattern Tracks. But if our music was placed in a Pattern Track, and we had traced out the chord progression for the music in our Chord Track, the primary Segment would sound quite different over this new chord progression. To repeat a useful scenario for this, background music authored in a major key could switch to a minor key when the villain enters the room, just by playing a controlling secondary Segment composed of minor chords. Indeed, there is nothing that says a secondary Segment has to be composed of any notes at all — it is perfectly acceptable to use them for Tempo, Chord, Style, Wave, and any other tracks.

In this chapter, we have attempted to supply an overview and a basic understanding of some of the power and flexibility DirectMusic can bring to an interactive application. It can be used simply to provide a MIDI-capable synthesizer, with the ability to deliver instrument collections that ensure uniform content playback across multiple sound cards. It can be used to synchronously play back real-time-rendered (MIDI) and prerendered (wave) sounds in synch with each other. DirectMusic provides the ability to seamlessly transition from one piece of music to another at appropriate (and content-defined) boundaries. And it gives a rich set of tools to composers and sound designers for creating variation and dynamically adjustable content. Being aware of some of the rules and restrictions for how it can best be used, I encourage you to experiment with the concepts presented here. It's often easiest to gain familiarity with a few tools at a time rather than being overwhelmed by the options; this chapter has merely touched the surface of the features DirectMusic supports.