Chapter 10. Audio Synthesis

Audio Effects on MIDI Sequences

There are four ways of applying audio effects to MIDI sequences:

Precalculation

Similar to what you've seen, this involves creating the audio effect at development time and playing the resulting MIDI sequence at execution time.

Sequence manipulation

Here, the MIDI sequence data structure can be manipulated at runtime using a range of methods from MIDI-related classes.

MIDI channel controllers

In this approach, a channel plays a particular instrument and has multiple controllers associated with it, which manage such things as volume and panning.

Sequencer methods

The Sequencer class offers several methods for controlling a sequence, including changing the tempo (speed) of the playback and muting or soloing individual tracks in the sequence.

Precalculation

As with sampled audio, using Java at execution time to modify a sequence can be time-consuming and difficult to implement. Several tools allow you to create or edit MIDI sequences, though you do need an understanding of music and MIDI to use them. Here are some of packages I've tinkered with:

The free version of Anvil Studio (http://www.anvilstudio.com/)

Supports the capture, editing, and direct composing of MIDI. It handles WAV files.

BRELS MIDI Editor (http://www.tucows.com/search)

A free, small MIDI editor. It's easiest to obtain from a software site, such as tucows.

Midi Maker (http://www.necrocosm.com/midimaker/)

Emulates a standard keyboard synthesizer. Available for a free 14-day trial.

Sequence Manipulation

Figure 9-2 shows the internals of a sequence.

Figure 9-2. The internals of a MIDI sequence

The regularity of the data structure means that it can be easy to modify at runtime, but you're going to need to understand the MIDI specification.

Doubling the sequence volume

Here is the basic code for playing a sequence:

     Sequence seq = MidiSystem.getSequence(getClass( ).getResource(fnm));     // change the sequence: double its volume in this case     doubleVolumeSeq(seq);     sequencer.setSequence(seq);  // load changed sequence     sequencer.start( );           // start playing it

This snippet omits the try/catch blocks you need in an actual code block. Look at PlayMidi.java in Chapter 7 for a complete version.

The sequence is modified after being loaded with getSequence( ) and before being assigned to the sequencer with setSequence( ).

Volume doubling is applied to every track in the sequence:

     private void doubleVolumeSeq(Sequence seq)     { Track tracks[] = seq.getTracks( );     // get all the tracks       for(int i=0; i < tracks.length; i++)  // iterate through them         doubleVolume(tracks[i], tracks[i].size( ));     }

doubleVolume( ) examines every MidiEvent in the supplied track, extracting its component tick and MIDI message. If the message is a NOTE_ON, then its volume will double (up to a maximum of 127):

     private void doubleVolume(Track track, int size)     {       MidiEvent event;       MidiMessage message;       ShortMessage sMessage, newShort;       for (int i=0; i < size; i++) {         event = track.get(i);          // get the event         message = event.getMessage( );  // get its MIDI message         long tick = event.getTick( );   // get its tick         if (message instanceof ShortMessage) {           sMessage = (ShortMessage) message;           // check if the message is a NOTE_ON           if (sMessage.getCommand( ) == ShortMessage.NOTE_ON) {              int doubleVol = sMessage.getData2( ) * 2;              int newVol = (doubleVol > 127) ? 127 : doubleVol;              newShort = new ShortMessage( );              try {                newShort.setMessage(ShortMessage.NOTE_ON,                                     sMessage.getChannel( ),                                     sMessage.getData1( ), newVol);                track.remove(event);                track.add( new MidiEvent(newShort,tick) );              }              catch ( InvalidMidiDataException e)              {  System.out.println("Invalid data");  }           }         }       }     }  // end of doubleVolume( )

Each MIDI message is composed from three bytes: a command name and two data bytes. ShortMessage.getCommand( ) is employed to check the name. If the command name is NOTE_ON, then the first byte will be the note number, and the second its velocity (similar to a volume level).

MIDI messages are encoded using three subclasses of MidiMessage: ShortMessage, SysexMessage, and MetaMessage. Each class lists constants representing various commands. The NOTE_ON and NOTE_OFF messages are ShortMessage objects, used to start and terminating note playing.

The volume is obtained with a call to ShortMessage.getData2( ) and then doubled with a ceiling of 127 since the number must fit back into a single byte. A new ShortMessage object is constructed and filled with relevant details (command name, destination channel ID, note number, new volume):

     newShort.setMessage(ShortMessage.NOTE_ON,                    sMessage.getChannel( ), sMessage.getData1( ), newVol);

The old MIDI event (containing the original message) must be replaced by an event holding the new message: a two-step process involving track.remove( ) and track.add( ). The new event is built from the new message and the old tick value:

     track.add( new MidiEvent(newShort,tick) );

The tick specifies where the event will be placed in the track.

MIDI Channel Controllers

Figure 9-3 shows the presence of 16 MIDI channels inside the synthesizer; each one acts as a "musician," playing a particular instrument. As the stream of MIDI messages arrive (individually or as part of a sequence), each message is routed to a channel based on its channel setting.

Each channel has a set of controllers associated with it. The set depends on the particular synthesizer; controllers defined in the General MIDI specification should be present, but there may be others. For example, controllers offering the Roland GS enhancements are found on many devices. General MIDI controllers include controls for volume level, stereo balancing, and panning. Popular Roland GS enhancements include reverberation and chorus effects. Each controller is identified by a unique ID, between 0 and 127.

A list of channel controllers, complete with a short description of each one, can be found at http://improv.sapp.org/doc/class/MidiOutput/controllers/. Another site with similar information is http://www.musicmarkup.info/midi/control.html.

Figure 9-3. A MIDI sequencer and synthesizer

The FadeMidi and PanMidi examples illustrate how to use channel controllers to affect the playback of an existing sequence. They both reuse several methods from PlayMidi.java, shown in Chapter 7.

Making a sequence fade away

FadeMidi.java (located in SoundExamps/SoundPlayer/) plays a sequence, gradually reducing its volume level to 0 by the end of the clip. The volume settings for all 16 channels are manipulated by accessing each channel's main volume controller (the ID for that controller is the number 7).

There's a fine-grain volume controller (ID number 39) that's intended to allow smaller change graduations, but many synthesizers don't support it.

The incremental volume reduction is managed by a VolChanger thread, which repeatedly lowers the volume reduction until the sequence has been played to its end.

Figure 9-4 gives the class diagrams for FadeMidi and VolChanger, showing only the public methods.

The main( ) method initializes FadeMidi and starts VolChanger:

     public static void main(String[] args)     { if (args.length != 1) {         System.out.println("Usage: java FadeMidi <midi file>");         System.exit(0);       }

Figure 9-4. Class diagrams for FadeMidi and VolChanger

       // set up the player and the volume changer       FadeMidi player = new FadeMidi(args[0]);       VolChanger vc = new VolChanger(player);       player.startVolChanger(vc);  // start volume manipulation     }

VolChanger is passed a reference to FadeMidi so it can affect the synthesizer's volume settings.

startVolChanger( ) starts the VolChanger thread running and supplies the sequence duration in milliseconds. The thread needs it to calculate how often to change the volume:

     public void startVolChanger(VolChanger vc)     {  vc.startChanging( (int)(seq.getMicrosecondLength( )/1000) );  }

The FadeMidi constructor looks similar to the one in PlayMidi:

     public FadeMidi(String fnm)     {      df = new DecimalFormat("0.#");  // 1 dp       filename = SOUND_DIR + fnm;       initSequencer( );       loadMidi(filename);       play( );       /* No need for sleeping to keep the object alive, since          the VolChanger thread refers to it. */     }

initSequencer( ) and loadMidi( ) are identical to the methods of the same name in PlayClip, and play( ) is slightly different. The most significant change is the absence of a call to sleep( ), which keeps PlayMidi alive until its sequence has finished. Sleeping is unnecessary in FadeMidi because the object is referred to by the VolChanger tHRead, which keeps calling its setVolume( ) method.

play( ) initializes a global array of MIDI channels:

     private static final int VOLUME_CONTROLLER = 7;     // global holding the synthesizer's channels     private MidiChannel[] channels;     private void play( )     { if ((sequencer != null) && (seq != null)) {         try {           sequencer.setSequence(seq);  // load MIDI into sequencer           sequencer.start( );   // play it           channels = synthesizer.getChannels( );           // showChannelVolumes( );         }         catch (InvalidMidiDataException e) {           System.out.println("Invalid midi file: " + filename);           System.exit(0);         }       }     }     private void showChannelVolumes( )     // show the volume levels for all the synthesizer channels     {       System.out.println("Syntheziser Channels: " + channels.length);       System.out.print("Volumes: {");       for (int i=0; i < channels.length; i++)         System.out.print( channels[i].getController(VOLUME_CONTROLLER) + " ");       System.out.println("}");     }

The references to the channels shouldn't be obtained until the sequence is playing (i.e., after calling sequencer.start( )) or their controllers will not respond to changes. This seems to be a bug in the Java Sound implementation.

Channels in the array are accessed using the indices 0 to 15 though the MIDI specification numbers them 1 to 16. For instance, the special percussion channel is MIDI number 10, but it is represented by channels[9] in Java.

In showChannelVolumes( ), MidiChannel.getController( ) obtains the current value of the specified controller. Supplying it with the ID for the volume controller (7) will cause it to return the current volume setting. A controller stores the data in a single byte, so the returned value will be in the range 0 to 127.

Getting and setting the volume

FadeMidi contains two public methods for getting and setting the volume, both used by VolChanger:

     public int getMaxVolume( )     // return the max level for all the volume controllers     { int maxVol = 0;       int channelVol;       for (int i=0; i < channels.length; i++) {         channelVol = channels[i].getController(VOLUME_CONTROLLER);         if (maxVol < channelVol)           maxVol = channelVol;       }       return maxVol;      }     public void setVolume(int vol)     // set all the controller's volume levels to vol     { for (int i=0; i < channels.length; i++)         channels[i].controlChange(VOLUME_CONTROLLER, vol);     }

getMaxVolume( ) returns a single volume, rather than all 16; this keeps the code simple. setVolume( ) shows how MidiChannel.controlChange( ) is used to change a specified controller's value. The data should be an integer between 0 and 127.

Changing the volume

VolChanger gets started when its startChanging( ) method is called. At this point, the sequence will be playing, and the MIDI channel controllers are available for manipulation:

     // globals     // the amount of time between changes to the volume, in ms     private static int PERIOD = 500;     private FadeMidi player;     private int numChanges = 0;     public void startChanging(int duration)     /* FadeMidi calls this method, supplying the duration of        its sequence in ms. */     {       // calculate how many times the volume should be adjusted       numChanges = (int) duration/PERIOD;       start( );     } // end of startChanging( )

VolChanger adjusts the volume every PERIOD (500 ms), but how many times? The duration of the sequence is passed in as an argument to startChanging( ) and is used to calculate the number of volume changes.

run( ) implements a volume reduction/sleep cycle:

     public void run( )     {       /* calculate stepVolume, the amount to decrease the volume          each time that the volume is changed. */       int volume = player.getMaxVolume( );       int stepVolume = (int) volume / numChanges;       if (stepVolume == 0)         stepVolume = 1;       System.out.println("Max Volume: " + volume + ", step: " + stepVolume);       int counter = 0;       System.out.print("Fading");       while(counter < numChanges){         try {           volume -= stepVolume;    // reduce the required volume level           if ((volume >= 0) && (player != null))             player.setVolume(volume);    // change the volume           Thread.sleep(PERIOD);          // delay a while         }         catch(InterruptedException e) {}         System.out.print(".");         counter++;       }       System.out.println( );     }

The MIDI volume bug

FadeMid.java doesn't work with J2SE 5.0 due to a bug associated with the volume adjustment of a sequencer. The offending line is in initSequencer( ):

     sequencer = MidiSystem.getSequencer( );

The sequencer is retrieved, but subsequent volume changes have no effect. The solution is to explicitly request the sequencer by finding it in on the list of available MIDI devices for the machine. This is packaged inside obtainSequencer( ):

     private Sequencer obtainSequencer( )     {       MidiDevice.Info[] mdi = MidiSystem.getMidiDeviceInfo( );       int seqPosn = -1;       for(int i=0; i < mdi.length; i++) {         System.out.println(mdi[i].getName( ));         if (mdi[i].getName( ).indexOf("Sequencer") != -1) {           seqPosn = i;    // found the Sequencer           System.out.println("  Found Sequencer");         }       }       try {         if (seqPosn != -1)           return (Sequencer) MidiSystem.getMidiDevice( mdi[seqPosn] );         else           return null;       }       catch(MidiUnavailableException e)       { return null; }     }  // end of obtainSequencer( )

The position of the sequencer in the MIDI device information array, mdi[], will vary depending on the audio devices attached to a given machine and the J2SE version, so some searching is required. The list printed on a test machine running J2SE 5.0 is shown in Example 9-2.

Example 9-2. MIDI device information in J2SE 5.0

 Roland MPU-401 MIDI Mapper Microsoft GS Wavetable SW Synth Roland MPU-401 Real Time Sequencer   Found Sequencer Java Sound Synthesizer 

The list generated on a different machine, using J2SE 1.4.2, is shown in Example 9-3.

Example 9-3. MIDI device information in J2SE 1.4.2

 Java Sound Synthesizer Java Sound Sequencer   Found Sequencer MIDI Mapper Microsoft GS Wavetable SW Synth 

The sequencer is obtained in initSequencer( ) by calling obtainSequencer( ):

     sequencer = obtainSequencer( );

The problem, which has been reported by several users in the Java Sound forums (e.g., at http://archives.java.sun.com/cgi-bin/wa?A0=javasound-interest), only seems to occur when the volume needs to be changed. For example, this extra work isn't required in PanMidi (the next example): the sequencer it obtains with MidiSystem.getSequencer( ) does respond to panning changes.

I'm at a loss as to why my workaround works since the sequencer object returned by MidiSystem.getSequencer( ) and the one obtained with my obtainSequencer( ) method appear to be the same.

Panning the sequence

PanMidi repeatedly switches its sequence from the left to the right speaker and back again. A PanChanger thread switches the pan settings in all the channel controllers at periodic intervals during the playing of the sequence.

PanMidi and PanChanger can be found in SoundExamps/SoundPlayer/.

The class diagrams for PanMidi and PanChanger are given in Figure 9-5.

Figure 9-5. Class diagrams for PanMidi and PanChanger

The main( ) method initializes the player and the thread, and then it calls PanMidi's startPanChanger( ) to start the thread running. startPanChanger( ) passes the duration of the sequence to the thread, so it can calculate the number of changes it will make.

The PanMidi pan methods used by PanChanger are getMaxPan( ) and setPan( ):

     // global constants     // private static final int BALANCE_CONTROLLER = 8; //not working?     private static final int PAN_CONTROLLER = 10;     public int getMaxPan( )     // return the max value for all the pan controllers     { int maxPan = 0;       int channelPan;       for (int i=0; i < channels.length; i++) {         channelPan = channels[i].getController(PAN_CONTROLLER);         if (maxPan < channelPan)           maxPan = channelPan;       }       return maxPan;     }     public void setPan(int panVal)     // set all the controller's pan levels to panVal     { for (int i=0; i < channels.length; i++)         channels[i].controlChange(PAN_CONTROLLER, panVal);     }

The only real difference in PanMidi from FadeMidi is the use of the PAN_CONTROLLER controller number.

The balance controller should work in this situation, but it didn't on my test machines. This bug has been reported by several people, so we may see a fix soon.

Changing the pan value

Unlike VolChanger, PanChanger carries out a cyclic series of changes to the pan value. However, the core of run( ) is still a loop repeatedly calling setPan( ) and sleeping for an interval.

The series of pan values that make up a single cycle are defined in a panVals[] array:

     // time to move left to right and back again     private static int CYCLE_PERIOD = 4000;  // in ms     // pan values used in a single cycle     // (make the array's length integer divisible into CYCLE_PERIOD)     private int[] panVals = {0, 127};     // or try     // private int[] panVals = {0, 16, 32, 48, 64, 80, 96, 112, 127,     //                          112, 96, 80, 64, 48, 32, 16};

The run( ) method cycles through the panVals[] array until it has executed for a time equal to the sequence's duration:

     public void run( )     { /* Get the original pan setting, just for information. It           is not used any further. */        int pan = player.getMaxPan( );        System.out.println("Max Pan: " + pan);        int panValsIdx = 0;        int timeCount = 0;        int delayPeriod = (int) (CYCLE_PERIOD / panVals.length);        System.out.print("Panning");        while(timeCount < duration){          try {            if (player != null)              player.setPan( panVals[panValsIdx] );            Thread.sleep(delayPeriod);    // delay          }          catch(InterruptedException e) {}          System.out.print(".");          panValsIdx = (panValsIdx+1) % panVals.length;                                      // cycle through the array          timeCount += delayPeriod;        }        System.out.println( );     }

Sequencer Methods

The Sequencer has methods that can change the tempo (speed) of playback. The easiest to use is probably setTempoFactor( ), which scales the existing tempo by the supplied float:

     sequencer.setTempoFactor(2.0f);   // double the tempo

Tempo adjustments only work if the sequence's event ticks are defined in the PPQ (ticks per beat) format since tempo affects the number of beats per minute. Sequencer.getTempoFactor( ) can be employed after calling Sequencer.setTempoFactor( ) to check whether the requested change has occurred. The Sequence class offers getdivisionType( ), which returns a float representing the sequence's division type. Sequence.PPQ for PPQ, or one of the many Society of Motion Picture and Television Engineers (SMPTE) types, use ticks per frame. This information can be used to determine if setTempoFactor( ) would work on the sequence.

Sequencer has two methods that act upon the sequence's tracks: setTrackMute( ), and setTrackSolo( ). Here's a fragment of code that sets and tests the mute value:

     sequencer.setTrackMute(4, true);     boolean muted = sequencer.getTrackMute(4);     if (!muted)       // muting failed