Audio Effects on MIDI Sequences


Audio Effects on Sampled Audio

There are three approaches for affecting sampled audio:


Precalculation

Using this approach, you create the audio effect at development time and play the resulting sound clip at execution time.


Byte array manipulation

Here, you store the sound in a byte array at runtime, permitting it to be modified using array-based operations.


Mixer controls

A mixer control, such as gain or panning, affects the sound signal passing through the mixer's audio line.

Precalculation

Manipulating audio inside Java can be time-consuming and complicated. If a sound effect is going to be used regularly (e.g., a fading scream, an echoing explosion), then it will probably be better to create it when the game is being developed and save the finished audio to a file for playing at runtime. This moves the overheads associated with sound effect generation out of the application. I've found WavePad useful for various editing, format conversion, and effects tasks (http://nch.com.au/wavepad/). Its supported effects include amplification, reverberation, echoing, noise reduction, fading, and sample rate conversion. It offers recording and CD track ripping. It's small (320 KB), free, and has a decent manual.

Many tools are out there: Do a search for "audio editor" at Google or visit a software site such as tucows (http://www.tucows.com/search).

Byte Array Manipulation

The most versatile manipulation approach in Java (but potentially tricky to get right) is to load the audio file as a byte array. Audio effects then become a matter of changing byte values, rearranging blocks of data, or perhaps adding new data. Once completed, the resulting array can be passed through a SourceDataLine into the mixer. The EchoSamplesPlayer.java application that follows shows how this can be done.

A variant of this approach is to employ streaming. Instead of reading in the entire file as a large byte array, the audio file can be incrementally read, changed, and sent to the mixer. However, this coding style is restricted to effects that only have to examine the sound fragment currently in memory. For example, amplification of the array's contents doesn't require a consideration of the other parts of the sound.


Making a sound clip echo

EchoSamplesPlayer.java completely loads a sound clip into a byte array via an AudioInputStream. Then an echoing effect is applied by creating a new byte array and adding five copies of the original sound to it; each copy is softer than the one before it. The resulting array is passed in small chunks to the SourceDataLine and to the mixer.

EchoSamplesPlayer is an extended version of the BufferedPlayer application described in Chapter 7. The main addition is a getSamples( ) method: This method applies the effect implemented in echoSamples( ). An isRequiredFormat( ) method exists for checking the input is suitable for modification. The program is stored in SoundExamps/SoundPlayer/.


To simplify the implementation, the echo effect is only applied to 8-bit PCM signed or unsigned audio. The choice of PCM means that the amplitude information is stored unchanged in the byte and isn't compressed as in the ULAW or ALAW formats. The 8-bit requirement means a single byte is used per sample, so I don't have to deal with big- or little-endian issues. PCM unsigned data stores values between 0 and 28 - 1 (255), and the signed range is -27 to 27 - 1 (-128 to 127). This becomes a concern when I cast a byte into a short prior to changing it.

The main( ) method in EchoSamplesPlayer is similar to the one in BufferedPlayer:

     public static void main(String[] args)     { if (args.length != 1) {         System.out.println("Usage: java EchoSamplesPlayer <clip>");         System.exit(0);       }       createInput("Sounds/" + args[0]);       if (!isRequiredFormat( )) {    // not in SamplesPlayer         System.out.println("Format unsuitable for echoing");         System.exit(0);       }       createOutput( );       int numBytes=(int)(stream.getFrameLength( )*format.getFrameSize( ));       System.out.println("Size in bytes: " + numBytes);       byte[] samples = getSamples(numBytes);       play(samples);       System.exit(0);   // necessary in J2SE 1.4.2 and earlier     }

The createInput( ) and createOutput( ) methods are unchanged from BufferedPlayer.


isRequiredFormat( ) tests the AudioFormat object that was created in createInput( ):

     private static boolean isRequiredFormat( )     // Only 8-bit PCM signed or unsigned audio can be echoed     {       if (((format.getEncoding( )==AudioFormat.Encoding.PCM_UNSIGNED) ||            (format.getEncoding( ) == AudioFormat.Encoding.PCM_SIGNED))&&            (format.getSampleSizeInBits( ) == 8))         return true;       else         return false;     }

AudioFormat has a selection of get( ) methods for examining different aspects of the audio data. For example, AudioFormat.getChannels( ) returns the number of channels used (1 for mono, 2 for stereo). The echoing effect doesn't need this information; all the frames, independent of the number of channels, will be amplified. Typically, channel information is required if an effect will differentiate between the stereo outputs, as when a sound is panned between speakers.

getSamples( ) adds the echoes after it has extracted the complete samples[] array from the AudioInputStream:

     private static byte[] getSamples(int numBytes)     {        // read the entire stream into samples[]        byte[] samples = new byte[numBytes];        DataInputStream dis = new DataInputStream(stream);        try {          dis.readFully(samples);        }        catch (IOException e)        { System.out.println( e.getMessage( ));          System.exit(0);        }        return echoSamples(samples, numBytes);     }

echoSamples( ) returns a modified byte array, which becomes the result of getSamples( ).

Different audio effects could replace the call to echoSamples( ) at this point in the code.


echoSamples( ) creates a new byte array, newSamples( ), big enough to hold the original sound and ECHO_NUMBER (4) copies. The volume of each one is reduced (decayed) (which is set to by DECAY (0.5) over its predecessor:

     private static byte[] echoSamples(byte[] samples, int numBytes)     {       int numTimes = ECHO_NUMBER + 1;       double currDecay = 1.0;       short sample, newSample;       byte[] newSamples = new byte[numBytes*numTimes];       for (int j=0; j < numTimes; j++) {         for (int i=0; i < numBytes; i++)  // copy the sound's bytes           newSamples[i + (numBytes*j)] = echoSample(samples[i], currDecay);         currDecay *= DECAY;       }       return newSamples;     }

The nested for loop makes the required copies one byte at a time. echoSample( ) utilizes a byte in the original data to create an "echoed" byte for newSamples[]. The amount of echoing is determined by the currDecay double, which shrinks for each successive copy of the original sound.

echoSample( ) does different tasks depending on if the input data are unsigned or signed PCM. In both cases, the supplied byte is translated into a short so it can be manipulated easily; then, the result is converted back to a byte:

     private static byte echoSample(byte sampleByte, double currDecay)     {       short sample, newSample;       if (format.getEncoding( ) == AudioFormat.Encoding.PCM_UNSIGNED) {         sample = (short)(sampleByte & 0xff);  // unsigned 8 bit -> short         newSample = (short)(sample * currDecay);         return (byte) newSample;       }       else if (format.getEncoding( )==AudioFormat.Encoding.PCM_SIGNED){         sample = (short)sampleByte;   // signed 8 bit -> short         newSample = (short)(sample * currDecay);         return (byte) newSample;       }       else         return sampleByte;    //no change; this branch should be unused     }

This byte-to-short conversion must be done carefully. An unsigned byte needs masking as it's converted since Java stores shorts in signed form. A short is two bytes long, so the masking ensures that the bits in the high-order byte are all set to 0s. Without the mask, the conversion would add in 1s when it saw a byte value above 127.

No masking is required for the signed byte to signed short conversion since the translation is correct by default.


Playing

play( ) is similar to the one in BufferedPlayer.java in Chapter 7. The difference is that the byte array must be passed through an input stream before it can be sent to the SourceDataLine:

     private static void play(byte[] samples)     {       // byte array --> stream       InputStream source = new ByteArrayInputStream(samples);       int numRead = 0;       byte[] buf = new byte[line.getBufferSize( )];       line.start( );       // read and play chunks of the audio       try {         while ((numRead = source.read(buf, 0, buf.length)) >= 0) {           int offset = 0;           while (offset < numRead)             offset += line.write(buf, offset, numRead-offset);         }       }       catch (IOException e)       {  System.out.println( e.getMessage( )); }       // wait until all data is played, then close the line       line.drain( );       line.stop( );       line.close( );     }  // end of play( )

Utilizing Mixer Controls

The mixer diagram in Figure 9-1 includes a grayish box labeled "Controls." Controls, such as gain and panning, affect the sound signal passing through an audio line. They can be accessed through Clip or SourceDataLine via a getControls( ) method that returns an array of available Control objects. Each object, suitably subclassed, allows its associated audio control to be manipulated.

Figure 9-1. Audio I/O to/from the mixer


The bad news is that the default mixer in J2SE 5.0 offers fewer controls than were present in J2SE 1.4.2 since controls tend to have an adverse effect on speed even when they're not being used. However, if a control is present, then it's much easier to apply than the byte array technique.

Adjusting a clip's volume and pan values

PlaceClip plays a clip, allowing its volume and pan settings to be adjusted via command-line parameters. It's called with the following format:

     java PlaceClip <clip file> [ <volume value> [<pan value>] ]

The volume and pan values are optional; if they are both left out, then the clip will play normally.

The volume setting should be between 0.0f (the quietest) and 1.0f (the loudest); -1.0f means that the volume is left unchanged. The pan value should be between -1.0f and 1.0f; -1.0f causes all the sound to be set to the left speaker, 1.0f focuses only on the right speaker, and values in between will send the sound to both speakers with varying weights, as in this example:

     java PlaceClip dog.wav 0.8f -1.0f

This will make the left speaker bark loudly. This mixing of volume and speaker placement is a rudimentary way of placing sounds at different locations in a game.

PlaceClip is an extended version of PlayClip, which was described in Chapter 7. The changes in PlaceClip are in the extra methods for reading the volume and pan settings from the command line and in the setVolume( ) and setPan( ) methods for adjusting the clip controls. The program is stored in SoundExamps/SoundPlayer/.PlaceClip's main( ) method is similar to the one in PlayClip.java:

     // globals     private float volume, pan;   // settings from the command line     public PlaceClip(String[] args)     {       df = new DecimalFormat("0.#");  // 1 dp     getSettings(args);   // get the volume and pan settings                            // from the command line       loadClip(SOUND_DIR + args[0]);       // clip control methods       showControls( );       setVolume(volume);       setPan(pan);       play( );       try {         Thread.sleep(600000);   // 10 mins in ms       }       catch(InterruptedException e)       { System.out.println("Sleep Interrupted"); }     }

loadClip( ) and play( ) are almost unchanged from PlayClip. (loadClip( ) uses a globally defined AudioFormat variable and has some extra println( )'s.) loadClip( ) includes a call to checkDuration( ), which issues a warning if the clip is one second or less in length. In that case, the clip won't be heard in J2SE 5.0 due to a Java Sound bug.

What controls are available?

showControls( ) displays all the controls available for the clip, which will vary depending on the clip's audio format and the mixer:

     private void showControls( )     { if (clip != null) {         Control cntls[] = clip.getControls( );         for(int i=0; i<cntls.length; i++)           System.out.println( i + ".  " + cntls[i].toString( ) );       }     }

getControls( ) returns information once the clip the class represents has been opened.


For the dog.wav example, executed using the J2SE 1.4.2 default mixer, showControls( )'s output is given in Example 9-1.

Example 9-1. showControls( )'s output
 0.  Master Gain with current calue: 0.0 dB (range: -80.0 - 13.9794) 1.  Mute Control with current value: Not Mute 2.  Pan with current value: 0.0 (range: -1.0 - 1.0) 3.  Sample Rate with current value: 22000.0 FPS (range: 0.0 - 48000.0) 

In this case, four controls are available: gain (volume), mute, panning, and sample rate.

Reverberation and balance controls may be available for some types of clips and mixers. In J2SE 5.0, panning, sample rate, and reverberation are no longer supported, and the balance control is only available for audio files using stereo.

In real-world audio gadgets, a pan control distributes mono input (input on a single channel) between stereo output lines (e.g., the lines going to the speakers). So, the same signal is sent to both output lines. A balance control does a similar job but for stereo input, sending two channels of input to two output lines.

In J2SE 1.4.2 and before, the pan and balance controls could be used with mono or stereo input, i.e., there was no distinction between them. Output lines were always opened in stereo mode. The default J2SE 1.4.2 mixer is the Java Sound Audio Engine.

The default mixer in J2SE 5.0 is the Direct Audio Device, with resulting changes to the controls. If the mixer receives mono input it will open a mono output line and not a stereo one. This means there's no pan control since there's no way to map mono to stereo. There is a balance control, but that's for mapping stereo input to stereo output.

In J2SE 5.0, the example will report that panning is unavailable since dog.wav was recorded in mono. The simplest solution is to convert it to stereo using WavePad (http://nch.com.au/wavepad/) or similar software. The balance controls will then be available, and setPan( ) can carry out panning by adjusting the balance.

Java audio controls

The various controls are represented by subclasses of the Control class: BooleanControl, FloatControl, EnumControl, and CompoundControl.

BooleanControl is used to adjust binary settings, such as mute on/off. FloatControl is employed for controls that range over floating point values, such as volume, panning, and balance. EnumControl permits a choice between several settings, as in reverberation. CompoundControl groups controls.

All these controls will function only if the clip is open.


As an example, here's a code fragment that turns mute on and off with a BooleanControl:

     BooleanControl muteControl =          (BooleanControl) clip.getControl( BooleanControl.Type.MUTE );     muteControl.setValue(true);     // mute on; sound is switched off          : // later on     muteControl.setValue(false);    // mute off; sound is audible again

Here's another that plays a clip at 1.5 times its normal speed via a FloatControl:

     FloatControl rateControl =          (FloatControl) clip.getControl( FloatControl.Type.SAMPLE_RATE );     rateControl.setValue( 1.5f * format.getSampleRate( ) );             // format is the AudioFormat object for the audio file

Setting the volume in PlaceClip

PlaceClip offers a volume parameter, ranging from 0.0f (off) to 1.0f (on). Additionally, no change to the volume is represented internally by the NO_VOL_CHANGE constant (the float -1.0f).

Unfortunately, the mixer's gain controls use the logarithmic decibel scale (related to the square of the distance from the sound source). Rather than grappling with a realistic mapping from my linear scale (0-1) to the decibel range, I use a linear equation to calculate the new gain:

     gain = ((range_max - range_min) * input_volume) + range_min

range_min and range_max are the minimum and maximum possible gain values; input_volume is the float obtained from the command line.

The drawback to this approach is that the logarithmic gain scale is being treated like a linear one. In practice, this means that the sound becomes inaudible when the supplied volume setting is 0.5f or less. On balance, this is a small price to pay for greatly simplified code.


setVolume( ) uses isControlSupported( ) to check for the volume control's presence before attempting to access/change its setting:

     private void setVolume(float volume)     {       if ((clip != null) && (volume != NO_VOL_CHANGE)) {         if (clip.isControlSupported(FloatControl.Type.MASTER_GAIN)) {           FloatControl gainControl = (FloatControl)                 clip.getControl(FloatControl.Type.MASTER_GAIN);           float range = gainControl.getMaximum( ) - gainControl.getMinimum( );           float gain = (range * volume) + gainControl.getMinimum( );           System.out.println("Volume: " + volume + "; New gain: " + gain);           gainControl.setValue(gain);         }         else           System.out.println("No Volume controls available");       }     }

FloatControl has several potentially useful methods, like shift( ), which is meant to change the control value gradually over a specified time period and returns without waiting for the shift to finish. Unfortunately, this particular method has never been fully implemented and currently modifies the control value in one step without any incremental changes in between.


Panning between the speakers in PlaceClip

setPan( ) is supplied with a pan value between -1.0f and 1.0fwhich will position the output somewhere between the left and right speakersor with NO_PAN_CHANGE (0.0f). The method pans first, looks for the balance control if panning is unavailable, and finally gives up if both are unsupported:

     private void setPan(float pan)     {       if ((clip == null) || (pan == NO_PAN_CHANGE))         return;   // do nothing       if (clip.isControlSupported(FloatControl.Type.PAN)) {         FloatControl panControl =            (FloatControl) clip.getControl(FloatControl.Type.PAN);         panControl.setValue(pan);       }       else if (clip.isControlSupported(FloatControl.Type.BALANCE)) {         FloatControl balControl =            (FloatControl) clip.getControl(FloatControl.Type.BALANCE);         balControl.setValue(pan);       }       else {         System.out.println("No Pan or Balance controls available");         if (format.getChannels( ) == 1)   // mono input           System.out.println("Your audio file is mono;                                   try converting it to stereo");       }     }



Killer Game Programming in Java
Killer Game Programming in Java
ISBN: 0596007302
EAN: 2147483647
Year: 2006
Pages: 340

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net