Introduction

Welcome to Fundamentals of Audio and Video Programming for Games .

This book is for software developers who want to add sound, music, or video to their programs. Not just any sound, music or video, but fancy stuff: special effects, 3-D effects, random and ambient effects, and so on. Although you ll learn how to do a straight rendition of a piece of music, sound byte, or video clip, this book focuses on how to create applications that impress. Typically, this application would be a game, although it could be any application requiring an impressive user interface. For example, video effects could be used in the title screens for many applications.

The primary philosophy of the book and its accompanying CD is to show and tell . In many ways, this publication is actually more of a CD with an accompanying book. In most chapters, we first direct you to a sample on the CD (written especially for this book), ask you to run it, and then examine the code required to achieve the effects. The samples exercise Microsoft technologies in ways that we hope will inspire you to add similar effects to your own programs. These samples are designed not just to show, but to show off, so, perhaps more accurately, this book s philosophy is to show off and tell.

This book is for programmers who are already adept at Visual C++. You should already have written a number of C++ applications or tools, and understand topics such as objects, inheritance, compiling, and debugging. Primarily, the main software tools we will be using are C++, with the development environment provided by Visual Studio.

If you are reading this book, you have probably already experimented with graphics and sound. The more experience that you have with Microsoft DirectX, the easier coding will be, although if you have no experience of the audio, music or video Software Development Kits (SDKs) within DirectX, you will be fine because we start from the beginning. The typical DirectX programmer understands Direct 3D much better than any of the other DirectX components . It has become a matter of pride , especially in the games development community, to be able to write your own 3-D matrix manipulation functions, mesh rendering code, and other hard- core 3-D graphics code. Games developers looking for a competitive edge might consider making awesome audio and imaginative use of video key components in their projects.

DirectX is based on Microsoft s Component Object Model (COM), which performs well, and is expected to be around for a long time. Since its mechanics are usually kept out of the way in most applications, we will do the same in this book.

The team writing this book is from the documentation group of Microsoft s Digital Media Division (DMD). Our division develops the company s audio/visual technology, both for programmers in the form of a whole range of SDKs, as well as the technology for audio and video components that go into Windows itself (such as Windows Media Player and Windows Movie Maker).

In this book, we will mainly be using the Microsoft DirectSound SDK and the Microsoft DirectShow SDK to explain the methods and functions that directly manipulate the output of sound or video, although we will also be referencing other Microsoft SDKs.

The book starts with an introduction to sound effects: first mono, then stereo, then 3-D, then 3-D with special and environmental effects. Following this, we examine the important topics of streaming sound and the use of property sets to get EAX environmental reverb effects to work. We ll also look at what s possible with 5.1/6.1/7.1 surround sound.

In many of today s games, there is a move away from using video clips (often referred to as cut scenes), partly because of the difficulty in obtaining good clips, and the lack of options in rendering them. This difficulty is partly due to the lack of connectivity between DirectShow (the video capture and rendering SDK) and the rest of DirectX. DirectX 9 has gone a long way towards remedying this issue, and the video chapters describe how to take advantage of this improvement. The video-based samples that accompany these chapters are fun and impressive, showing how to make a game out of rendering the video, and how to render video to a Direct3D surface (for example, onto a wall in a room).

The book also explains the techniques involved in capturing raw sound and video, such as cleaning noise out of samples, and going through the process of preparing them for repeated rendering. In this case, we will be using some tools developed outside of Microsoft.

So, close your door, install the book CD, go to the directory of executables, and pump up the volume.

Fun was never meant to be quiet.

Acknowledgements

One of the main differences between SDK documentation and a book is that since the SDK documentation comes out with the product, it usually lacks the benefit of experience using the material. A book should not simply be a rehash of the SDK docs, but a genuine attempt to add experience that only comes with time, consequently, all of the chapters and samples have been written especially for this book. This takes some resources, and thankfully, I have been able to leverage input from a number of people who have a lot of experience in particular areas of audio and video.

First mention goes to Mike Wasson, a programming writer with several years to his credit on the Microsoft DirectShow documentation team. He is responsible for the chapters on video, and the wonderful samples that go along with them. Next there is Bill Birney, a veteran of Hollywood as well as a technical writer for Microsoft, who contributed the two chapters on audio and video quality, along with a number of the music and sound tracks available on the book CD. Jim Travis, also a programming writer at Microsoft, contributed the chapter on audio special-effect parameters. Audio effects are Jim s specialty, coming from a background in theatrical presentations.

Cornel Moiceanu, a tools programmer at Microsoft, contributed some of his own compositions to the music tracks available on the CD.

Originating the content of a book is of course only part of the story. Cathy McDonald took on the awesome task of editing our prose , sagely advised by Terry Dorsey, and ably assisted by Steve Hug and Katherine Enos, while Greg Lovitt mastered the layout and production, assisted by Henry Bale.

At Microsoft Press, Juliana Aldous Atkinson helped the book proposal become reality, before handing over the publishing task to Robin Van Steenburgh. Thanks also to Lynn Finnel, an editor at MS Press, for her hard work and persistence with the project, Tess McMillan for building the CD, and to Joel Panchot, the MS Press artist, for so many inspired pieces of artwork.

There are also those who contributed their time in technically reviewing the content and code samples. Stephen Estrop and Dennis Evseev provided invaluable feedback on the VMR chapters, while Dugan Porter, Stephen Handley and Alan Ludwig provided insightful feedback on the audio chapters.

From the management side, User Education manager Tom Woolums approval of the budgets and resources for the project was, of course, essential, and I also appreciate the support of managing editor Karen Strudwick and production manager Keith Gabbert.

Finally I must thank the development teams here at Microsoft who created DirectSound and DirectShow, far too many people to mention by name .

For my contribution, I wrote seven of the eight chapters on audio, and the samples that go along with them.

Thanks for buying this book, and I hope you enjoy working with it as much as we did putting it together.

Dr. Peter Turcan
SDK Writing Manager
Digital Media Division
Microsoft Corporation