^[2] Stefan Saroiu et al., "An Analysis of Internet Content Delivery Systems," in Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI) (Berkeley, CA: USENIX Association, December 2002).

Multimedia applications allow you to create engaging interactive experiences. Remember, the tricky part is not creating your content but delivering your content efficiently .

Multimedia Basics

This section introduces you to some fundamental multimedia concepts. Learning these concepts is a necessary starting point from which to build. You can think of multimedia as a collection of many different forms of data types (file formats) that are all used together to give the audience a richer, potentially interactive experience.

Web Multimedia Datatypes

Before you can understand the process and procedures behind efficiently delivering your multimedia, you first need to understand the strengths and weaknesses of the various multimedia file types you will encounter on the web. Let's first take a look at audio files.

Audio Data Types

Effective audio can help deliver your message with impact. Audio is as important as visuals in your presentation. Audio can convey a sense of emotion much better than a photo. You should learn as much about audio and audio compression as possible so that you can make an accurate assessment of the audio needs of your project.

Audio can be contained in and delivered with different types of files. For example, you might digitize your audio and save the file in Sound Designer II format on your hard drive, but when it comes time to deliver that same audio file, you'll want to compress the data into something like the MP3 format. Compression saves storage space and speeds delivery.

Most of the audio files you'll find on the Internet are in one of the formats discussed in the next sections.

MP3

MP3 is a compressed audio format that sounds excellent . MP3 is designed for delivering music but can be used for voice as well. MP3 uses perceptual encoding, where the algorithm "listens" to the sound and removes frequencies you cannot hear. The MP3 format lets you choose a target bit-rate setting to encode to. A setting of 128K sounds just like a normal audio CD but is one tenth the size . A setting of 40K to 60K is perfect for a simple voiceover track. MP3 can also be used for archiving CDs or other audio samples. Because audio is sampled at 16-bit 44.1K takes up 44.1K of disk space for every second in length (88.2K for stereo files), a normal CD track takes about 10.5MB of hard disk space. MP3s, on the other hand, can save files in an archive format, and the same file is only about 2MB but sounds exactly the same. MP3 compression is lossy (see http://www.mpeg.org for more information).

Qdesign Music

Qdesign Music is a compressed audio format that comes in standard and pro versions. The psycho -acoustic parametric coding algorithm is designed for low data rate applications. This codec allows you to save compressed audio in different sampling rates (11, 22, or 44kHz), at different bit depths (8 or 16-bit), and in mono or stereo. When used with Cleaner, this codec allows you to adjust the track volume, change the dynamic range, add reverb, and apply other effects normally seen only on external hardware devices such as compressors.

The Qdesign Music codec also comes in a professional versionfor more money, of course. It allows control over targeting a specific bit-rate, has a setting so you can tell it to lean more toward "quality" or "speed" (size), and features two pass variable bit-rate encoding for more precise data-rate targeting. See http://www.qdesign.com for more information.

Qualcomm PureVoice

Qualcomm PureVoice is a codec designed for encoding voice signals. It has some amazing compression capabilities that can be helpful when you are simply using voice as your source material. The codec allows you to change the sample rate, the bit rate, the number of channels, and the setting that determines whether to use the "full rate" or "half rate" settings. The full rate settings can compress your audio in a 9:1 ratio, and the half rate setting can achieve a 19:1 compression ratio. This is pretty heavy compression, but nonetheless the audio is more than acceptableit sounds like a long-distance phone call.

IMA 4:1

IMA 4:1 is a compressed delivery format from the Interactive Multimedia Association. As its name implies, IMA compresses audio by 75 percent with high-quality sound. Eight- and 16-bit IMA has the same quality and file size. The limitation of IMA is that you cannot IMA-compress an AIFF file for playback on a Windows system. This codec was originally designed for audio that needs to be played off a CD-ROM in real time, and it does the job well. It is one of the oldest codecs, but it is still used today for CD-ROM work and backward compatibility.

MPEG-4 Audio

MP4 is the new audio standard replacing MP3. MP4 supports bit rate, sample rate, depth, channel, hinted streaming tracks, and more. What makes this the codec of choice is that its compression algorithms are better. MP4 uses the Advanced Audio Coding (AAC) method to produce smaller files and higher quality. In fact, AAC-compressed sounds deliver higher quality than uncompressed CD audio. A 20-second 16-bit, 22.5kHz audio file of 900K compresses to 128K in MP3 at 40kHz and sounds muddy. The same file compressed with MP4 is 108K and sounds nearly identical to the original. The trouble is that because this codec is so new, most people don't have the means to decode it (another thing to worry about!). See http://www.apple.com/mpeg4/ and http://mpeg.telecomitalialab.com/ for more information.

WAVE

This format is used extensively in Microsoft Windows to store and deliver audio, and is also used in wavetable synthesis, such as E-mu's SoundFont. Conversion tools can convert .wav files to other operating systems. See http://www.microsoft.com/ for more information.

QuickTime Audio

QuickTime is both a file container and delivery format; that is, you can use QuickTime to embed MP3, AIFF, MP4, and video in your web pages and CD-ROMs. The QuickTime format is cross-platform for people with Macs and PCs. You can use Quick Time files to edit your audio and then compress them for distribution, making this a very versatile format.

The next section explains video formats for the web and their strengths and weaknesses.

Video Data Types

When video is digitized, it becomes "data" that needs to be contained inside some type of file format, such as QuickTime. Once video is stored in an electronic file, it can be edited (rearranged in a new order other than that in which it was shot), and then the video can be compressed and delivered to your target device, such as the web or a CD-ROM. Here are the different types of video formats you will encounter:

• QuickTime (QT) QT is the king of video formats. Over 15 years old, QT is cross-platform, supports both progressive downloading and real-time streaming, and can get through firewalls. The professional version is for media creators , offering additional file translation and sampling capabilities that are well worth the cost. Nearly every video application supports QT. It can handle VR 3D fly-throughs, and supports scripting for interactive presentations. QuickTime requires a plug-in, which most modern browsers bundle, or you can download it from http://www.apple.com/quicktime.

• RealMedia This is RealNetworks' cross-platform audio and video streaming delivery technology. It allows different target bandwidths and desktop or web playback. Unlike QuickTime, RealVideo files are only stored compressed, so there's no way to revert to raw video or audio for editing.

• Windows Media Player Microsoft's answer to QuickTime. The Windows Media Player will play back either compressed or streaming WMP files in real time to all target bandwidths, such as a DSL or a 56Kbps modem. WMP can deliver compressed, downloadable files or stream a live signal. See http://www.microsoft.com for more information.

Because compression for QuickTime, RealPlayer, and Windows Media Player are all very similar, I explain the concepts further in the "Codecs" section, later in this chapter.

Animation

Animation on the Internet comes in two primary forms: the traditional 2D animation and the more recent 3D animation. 2D animation is the equivalent of cartoon-style frame-by-frame animation. Modern computers have made this process much easier by way of keyframes. Now artists need only to draw keyframes of the animation and let the computer interpret the motion between keyframes, a process with the delightful name of tweening. Applications such as Flash from Macromedia work this way.

There are currently three main data types for web-based animation:

• Flash The most popular animation format on the web. Vector-based format files are generally small and bandwidth friendly. Flash files are compressed into SWF file format and delivered with Macromedia Shockwave technology. Flash is nearly ubiquitous on the web; 95 percent of all browsers are able to display Flash animations. For all its benefits, Flash has a clumsy professional development environment. Visit http://www.macromedia.com for more information.

• Shockwave and Shockwave 3D Director allows the creation of vector or bitmapped-based content compressed into DCR (Director Compressed Resource) files that are played back on the web using Shockwave. Macromedia and Intel recently introduced Shockwave 3D, which also supports 3D data from applications like 3ds max or Maya for 3D experiences. Your audience can fly through online stores and examine items with HTTP links. However, creating 3D worlds with textures, lighting, and Lingo code takes a lot of skill. DCR files and the plug-in are much larger than their Flash counterparts. For more information, visit the Macromedia web site at http://www.macromedia.com.

• Cult3d Cult3D tries to blend the simplicity of Flash with the capabilities of Director for web 3D. Cycore has a large installed base of Cult3D player downloads. Cult3D is designed with 3D e-commerce in mind, with XML database interaction and shopping cart functions built-in. For more information, visit their web site at http://www.cult3d.com.

Limitations to Multimedia

You didn't think there weren't going to be any limitations to multimedia, did you? Multimedia work requires cutting-edge technology. When you are developing multi media, you are working in the latest video, audio, and image-editing applications, and there are always limitations and issues you face when you are trying to stay current with technology.

In short, when you're a multimedia developer, you always have to have the latest technology to impress your clients . That means constant upgrading of software, and thus operating systems. Unlike accountants , who can get away with using one version of Excel for years, multimedia developers are constantly upgrading their hardware and software.

Once you get used to continuously upgrading, your next limitation will be the raw speed of your connection to the Internet. You generally need the fastest connection you can get because you are going to not only be transferring large files, but you will also need to test regularly video and audio playback on the web.

Processing Speed

Get the fastest processor you can afford because compression often is time-consuming and editing video and/or audio is CPU- intensive . However, the real limitation here is going to be your customer's CPU speed. You will need to make a CPU cut-off point for some of your projects just to protect yourself in some cases. For full-blown multimedia projects with audio, video, text, and photos, you'll have to test your finished piece on many different computers to find the ones on which it plays well, those on which it plays adequately, and the ones on which it just doesn't work at all. Then you will either have to go back to your project and more tightly compress the audio and video or print a CPU requirement on your web site or CD-ROM package.

Bandwidth

The speed of your connection will always feel like a limiting factor, regardless of how fast it is. When I started using the Internet, 1200- and 2400-baud modems were the fastest modems you could use, but they felt slow to me. A few years later, everyone had 56,000 baud, but that speed felt faster only temporarily because larger sites made even that speed feel slow. Today I'm sitting at the opposite end of a 6Mb DSL (6,000,000 bits/second) connection at home, and it still feels slow to me. Remember:

• It's not your connection speed but your customers' speed that's important.

• Regardless of your speed, you need to learn to compress your files as much as possible.

• Considering that most people don't upgrade, you also need to make versions of your files backward compatible.

Codecs

When you have files on your hard drive and you plan on delivering them either over the Internet, via CD-ROM, or even by email, they need to be compressed. Compression actually has two parts to it: compression and decompression . This is where the term codec comes from: co mpression and dec ompression. Compression is what you do to a video or sound file to make it smaller, and decompression is how it plays the file back. You have control over both aspects of the process using different codecs.

You can actually purchase codecs for video or audio that are professional versions designed for multimedia professionals. For example, the Sorenson Pro Developer Codec is used by Industrial Light & Magic (ILM) to compress all the Star Wars trailers you see on the Internet (http://www.apple.com/ trailers /).

Why Codecs Are Necessary

Without the use of compression, video and audio could not effectively be distributed over the Internet. Consider the Star Wars trailer, for example. The high-speed broadband version of that video clip on the Internet is about 25MB in size. That same clip uncompressed could easily be 250 to 500MB in size; this would take up the better part of an entire CD-ROM.

In addition to delivery, codecs make video playback possible. Video is a CPU-intensive medium simply because there is so much data involved. One second of NTSC video captured from the miniDV format is about 3.5MB of disk space. This means that every 60 seconds of video is 210MB of data, and every five minutes is a little over 1GB of data (or 9,000 frames of images). Now, even the latest computers could have trouble trying to process 1GB of data so fast that you don't see any jerky playback or pauses. This is why codecs are needed. You can easily reduce the size of your video 10:1, 20:1, 50:1, or even 100:1 using modern codecs. In order for your computer to play five minutes of video (210MB prior to compression), all it has to handle is about 2MB of data. Because 2MB is a smaller amount of data, older and slower computers can now play back video and see your creations.

How Codecs Work

Compression is an extremely complex operation that in some cases takes days for your computer to calculate. But remember that the compression you do prior to delivery creates smaller finished video, which plays smoother on more machines.

Compressing video works by removing redundant data in two ways: on a frame-per-frame (spatial) basis, and on an over-time (temporal) basis.

Spatial compression is where the compression application looks at every frame of your video and then groups every pixel in that one frame based on how close pixel colors are to one another. The algorithm then sets similar RGB color values to the same value. What this allows is instead of assigning every pixel three values between 0 and 255, you have big groups of pixels all with the same value, resulting in at least a 2:1 savings in compression. This is similar to how static image compression works, across a set of "images."

In addition to pixel compression, codecs can do something called temporal compression. Temporal is an over-time compression technique over which you really don't have too much control. The Sorenson codec, for example, creates natural keyframes when the video changes dramatically from one scene to the next. These keyframes are compressed more lightly than your entire video and are used by the codec as references on how to draw the frames around it.

Temporal compression smoothes pixels over time and tries to remove video noise. Temporal settings also can help you remove one-frame artifacts.

Streaming Media

Streaming content is multimedia content that is played on the user 's computer while it is being downloaded from a server. Streaming media can protect content by preventing users from saving data directly on their hard drives . It also has the advantage of being able to stream very long pieces of data. Streams of 90 minutes or more are possible, provided your users have a stable connection. The stream quality, or bits per second, can depend on the user's connection speed.

QuickTime

Apple's QuickTime is the best solution for video delivery on the Macintosh and some would say on the PC as well. The streaming aspect of QuickTime is no less than awesome with its feature set, performance, and ease of use for both the developer and the client.

Apple provides two types of streaming technologies:

• Real-Time, for streaming video over the Internet during a live event.

• Progressive, for delivering video over the Internet with prerecorded material.

With real-time, you can hook up your video camera to your computer, which is running a copy of QuickTime's Broadcaster. This allows you to send live video over a streaming server out to thousands of viewers (see Figure 13.1). Broadcaster takes advantage of all of QuickTime's codecs such as MP3 streaming, MP4 streaming, and video streaming.

Figure 13.1. The QuickTime streaming server is powerful and simple to use.

NOTE: With the new ISO standards, MPEG4 is now starting to show up in cellphones and other PDA devices. The QuickTime streaming server will enable you to communicate with all of these devices.

In terms of compression for streaming QuickTime content, there is not much to do other than normal compression. Video compression is both spatial and temporal. With streaming content, you pick a target bandwidth, video size, and frame rate; enable "hinted tracks;" and let the file compress. Once it's done, you upload the file to your server and make a link to it on your web page. When a user clicks on this link, the video is requested , sent to the client, buffered for a moment, and then playback starts. Remember:

• Dial-up modems are almost useless for video.

• Your client should be able to handle a 256Kbps download or it's not worth it.

• Use QuickTime progressive for content below 256Kbps downloads.

QuickTime progressive downloads begin playing as soon as enough content is buffered to allow smooth playback for the duration of the clip.

For More Information If you think you'll use QuickTime, go to http://www.apple.com/quicktime/tools_tips/tutorials/activex.html for instructions on making your QuickTime files viewable on all platforms and browsers. If you are using version 5.5 SP2 of Internet Explorer for Windows, you must view QuickTime as an ActiveX control because Microsoft has discontinued support for Netscape style plug-ins.

I've found that people are much more accepting of temporal compression (over-time) of the video than spatial (frame-per-frame) compression. By cutting the frame rate from l5fps to 10fps or even 8fps, each frame has 50 percent more data per frame for the same file size, which will increase the quality of the picture. The minimum size should be 320x240. If your clients have a fast Internet connection, 400x300 is better. Anything smaller than 320x240 is just too small. To maintain good quality, always up the data rate in proportion to the image size. Also remember that doubling image size (320x240 to 640x480) requires a 4X (not 2X) increase in data rate.

Use this data rate formula to help target your movie for the right delivery medium:

Date Rate = (frames per second) x (movie width) x (movie height) divided by 35000

This translates to DR = FPS * W * H / 35000.

Here is an example: A 320x240 movie with 15 frames per second needs to be compressed to about 32.9K of data per second. Realistically, I would round this up to 35K.

Gamma, or the relative brightness of computer displays, is another issue that you need to understand. Macs and PCs display images with different gamma levels. If you're working with a compression tool that supports gamma adjustment (Cleaner, for example), an image you create on a Mac and display on a PC will look too dark. Conversely, an image created on a PC and displayed on a Mac will look too bright. The cross- platform gamma adjustment with Cleaner is +25 to +30 when going from Mac to PC and 30 when going from PC to Mac. Positive numbers lighten the image, and negative numbers darken the image.

RealMedia

RealMedia is bigger on the Windows operating system than it is on the Macintosh, mainly because RealNetworks can't compete with a player that comes embedded into the operating system such as QuickTime.

There are three different type of streaming content distribution technologies:

• Unicast Unicast streams are simple point-to-point streams, similar to a telephone call from the host server to a individual client computer. To reach many clients at once, the server must send many streams simultaneously , which is a less efficient use of bandwidth. However, viewers of a unicast stream can randomly access movies, playing only the parts they want to see. Typically, unicast is used to stream pre-recorded movies that are stored on a host computer.

• Multicast Multicast streams are sent directly to a group address, which can then be simultaneously accessed by many client computers.

• Reflected Reflected multicast streams take live media from another source, such as a radio or TV broadcast, and stream it out to viewers as a series of unicasts .

Because the RealMedia compression tool(s) save compression attributes, picking the same compression setting for many clips is easy. Remember that video on the web really isn't effective unless you can target a 256K user. This means that the best settings for the RealEncoder are these:

• 256K DSL/cable modem

• 2-pass encoding

• Variable bit rate encoding

Windows Media

Microsoft (and some web reviews) say that their new Media Player provides the best audio and video quality on the web. What they don't mention is that the quality is limited to the bandwidth of your user's connection speed. It's their connection speed that determines how high-quality a streaming file their computer can handle. The latest version of the player has incorporated many new features, including these:

• Windows Media Audio 8 (WMA8) encoding

• Smart Transcode support for the best quality transfer to portable devices

• Windows Media Audio and Video 8 decoding

• A new enterprise deployment pack for larger scale (ISP- sized ) streaming solutions

Windows Media Player is more than a streaming application; it's truly a complete "media player."

Windows Media Player has an impressive set of features and is integrated into Windows; unlike Real, which often has issues with streaming video through firewalls. See www.WindowsMedia.com for more information.

Regardless of which technology you choose to use, the "Audio Compression and Optimization" and "Video Optimization" sections will give you a start on optimizing your audio and video for delivery over the Internet or other forms of media such as CD-ROMs.

Streaming versus Downloading

The three types of video delivery methods for playback are these:

• Streaming Video that is played while it is being downloaded.

• Progressive downloading The progressive format is a blending of streaming and simple downloading. With progressive video, you get the benefit of video starting right away (with a little buffer), and when the video has completely loaded, you have the ability to save it to your computer for offline playback.

• Just plain downloading In order for downloaded video to work correctly, the entire video must download before it can start playing. The advantage, however, is that once the entire video has downloaded, it will play flawlessly, assuming the user's computer is up to the task.

Multimedia Production Tips

Media optimization projects of all sizes benefit from planning right from the beginning. To give your projects a better chance of success, keep these questions in mind while you are planning your projects:

• What are your goals, who defines them, and how will you know when you've met them?

• Who is your target audience, what kind of computers do they use, and how do they connect to the Internet?

• What are the limiting factors to delivering your media to your target audience? (This might be connection speed, CPU speed, server disk space, and so on.)

• What copyright restrictions apply to your source material if it's not original content?

Along with a project plan, you also may want to develop a storyboard and script. It is wise to consider, particularly with the bandwidth requirements of streamed audio and video, that your audience most likely will experience the end result in a small window on a computer screen.