by Jason Wolf
In this chapter, you'll learn how to optimize both the
Multimedia Dominates Traffic, Film at 11
Most streaming-media files are low in bit-rate (less than 56Kbps), small in size (less than 1MB), and short in duration (under 10 minutes). However, a small fraction (3 percent) of streaming media is responsible for nearly half of this traffic.
P2P traffic is also on the rise. At a study at the University of Washington in May 2002, peer-to-peer traffic (
 Maureen Chesire, Alec Wolman, Geoffrey M. Voelker, and Henry M. Levy, "Measurement and Analysis of a Streaming-Media Workload," in Proceedings of the Third USENIX Symposium on Internet Technologies and Systems (Berkeley, CA: USENIX Association, 2001).
 Stefan Saroiu et al., "An Analysis of Internet Content Delivery Systems," in Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI) (Berkeley, CA: USENIX Association, December 2002).
Multimedia applications allow you to create engaging interactive experiences. Remember, the tricky part is not creating your content but delivering your content
This section introduces you to some fundamental multimedia concepts. Learning these concepts is a necessary starting point from which to build. You can think of multimedia as a collection of many different forms of data types (file formats) that are all used together to give the audience a richer,
Before you can understand the process and procedures behind
Effective audio can help deliver your message with impact. Audio is as important as visuals in your presentation. Audio can
Audio can be contained in and delivered with different types of files. For example, you might digitize your audio and save the file in Sound Designer II format on your hard drive, but when it comes time to deliver that same audio file, you'll want to compress the data into something like the MP3 format. Compression saves storage space and speeds delivery.
Most of the audio files you'll find on the Internet are in one of the formats discussed in the
MP3 is a compressed audio format that sounds
Qdesign Music is a compressed audio format that comes in standard and pro versions. The
The Qdesign Music codec also comes in a professional versionfor more money, of course. It allows control over targeting a specific bit-rate, has a setting so you can tell it to lean more toward "quality" or "speed" (size), and features two pass variable bit-rate encoding for more precise data-rate targeting. See http://www.qdesign.com for more information.
Qualcomm PureVoice is a codec designed for encoding voice signals. It has some amazing compression capabilities that can be helpful when you are simply using voice as your source material. The codec allows you to change the sample rate, the bit rate, the number of channels, and the setting that determines whether to use the "full rate" or "half rate" settings. The full rate settings can compress your audio in a 9:1 ratio, and the half rate setting can achieve a 19:1 compression ratio. This is pretty heavy compression, but nonetheless the audio is more than acceptableit sounds like a long-distance phone call.
IMA 4:1 is a compressed delivery format from the Interactive Multimedia Association. As its
MP4 is the new audio standard replacing MP3. MP4 supports bit rate, sample rate, depth, channel, hinted streaming tracks, and more. What makes this the codec of choice is that its compression algorithms are better. MP4 uses the Advanced Audio Coding (AAC) method to produce smaller files and higher quality. In fact, AAC-compressed sounds deliver higher quality than uncompressed CD audio. A 20-second 16-bit, 22.5kHz audio file of 900K compresses to 128K in MP3 at 40kHz and sounds muddy. The same file compressed with MP4 is 108K and sounds nearly identical to the original. The trouble is that because this codec is so new, most people don't have the means to decode it (another thing to worry about!). See http://www.apple.com/mpeg4/ and http://mpeg.telecomitalialab.com/ for more information.
This format is used extensively in Microsoft Windows to store and deliver audio, and is also used in wavetable synthesis, such as E-mu's SoundFont. Conversion tools can convert .wav files to other operating systems. See http://www.microsoft.com/ for more information.
QuickTime is both a file container and delivery format; that is, you can use QuickTime to embed MP3, AIFF, MP4, and video in your web pages and CD-ROMs. The QuickTime format is cross-platform for people with Macs and PCs. You can use Quick Time files to edit your audio and then compress them for distribution, making this a very versatile format.
The next section explains video formats for the web and their strengths and weaknesses.
When video is digitized, it becomes "data" that needs to be contained inside some type of file format, such as QuickTime. Once video is stored in an electronic file, it can be edited (rearranged in a new order other than that in which it was shot), and then the video can be compressed and delivered to your target device, such as the web or a CD-ROM. Here are the different types of video formats you will encounter:
QT is the king of video formats. Over 15
RealMedia This is RealNetworks' cross-platform audio and video streaming delivery technology. It allows different target bandwidths and desktop or web playback. Unlike QuickTime, RealVideo files are only stored compressed, so there's no way to revert to raw video or audio for editing.
Windows Media Player Microsoft's answer to QuickTime. The Windows Media Player will play back either compressed or streaming WMP files in real time to all target bandwidths, such as a DSL or a 56Kbps modem. WMP can deliver compressed, downloadable files or stream a live signal. See http://www.microsoft.com for more information.
Because compression for QuickTime, RealPlayer, and Windows Media Player are all very similar, I explain the concepts further in the "Codecs" section, later in this chapter.
Animation on the Internet comes in two primary forms: the traditional 2D animation and the more recent 3D animation. 2D animation is the equivalent of cartoon-style frame-by-frame animation. Modern computers have made this process much easier by way of keyframes. Now
There are currently three main data types for web-based animation:
The most popular animation format on the web. Vector-based format files are
Shockwave and Shockwave 3D
Director allows the creation of vector or bitmapped-based content compressed into DCR (Director Compressed Resource) files that are played back on the web using Shockwave. Macromedia and Intel recently introduced Shockwave 3D, which also supports 3D data from applications like 3ds max or Maya for 3D experiences. Your audience can fly through online stores and examine items with HTTP links. However, creating 3D
You didn't think there weren't going to be any limitations to multimedia, did you? Multimedia work requires cutting-edge technology. When you are developing multi media, you are working in the latest video, audio, and
In short, when you're a multimedia developer, you always have to have the latest technology to impress your
Once you get used to continuously upgrading, your next limitation will be the raw speed of your connection to the Internet. You generally need the
Get the fastest processor you can afford because compression often is
The speed of your connection will always feel like a limiting factor, regardless of how fast it is. When I started using the Internet, 1200- and 2400-baud modems were the fastest modems you could use, but they felt slow to me. A few years later, everyone had 56,000 baud, but that speed felt faster only temporarily because larger sites made even that speed feel slow. Today I'm sitting at the
It's not your connection speed but your customers' speed that's important.
Regardless of your speed, you need to learn to compress your files as much as possible.
Considering that most people don't upgrade, you also need to make versions of your files backward compatible.
When you have files on your hard drive and you plan on delivering them either over the Internet, via CD-ROM, or even by email, they need to be compressed. Compression actually has two
You can actually purchase codecs for video or audio that are professional versions designed for multimedia professionals. For example, the Sorenson Pro Developer Codec is used by Industrial Light & Magic (ILM) to compress all the
trailers you see on the Internet (http://www.apple.com/
Without the use of compression, video and audio could not effectively be distributed over the Internet. Consider the Star Wars trailer, for example. The high-speed broadband version of that video clip on the Internet is about 25MB in size. That same clip uncompressed could easily be 250 to 500MB in size; this would take up the better part of an entire CD-ROM.
In addition to delivery, codecs make video playback possible. Video is a CPU-intensive medium simply because there is so much data involved. One second of NTSC video captured from the miniDV format is about 3.5MB of disk space. This means that every 60 seconds of video is 210MB of data, and every five minutes is a little over 1GB of data (or 9,000
Compression is an extremely complex operation that in some cases takes days for your computer to calculate. But remember that the compression you do prior to delivery creates smaller finished video, which plays smoother on more machines.
Compressing video works by removing redundant data in two ways: on a frame-per-frame (spatial) basis, and on an over-time (temporal) basis.
Spatial compression is where the compression application looks at every frame of your video and then groups every pixel in that one frame based on how close pixel colors are to one another. The algorithm then sets similar RGB
In addition to pixel compression, codecs can do something called temporal compression. Temporal is an over-time compression technique over which you really don't have too much control. The Sorenson codec, for example, creates natural keyframes when the video changes dramatically from one scene to the next. These keyframes are compressed more lightly than your entire video and are used by the codec as references on how to draw the frames around it.
Temporal compression smoothes pixels over time and tries to remove video noise. Temporal settings also can help you remove one-frame artifacts.
Streaming content is multimedia content that is played on the
Apple's QuickTime is the best solution for video delivery on the Macintosh and some would say on the PC as well. The streaming aspect of QuickTime is no less than
Apple provides two types of streaming technologies:
Real-Time, for streaming video over the Internet during a live event.
Progressive, for delivering video over the Internet with
With real-time, you can hook up your video camera to your computer, which is running a copy of QuickTime's Broadcaster. This allows you to send live video over a streaming server out to thousands of
Figure 13.1. The QuickTime streaming server is powerful and simple to use.
NOTE: With the new ISO standards, MPEG4 is now starting to show up in cellphones and other PDA devices. The QuickTime streaming server will enable you to communicate with all of these devices.
In terms of compression for streaming QuickTime content, there is not much to do other than normal compression. Video compression is both spatial and temporal. With streaming content, you pick a target bandwidth, video size, and frame rate; enable "hinted tracks;" and let the file compress. Once it's done, you upload the file to your server and make a link to it on your web page. When a user clicks on this link, the video is
Dial-up modems are almost useless for video.
Your client should be able to handle a 256Kbps download or it's not worth it.
Use QuickTime progressive for content below 256Kbps downloads.
QuickTime progressive downloads begin playing as soon as enough content is buffered to allow smooth playback for the duration of the clip.
For More Information
If you think you'll use QuickTime, go to http://www.apple.com/quicktime/tools_tips/tutorials/activex.html for instructions on making your QuickTime files viewable on all platforms and browsers. If you are using version 5.5 SP2 of Internet Explorer for Windows, you must view QuickTime as an ActiveX control because Microsoft has discontinued support for Netscape style plug-ins.
I've found that people are much more accepting of temporal compression (over-time) of the video than spatial (frame-per-frame) compression. By cutting the frame rate from l5fps to 10fps or even 8fps, each frame has 50 percent more data per frame for the same file size, which will increase the quality of the picture. The minimum size should be 320x240. If your clients have a fast Internet connection, 400x300 is better. Anything smaller than 320x240 is just too small. To maintain good quality, always up the data rate in proportion to the image size. Also remember that doubling image size (320x240 to 640x480) requires a 4X (not 2X) increase in data rate.
Use this data rate formula to help target your movie for the right delivery medium:
Date Rate = (frames per second) x (movie width) x (movie height) divided by 35000
Here is an example: A 320x240 movie with 15 frames per second needs to be compressed to about 32.9K of data per second. Realistically, I would round this up to 35K.
Gamma, or the relative brightness of computer displays, is another issue that you need to understand. Macs and PCs display images with different gamma levels. If you're working with a compression tool that supports gamma adjustment (Cleaner, for example), an image you create on a Mac and display on a PC will look too dark. Conversely, an image created on a PC and displayed on a Mac will look too bright. The cross- platform gamma adjustment with Cleaner is +25 to +30 when going from Mac to PC and 30 when going from PC to Mac. Positive numbers lighten the image, and negative
RealMedia is bigger on the Windows operating system than it is on the Macintosh,
There are three different type of streaming content distribution technologies:
Unicast streams are simple point-to-point streams, similar to a telephone call from the host server to a individual client computer. To reach many clients at once, the server must send many streams
Multicast streams are sent directly to a
Reflected multicast streams take live media from another source, such as a radio or TV broadcast, and stream it out to viewers as a series of
Because the RealMedia compression tool(s) save compression attributes, picking the same compression setting for many clips is easy. Remember that video on the web really isn't effective unless you can target a 256K user. This means that the best settings for the RealEncoder are these:
256K DSL/cable modem
Variable bit rate encoding
Microsoft (and some web reviews) say that their new Media Player provides the best audio and video quality on the web. What they don't mention is that the quality is limited to the bandwidth of your user's connection speed. It's their connection speed that determines how high-quality a streaming file their computer can handle. The latest version of the player has incorporated many new features, including these:
Windows Media Audio 8 (WMA8) encoding
Smart Transcode support for the best quality transfer to portable devices
Windows Media Audio and Video 8 decoding
A new enterprise deployment pack for larger scale (ISP-
Windows Media Player is more than a streaming application; it's truly a complete "media player."
Windows Media Player has an impressive set of features and is integrated into Windows; unlike Real, which often has issues with streaming video through firewalls. See www.WindowsMedia.com for more information.
Regardless of which technology you choose to use, the "Audio Compression and Optimization" and "Video Optimization" sections will give you a start on optimizing your audio and video for delivery over the Internet or other forms of media such as CD-ROMs.
The three types of video delivery
Streaming Video that is played while it is being downloaded.
Progressive downloading The progressive format is a blending of streaming and simple downloading. With progressive video, you get the benefit of video starting right away (with a little buffer), and when the video has completely loaded, you have the ability to save it to your computer for offline playback.
Just plain downloading
In order for downloaded video to work correctly, the entire video must download before it can start playing. The advantage, however, is that once the entire video has downloaded, it will play flawlessly,
Media optimization projects of all sizes benefit from planning right from the beginning. To give your projects a better chance of success, keep these questions in mind while you are planning your projects:
What are your goals, who defines them, and how will you know when you've met them?
Who is your target audience, what kind of computers do they use, and how do they connect to the Internet?
What are the limiting factors to delivering your media to your target audience? (This might be connection speed, CPU speed, server disk space, and so on.)
What copyright restrictions apply to your source material if it's not original content?
Along with a project plan, you also may want to develop a storyboard and script. It is wise to consider, particularly with the bandwidth requirements of streamed audio and video, that your audience most likely will experience the end result in a small window on a computer screen.