Video Renderers Available to DirectShow Programmers

Video Renderers Available to DirectShow Programmers

Because video rendering depends so much on graphics hardware, it should not be surprising that as video hardware has improved over the past few years, DirectShow has added new Video Renderer filters capable of exploiting each new advance.

The (Plain Old Vanilla) Video Renderer

The initial video renderer was called, logically, the Video Renderer. It was designed using the 1.0 release of Microsoft DirectDraw back when the Intel 386 was the standard CPU and the high-end graphics cards had 4 MB of on-board memory, although many systems still had cards with only 512 KB or less. Because of the limited capabilities of the early hardware, the Video Renderer has no minimum hardware requirements. If it cannot allocate sufficient memory directly on the graphics card using DirectDraw, it will fall back to using the graphics device interface (GDI) and system memory to display the bitmaps. The Video Renderer filter was upgraded to use DirectDraw 3 if the hardware supported it. To maintain backward compatibility with existing DirectShow applications, this filter remains the default video renderer except when an application is running on Windows XP.

The Overlay Mixer

A couple of years after the Video Renderer filter was created, when DVD drives were introduced, CPUs were still not fast enough to perform the MPEG-2 decoding in software. Hardware decoders were used, equipped with video ports, which were cables that carried the decoded video signal directly from the decoder card to the video card, bypassing the host CPU and the system bus entirely. Many TV tuner cards, notably the popular ATI All-In-Wonder card, also use video port technology, although the tuner or decoder and graphics card are physically on the same board. To support video ports and a new feature on graphics cards called the overlay surface, Microsoft developed a new video renderer called the Overlay Mixer. Unlike the Video Renderer, the Overlay Mixer has a minimum hardware requirement, namely that the graphics card must have an overlay. The Overlay Mixer doesn t try to use GDI and system memory; if the graphics hardware offers no overlay surface, the filter will return an error and refuse to connect.

An overlay surface is a physical portion of memory on the graphics card distinct from the primary surface, which is the memory used to render the entire desktop. Overlay surfaces accept input in various YUV formats, which is conveniently the format in which decoders output their decoded video frames. The graphics hardware, which is significantly faster than the system CPU, converts the video into native RGB values for the display monitor inside the card s digital-to-analog converter.

For example, as a DVD is playing, the Overlay Mixer allocates sufficient memory on the overlay surface to contain the video rectangle of the specified size. It also notifies the graphics card of the size and position of that rectangle relative to the desktop. That rectangle is then filled with a color key (typically magenta) in the primary surface. Each time the card refreshes its frame buffer, it goes across each scan line, grabbing pixels from the primary surface memory and examining the color value. If the color value matches the color key, the graphics hardware discards that primary surface pixel and instead retrieves the corresponding pixel from the overlay surface memory.

Another new feature of the Overlay Mixer was its ability to handle secondary input streams containing bitmaps with closed captioning or DVD sub-picture text or graphics. If such streams are present, the Overlay Mixer will write that data over the current video frame before delivering the frame to the overlay surface.

By the time of DirectX 8, around 1999, graphics hardware had continued to advance in both speed and the amount of memory available. These advances meant that even when displaying full-screen video at 30 frames per second (fps), the graphics card was spending much of its time waiting around doing nothing. It was like having a Ferrari but being able to use it only to go to the corner store. So, because software developers abhor a vacuum, they began to think of ways to put all that wasted horsepower to work, and the result was DirectX Video Acceleration, or DXVA.

DXVA is the 2D equivalent of 3D graphics accelerators, and it uses the same principles to speed up the rendering of graphics for games and other computer-generated graphics. If a graphics driver and a software decoder both support DXVA, the decoder is able to make use of the powerful graphics processing unit to perform, in its otherwise idle moments, some of the more computationally expensive decoding operations. By having the graphics processing unit (GPU) help it decode video, the decoder s demands on the CPU are significantly reduced, so that with DXVA on a typical modern system, it s possible to play back a full-screen DVD while putting only minimal demands on the CPU. The Overlay Mixer and both versions of the Video Mixing Renderer all support DXVA through API calls that enable the transfer of data between the decoder and the graphics hardware.

The Video Mixing Renderer 7

For all its advances over the Video Renderer, the Overlay Mixer still could not perform any true mixing of video images (also known as alpha blending), and it could not accept more than one video stream as input. Internally, it was still using older versions of DirectDraw. To address these limitations, the Video Mixing Renderer 7 (VMR 7) was introduced for the Windows XP operating system. The VMR 7 enables true blending of multiple incoming video streams, allowing for a whole range of new effects. It s also more resourceful than the Overlay Mixer in finding some video memory to use. It will first attempt to allocate memory on the overlay surface. If none is available, it will try the primary surface, then system video memory in the AGP address space, and finally, if all else fails, it will use system memory.

Another advance in the VMR 7 was its use of DirectDraw 7 rather than the mix of DirectDraw 3 and DirectDraw 1 used in the Overlay Mixer and Video Renderer. The VMR 7 didn t use Direct3D 8 to control the graphics hardware because this API didn t provide the required support for off-screen plain surfaces or DXVA decoding. But the most interesting innovation in the VMR 7 was its component architecture, which enables you to create your own custom allocator-presenter components (explained in the Opening the Hood of the VMR section later in this chapter) and take complete control over the rendering process. By writing your own allocator-presenter, you can render your video surfaces onto Direct3D geometries and create stunning video effects.

The Video Mixing Renderer 9

Adoption of the VMR 7 has been limited by the fact that it was available only for applications running on Windows XP. This situation has been rectified with the latest of the DirectShow video renderers, the Video Mixing Renderer 9 (VMR 9). This filter offers all the features of the VMR 7, simplifies the code required to insert your own allocator-presenter, and is available on all platforms supported by DirectX 9. The VMR 9 uses Direct3D 9 to control the graphics hardware. It offers support for hardware deinterlacing capabilities available on the latest graphics cards, and its support for Direct3D 9 pixel shaders means that you can control hue, saturation, and brightness on a per-stream basis. As you might expect, the VMR 9 hardware requirements are the most stringent of all the video renderers: the VMR 9 requires a fairly recent graphics card.

Almost as interesting as what the VMR 9 supports is what it doesn t support: video ports and overlay surfaces. Although these technologies were instrumental in making quality video possible on the PC, neither is necessary any longer given the sheer speed of modern CPUs and graphics hardware. Video ports are no longer needed because full-screen video can stream at 30 fps from a TV device or a DVD drive into system memory, where it s fully or partially decoded, and then back to the video graphics card through the AGP port.

Overlay surfaces are problematic because there s only one overlay surface per system, but users often want to display multiple video clips simultaneously. Also, the overlay requires a key color, which can interfere with any effects that the Windows shell is trying to do with windows that interest the video playback window. And because an RGB image of the video frame never actually exists in the overlay memory, pressing the Print Screen key copies only the color key, not the video image that the user expects. Therefore, it s much preferable, and now possible, to render video streams onto the device s primary surface and this is what the VMR 9 does.