Opening the Hood of the VMR

Opening the Hood of the VMR

Before we start to use the VMR in a DirectShow program, we need to examine the internal structure of the VMR. Unlike most other DirectShow filters, the VMR is not conceived of as a monolithic entity, but as a set of functional units, each with their own interfaces, as shown in Figure 9-1.

Figure 9-1. The Video Mixing Renderer, which is composed of a number of functional units

The Mixer is a VMR COM component charged with managing multiple video streams as they enter the VMR. Each stream is assigned a Z value when it passes through the Mixer; the ordering of Z values determines the front-to-back order of the video streams. These streams can be thought of as a series of planes, the frontmost of which has the lowest Z value (closest to zero), while increasing Z values represent planes that are progressively deeper in the background and are rendered as being behind streams with lower Z values. Thus, a stream with a Z value of 1 would be rendered in front of a stream with a Z value of 2, 5, or 15.

The Compositor is another VMR COM component. It actually performs the task of blending the various video streams into a single stream. In the strictest sense, the Compositor is where the video mixing and rendering takes place, assisted by the other logical units within the VMR. The Compositor can apply a number of effects to the streams, such as making them transparent or mapping them to particular portions of a 3D object.

The allocator-presenter (yet another COM component) provides an interface between the graphics display hardware on the PC and the VMR. The default allocator-presenter is optimized for 2D video rendering, but by writing your own custom allocator-presenter, you can assume total control over how the composited video frame is rendered by the graphics hardware. With either version of the VMR, you can render video onto Direct3D objects and manipulate those objects using all the power of DirectX.

The Core Synchronization Unit (yes, a COM component) uses the IReferenceClock interface provided by the Filter Graph Manager to keep the operation of the VMR well synchronized, ensuring that video streams passing through the VMR remain synchronized with the rest of the filter graph. If the VMR gets overloaded, the Core Synchronization Unit can drop frames from the mix to prevent VMR desynchronization with the rest of the filter graph.

Finally, the Window Manager is used only if the VMR is operating in windowed mode; that is, if the VMR is in charge of maintaining its own display window for the VMR output stream. The VMR will create a window for output by default to maintain backward compatibility with existing applications this is the way the old Video Renderer works. If a windowless mode is used, the VMR will draw its output into your application s window in a rectangle that you specify. Windowless mode is the recommended mode of operation for the VMR because you can handle all Windows messages directly in your application. VMR settings for windowless mode need to be established when the VMR is instantiated and initialized.

The VMR is configured by default with four input pins, each of which can receive its own video stream. One pin is created when the filter is instantiated, and three more are added when the first pin is connected. You can configure the filter for less than four pins, but it isn t necessary to use all available pins, and having extra unused pins on the VMR won t slow rendering. However, if you do have more than four input streams, you can create any additional pins that you might need (up to 16 inputs), but you need to do so before you start connecting any pins on the VMR.

Operating the VMR

The VMR is functionally similar to the DirectShow Video Renderer filter, except that it requires that the system have a fairly recent graphics card. The VMR requires full hardware acceleration, and Direct3D must be enabled. (You can check these capabilities on your system by using the DXDiag utility in the DirectX SDK.) The VMR sits at the downstream end of a filter graph and receives a number of video streams as input anywhere from 1 to 16 streams. However, because of the unusual setup requirements of the VMR, a few instructions need to be followed to ensure error-free use of the VMR in your filter graphs.

The VMR must be added to the filter graph and fully configured before it s connected to any other filters. Connecting the VMR to other filters freezes the VMR configuration. The VMR can t be reconfigured to change the number of input streams, go from windowed to windowless mode, and so on once it has been connected to another filter. After the VMR has been instantiated, it must be programmed with the number of input streams it will receive, and then the VMR must be put into windowless mode, if it s to be used in that way. The VMR defaults to windowed mode, and you can t switch between windowless mode and windowed mode once the VMR has been connected to an upstream filter, so you need to set that up before any connections have been made. (The same is true for setting the number of input streams.)

One final adjustment, for the mixing preferences, must be set before the VMR is connected to any upstream filters. The mixing preferences are used to determine whether the output stream is to be decimated (lose resolution) and to set up any filtering used on the input streams. Changing the mixing preferences is recommended only for advanced applications. The VMR uses sensible default values, and it s rare that an application would need to change them.

Once this initialization work is complete, the VMR can receive connections from upstream filters. The VMR is capable of receiving a broad set of video formats, and it will convert them internally to an output format suitable for rendering. If the VMR can t negotiate a connection between an upstream video source and its input pin, either the conversion couldn t be performed inside the VMR or the VMR has run out of memory. (The VMR, unlike the old Video Renderer, doesn t use system memory, only video memory, either resident on the video card itself or AGP memory on the motherboard.)

When mixing different video sources, it becomes important to recognize the differences inherent in various video formats. Video designed to display on a computer is generated in a progressive-scan format, meaning that the entire image is drawn every frame. That s different from video designed for broadcast TV, where the two-fields-per-frame are interlaced together. For display on a computer monitor, the interlaced signal must be deinterlaced, that is, converted to progressive-scan format. That s something that will happen in the VMR before the video is displayed, using one of any number of possible techniques, each of which has its own quality vs. speed tradeoffs.

The VMR has an interface that allows you to adjust the deinterlacing techniques used on each video stream. These techniques are based inside the hardware on the graphics cards, so if your DirectShow program is running on a computer with very basic graphics hardware, there might not be a lot of options available. On the other hand, a top-of-the-line graphics card will be able to exploit the features of that card for deinterlacing. Full information on deinterlacing in the VMR can be found in the Setting Deinterlace Preferences page of the DirectX SDK documentation.