Chapter 10: Taking Video to the Third Dimension

In this chapter, we take video out of the window and move it into the world of 3-D. The key feature of the VMR that makes this possible is renderless mode, which gives us near complete control over how we present the video frames .

Video Cut Scene

In games , a common use for video is to create cut scenes ” short, full-screen video clips. You might use a cut scene as a title sequence, a segue between levels, or in the closing credits. Cut scenes can give your game a dramatic, cinematic feel. The first sample application in this chapter shows how to render a cut scene that combines video with 3-D elements.

Build and run the Ducky sample that is located in the AVBook\bin directory. The Ducky sample plays a video of waves crashing against rocks, while a 3-D model of a duck rotates in front of the video. (See Plate 6. We re not sure what kind of game would use this cut scene, we just liked the duck.) You can use the arrow keys to move the camera around the duck. Press the Escape key to quit the sample.

Keyboard control	Camera movement
Left/right arrow	Rotate left/right (yaw)
Shift + left/right arrow	Slide left/right
Up/down arrow	Slide forward/backward
Shift + up/down arrow	Rotate up/down (pitch)

About the 3-D Framework

All of the Direct3D sample applications in this book are built on the same framework, which handles common tasks such as setting up Direct3D, rendering the scene, and so on. If you re curious and want to see the framework without any of the DirectShow code, look at the D3DApp_Basic sample that is located in the AVBook\bin directory. The framework was designed to make the code easy to follow, so it s not necessarily very efficient, and it s certainly not a full-fledged game engine. For example, there is no collision detection, so the camera can move freely through meshes, which produces a rather disconcerting effect.

The following table lists all the C++ classes defined in the base framework. We won t describe all of the code in these classes, but the source files are well commented.

Class	Description
CCamera	Defines the camera, which is used to set the view matrix.
CClock	Simple timer for controlling the frame rate of the 3-D scene.
CD3D	Manages the IDirect3D9 interface.
CDevice	Manages the Direct3D device.
CDisplayModes	Helper class to enumerate the available display modes.
CFrame	Hierarchical frame class, used to build the scene graph.
CGame	Manages the entire rendering process. This class contains the device, clock, camera, keyboard, and scene objects.
CKeyboard	Manages keyboard input, using DirectInput.
CMatrixStack	Manages a matrix stack, used to store local world transforms when the scene graph is rendered. This class is a simple wrapper for the ID3DXMatrixStack interface, which is part of the Direct3D utility library.
CMesh	Manages a 3-D mesh. This class is a simple wrapper for the ID3DXMesh interface from the Direct3D utility library.
CRefCountedObject	Base class that implements reference counting using the AddRef and Release methods (but not QueryInterface ). This class is defined so that reference counting can be applied to objects that do not have COM interfaces.
CScene	Abstract class for managing a Direct3D scene. Each Direct3D sample application in this book defines its own subclass that derives from CScene , whose purpose is to load the scene and render it. The subclass contains the scene graph, meshes, and other scene objects.

VMR Plug-In Components

In the previous chapter, we used the VMR filter as is, but the VMR has two plug-in components, shown in Figure 10.1, that you can replace with your own custom components:

Allocator-presenter

• This component creates Direct3D surfaces (allocation) and draws video frames on the screen (presentation).

Compositor

• This component mixes the incoming video streams.

The filter itself manages everything else: pin connections, format negotiation with the decoders, and all the other DirectShow overhead.

Figure 10.1: VMR plug-in components.

By replacing the allocator-presenter or the compositor with your own custom versions, you can control those portions of the rendering process. The allocator-presenter can be replaced when the VMR is in renderless mode (in fact, this is the definition of renderless mode), while the compositor can be replaced in any mode. In this chapter, we focus on the allocator-presenter. In the next chapter, we show how to write a custom compositor.

Mixing Mode versus Pass-Through Mode

In order for the VMR to mix two or more video streams, it must load the compositor into memory. The VMR s default compositor also performs several other useful functions besides mixing. For example, it automatically deinterlaces the source video when needed. Therefore, it can be useful to have the compositor loaded even if you are rendering a single video stream. Whether the compositor is loaded also affects the operations of the allocator-presenter.

When the VMR loads the compositor, it is said to be operating in mixing mode. Otherwise, it is in pass-through mode. Which of these two modes the VMR selects by default depends on the rendering mode, and also on which version of the VMR you are using. The following table lists the default behavior of both VMR versions.

Version	Rendering mode	Default behavior
VMR-7	Windowed	Pass-through
	Windowless	Pass-through
	Renderless (custom allocator-presenter)	Pass-through
VMR-9	Windowed	Mixing
	Windowless	Mixing
	Renderless	Pass-through

As the previous table shows, the VMR-7 always defaults to pass-through mode. The VMR-9 defaults to mixing mode when it s in windowed mode or windowless mode, but it defaults to pass-through mode when it s in renderless mode. You can always activate mixing mode by calling IVMRFilterConfig::SetNumberOfStreams on the VMR-7, or IVMRFilterConfig9::SetNumberOfStreams on the VMR-9. This method must be called before you connect any of the VMR s input pins.

Why is all this important? Because the VMR behaves differently in mixing mode and pass-through mode, and these differences will affect your custom allocator-presentor.

The VMR Rendering Process: Mixing Mode

In mixing mode, shown in Figure 10.2, the VMR composites the incoming video streams onto a single Direct3D surface, called the back-end surface . The video decoders do not touch the back-end surface; instead, the VMR assigns each decoder its own pool of surfaces. The allocator-presenter creates the back-end surface and the VMR creates the decoder surfaces.

Whenever a decoder generates a new video frame, the VMR collects the most recent frame from each decoder and passes them all to the compositor, which mixes them onto the back-end surface. At that point, the VMR calls into the allocator-presenter to present the back-end surface to the screen. The allocator-presenter might do something as simple as call StretchRect , or it might perform a complex texture operation. The VMR s default allocator-presenter uses a flipping chain that is associated with the video window.

Figure 10.2: VMR mixing mode.

The VMR Rendering Process: Pass-Through Mode

In pass-through mode, shown in Figure 10.3, the VMR performs no video mixing. Therefore, only one video stream is allowed. Nor does the VMR create a back-end surface: The video frames are presented as they come out of the decoder, without any modification by the compositor. The allocator-presenter creates the decoder surfaces and presents the decoded video frames. The compositor is not loaded, so it is not involved in the process at all.

Figure 10.3: VMR pass-through mode.

In the Ducky sample, we use pass-through mode, because it keeps things slightly simpler than mixing mode. However, mixing mode is more powerful, as the next sample application will demonstrate .

Writing a Custom Allocator-Presenter

An allocator-presenter must implement two COM interfaces: IVMRSurfaceAllocator9 , which defines methods for allocating surfaces, and IVMRImagePresenter9 , which defines methods for presenting surfaces. These two interfaces provide a consistent way for the VMR to communicate with any allocator-presenter.

The allocator-presenter receives its allocation requirements from the VMR, based on the video format and the mixing mode. The VMR also tells the allocator-presenter when to present each video frame. The Direct3D device is created either by the allocator-presenter or directly by the application. Either way, the VMR must be notified of the device pointer.

On the application side, the application must configure the VMR for renderless mode and hook up the custom allocator-presenter. As far as building and running the filter graph, we can reuse most of the code from the previous chapter.

Outline of the Ducky Sample

The Ducky sample adds some new C++ classes to the D3DApp_Basic framework.

Class	Description
CAllocator	Implements the custom allocator-presenter.
CCriticalSection	Thin wrapper for a Windows CRITICAL_SECTION structure.
CGraph	Manages the DirectShow filter graph.
CGraphEventHandler	Defines a callback for handling filter graph events.
CLock	Locks a critical section before entering a code path , and automatically unlocks the critical section on exit.
CVmrGame	Derives from CGame but adds video functionality.

The CGraph and CGraphEventHandler classes are adapted from the Mangler sample in Chapter 9, with minor modifications to handle the VMR in renderless mode.

Here s the overall sequence of events when the Ducky sample runs:

Initialization

1. Initialize COM.

2. Create the application window.

3. Create the Direct3D object and the Direct3D device.

4. Initialize DirectInput for keyboard input.

5. Load the scene: Load meshes, set up lights, set the render states, and build the scene graph hierarchy. Set the initial camera position and the projection matrix.

6. Initialize DirectShow: Configure the VMR for renderless mode, build the filter graph, and set up the custom allocator-presenter.

Rendering

1. Show the application window.

2. Start the DirectShow filter graph.

Clean up

1. Stop the filter graph.

2. Release everything.

3. Shut down COM.

While the filter graph is running, the VMR notifies the allocator-presenter whenever a new video frame should be rendered. This happens on a callback function, so the message loop of the sample application does not do very much. Most of the code in the Ducky sample is generic Windows programming or Direct3D setup code, and will not be covered in detail.

Building the Filter Graph

The code to build the filter graph is quite similar to the Mangler sample from the previous chapter, except the VMR must be configured for renderless mode instead of windowless mode. We define a new function for this purpose, InitVMR_RenderlessMode , which replaces the InitVMR_WindowlessMode function that was used in the Mangler sample.

 HRESULT CGraph::InitVMR_RenderlessMode(      IVMRSurfaceAllocatorNotify9 **ppAlloc)  {      // Create the Filter Manager.      HRESULT hr = m_pGraph.CoCreateInstance(CLSID_FilterGraph);      m_pGraph.QueryInterface(&m_pControl);      m_pGraph.QueryInterface(&m_pEvent);      // Create the VMR-9 and add it to the graph.      hr = AddFilterByCLSID(m_pGraph, CLSID_VideoMixingRenderer9, &m_pVMR);      // Set the VMR-9 to renderless mode.      CComQIPtr<IVMRFilterConfig9> pConfig(m_pVMR);      hr = pConfig->SetRenderingMode(VMR9Mode_Renderless);      // Return the IVMRSurfaceAllocatorNotify9 interface pointer      // to the caller.      return m_pVMR.QueryInterface(ppAlloc);  } 

Again, we ve left out some HRESULT tests, in the interest of making the code examples shorter. To establish renderless mode, we call the IVMRFilterConfig9::SetRenderingMode method with the VMR9Mode_Renderless flag. Then we query the VMR for the IVMRSurfaceAllocatorNotify9 interface, which controls the VMR in renderless mode ” analogous to the role of the IVMRWindowlessControl9 interface in windowless mode. As in the Mangler sample, the order of these method calls is important, because the VMR does not expose the IVMRSurfaceAllocatorNotify9 interface until the application has specified renderless mode.

The InitVMR_RenderlessMode function is called from the CVmrGame::InitializeVMR function, which initializes the VMR and the allocator-presenter.

 HRESULT CVmrGame::InitializeVMR(      HWND hwnd, WindowMode mode, CScene *pScene)  {      // Set up the VMR for renderless mode.      HRESULT hr = m_Graph.InitVMR_RenderlessMode(&m_pVmrNotify);      // Tell the VMR about the Direct3D device and the monitor.      HMONITOR hMonitor = g_d3d->GetAdapterMonitor(D3DADAPTER_DEFAULT);      hr = m_pVmrNotify->SetD3DDevice(m_pDevice, hMonitor);      // Hook up the VMR and the allocator-presenter.      hr = m_pVmrNotify->AdviseSurfaceAllocator(0, m_pAlloc);      hr = m_pAlloc->AdviseNotify(m_pVmrNotify);      return hr;  } 

The application gives the VMR a pointer to the Direct3D device by calling IVMRSurfaceAllocatorNotify9::SetD3DDevice . This method also specifies the monitor that we are using, which is easily obtained by calling IDirect3D9::GetAdapterMonitor . The Ducky sample always uses the primary display adapter.

Note  

When you create the Direct3D device, set the device type to D3DDEVTYPE_HAL . Otherwise the SetD3DDevice method will fail, because the VMR requires hardware acceleration. Also set the D3DCREATE_MULTITHREADED flag.

The IVMRSurfaceAllocatorNotify9::AdviseSurfaceAllocator method informs the VMR about the custom allocator-presenter. It takes a pointer to the allocator-presenter s IVMRSurfaceAllocator9 interface, represented by the m_pAlloc variable. The IVMR SurfaceAllocator9::AdviseNotify method does the reverse ” it tells the allocator-presenter about the VMR. Together, these two methods establish the necessary link between the VMR and the allocator-presenter. (We ll see how AdviseNotify is implemented shortly.)

Now we can build the rest of the filter graph by loading the video file and connecting it to the VMR. This code is exactly the same as the code in the previous chapter.

 hr = m_Graph.RenderVideoStream(wszVideoFile); 

Note that a second call to RenderVideoStream would fail in this case, because the VMR is in pass-through mode and cannot mix a second video stream.

That covers the graph-building side of things. Next we ll look at the code for the allocator-presenter, and then we ll show how the sample application renders the 3-D scene.

Implementing the Allocator-Presenter

Here is the declaration of the CAllocator class, which implements our custom allocator-presenter.

 class CAllocator :      public IVMRSurfaceAllocator9, public IVMRImagePresenter9  {      friend class CVmrGame;  public:      CAllocator(CVmrGame *pGame);      virtual ~CAllocator();      HRESULT SetDevice(IDirect3DDevice9 *pDevice);      // IUnknown methods.      STDMETHODIMP_(ULONG) AddRef();      STDMETHODIMP_(ULONG) Release();      STDMETHODIMP QueryInterface(REFIID riid, void **ppv);      // IVMRSurfaceAllocator9 methods.      STDMETHODIMP AdviseNotify(IVMRSurfaceAllocatorNotify9* pNotify);      STDMETHODIMP GetSurface(DWORD_PTR dwUserID, DWORD SurfaceIndex,          DWORD SurfaceFlags, IDirect3DSurface9** ppSurface);      STDMETHODIMP InitializeDevice(DWORD_PTR dwUserID,          VMR9AllocationInfo* pAllocInfo, DWORD* pNumBuffers);      STDMETHODIMP TerminateDevice(DWORD_PTR  dwID);      // IVMRImagePresenter9 methods.      STDMETHODIMP PresentImage(DWORD_PTR dwUserID,          VMR9PresentationInfo* pPresInfo);      STDMETHODIMP StartPresenting(DWORD_PTR dwUserID) { return S_OK; }      STDMETHODIMP StopPresenting(DWORD_PTR dwUserID)  { return S_OK; }  protected:       IDirect3DSurface9 *m_pVideoSurf;  private:      void ReleaseSurfaces();      long m_nRefCount;            // Our reference count.      CCriticalSection m_CritSec;  // Critical section object.      CVmrGame* m_pGame;           // Parent game object.      // VMR interface pointers.      CComPtr<IVMRFilterConfig9> m_pConfig;      CComPtr<IVMRSurfaceAllocatorNotify9> m_pNotify;      // D3D interface pointers.      CComPtr<IDirect3DDevice9> m_pDevice;      // Surfaces.      IDirect3DSurface9 **m_pSurf;    // Surfaces we provide to the VMR.      DWORD m_dwNumSurfaces;          // The number of surfaces.  }; 

The CAllocator class inherits both of the COM interfaces that it supports. The public declaration section begins with three functions that are specific to this class, and are not part of the COM interfaces: the constructor, the destructor, and the SetDevice method. These are followed by declarations of IVMRSurfaceAllocator9 and IVMRImagePresenter9 , plus the IUnknown methods, which are inherited from the other two interfaces.

Implementing the IUnknown Methods

Because the allocator-presenter exposes two COM interfaces, it must implement the methods in IUnknown . Fortunately, unless you re doing something complicated, you can use boilerplate code for all of your IUnknown implementations . Here is the code for AddRef and Release .

 STDMETHODIMP_(ULONG) CAllocator::AddRef()  {      return InterlockedIncrement(&m_nRefCount);  }  STDMETHODIMP_(ULONG) CAllocator::Release()  {      assert(m_nRefCount >= 0);      ULONG uCount = InterlockedDecrement(&m_nRefCount);      if (uCount == 0)      {          delete this;      }      // For thread safety, return the temporary variable, not the      // class member variable.      return uCount;  } 

The reference count is stored in the m_nRefCount variable. The InterlockedIncrement and InterlockedDecrement functions increase and decrease the reference count by one. Both of these functions are thread-safe. It s crucial that we make the CAllocator object safe for multithreaded use, because the allocator-presenter is called from at least two threads, the sample application thread and the DirectShow streaming thread. When the reference count goes to zero in the Release method, the object deletes itself.

Here is the code for QueryInterface .

 STDMETHODIMP CAllocator::QueryInterface(REFIID iid, void **ppv)  {      if (ppv == NULL)      {          return E_POINTER;      }      if (iid == __uuidof(IUnknown))      {          *ppv = static_cast<IUnknown*>(               static_cast<IVMRSurfaceAllocator9*>(this));      }      else if (iid == __uuidof(IVMRSurfaceAllocator9))      {          *ppv = static_cast<IVMRSurfaceAllocator9*>(this);      }      else if (iid == __uuidof(IVMRImagePresenter9))      {          *ppv = static_cast<IVMRImagePresenter9*>(this);      }      else      {          return E_NOINTERFACE;      }      AddRef();      return S_OK;  } 

In C++ implementations of COM objects, an interface pointer is really just a pointer to the class that exposes the interface. You can think of the interface as a window into the class ” the client can see only those class methods defined in the interface. The class provides the implementation, while the interface provides the type definition. The client knows about the interface, but nothing about the class, so the class pointer must be coerced to an interface pointer. The static cast operation is valid here because the class inherits the interface.

You might notice the double cast operation when the IUnknown interface is requested . This unusual construct is needed because CAllocator inherits IUnknown twice, once from IVMRSurfaceAllocator9 and once from IVMRImagePresenter9 . Therefore, casting the this pointer directly to an IUnknown pointer is ambiguous as far as the compiler is concerned . To disambiguate, we must cast the pointer first to an IVMRSurfaceAllocator9 or IVMRImagePresenter9 pointer (the choice is arbitrary), and from there to an IUnknown pointer. Just one of the minor obscure mysteries of COM.

Before returning the interface pointer, the object increments its own reference count. The client will later call Release on the interface pointer, for a net change of zero.

Note  

Some people prefer to use ATL to implement COM objects. If you re familiar with the ATL COM wizards and are comfortable with them, by all means use them. Our personal feeling is that ATL is great for doing anything more complicated than what we ve done here ” for example, if your COM object needs a class factory, a type library, or (shudder) a proxy/stub DLL, then ATL is definitely the way to go. But for a simple IUnknown implementation, we find it s easier just to cut and paste the code that we ve shown.

Implementing the IVMRSurfaceAllocator9 Methods

The IVMRSurfaceAllocator9 interface controls how the allocator-presenter creates Direct3D surfaces. The interface contains four methods. We ll discuss them in the order that they are typically called.

Advise Notify

This method should be called by your application to inform the allocator-presenter about the VMR, as we saw in the CVmrGame::InitializeVMR method.

 HRESULT AdviseNotify(    IVMRSurfaceAllocatorNotify9* pNotify  ); 

The pNotify parameter is a pointer to the VMR-9 filter s IVMRSurfaceAllocatorNotify9 interface. The allocator-presenter needs to store the pointer; it will be used later to call methods on the VMR. Here is the CAllocator implementation.

 HRESULT CAllocator::AdviseNotify(IVMRSurfaceAllocatorNotify9* pNotify)  {      CLock lock(&m_CritSec); // Hold the critical section.      if (!pNotify)      {          return E_POINTER;      }      m_pNotify = pNotify;      return S_OK;  } 

Note the use of the CLock object, declared in the first line. This object holds the critical section of the allocator-presenter until the lock variable goes out of scope. Holding the critical section protects the allocator-presenter s instance data from being modified by another thread until the critical section is released. You ll see this paradigm throughout the CAllocator code.

InitializeDevice

The VMR calls this method whenever it needs the allocator-presenter to create one or more Direct3D surfaces. The method might be called several times as the VMR negotiates different formats with the decoder.

 HRESULT InitializeDevice(      DWORD_PTR  dwUserID,      VMR9AllocationInfo* pAllocInfo,      DWORD*  pNumBuffers  ); 

The dwUserID parameter contains whatever value the sample application used for dwUserID when it called IVMRSurfaceAllocatorNotify9::AdviseSurfaceAllocator . If your sample application connects several instances of the VMR filter to the same allocator-presenter object, you can use this value to identify which filter instance is calling the allocator-presenter. Because the Ducky sample uses only one VMR instance, we can ignore this parameter. (In Chapter 12, we ll see a sample that uses more than one VMR instance.)

The pAllocInfo parameter is a pointer to a VMR9AllocationInfo structure. The VMR fills in this structure with information about the Direct3D surfaces that it needs. We ll examine the structure fields in a moment.

The pNumBuffers parameter is a pointer to a DWORD value. When the VMR calls InitializeDevice , it sets the value equal to the maximum number of surfaces that it requires. Before the method exits, the allocator-presenter should set the value equal to the actual number of surfaces that were created.

Here is the definition of the VMR9AllocationInfo structure .

 typedef struct VMR9AllocationInfo {    DWORD     dwFlags;    DWORD     dwWidth;    DWORD     dwHeight;    D3DFORMAT Format;    D3DPOOL   Pool;    DWORD     MinBuffers;    SIZE      szAspectRatio;    SIZE      szNativeSize;  }; 

The structure members describes the allocation requirements:

• The dwFlags field indicates what type of surface to create. For example, the VMR may request an offscreen surface or a texture. The valid flags are defined by the VMR9SurfaceAllocationFlags enumeration.

• The dwWidth and dwHeight fields define the size of the surface.

• The Format field defines the surface format. The format could be just about any Direct3D surface format. If the value is D3DFMT_UNKNOWN , it means you can use any format compatible with the current display mode. (Generally, you should pick one that has the same color depth as the display mode.) In mixing mode, this field refers to the format for the back-end surface. In pass-through mode, it refers to the format requested by the video decoder. Often this is a YUV format, but it depends entirely on the output formats supported by the decoder. If the source file contains uncompressed video, there won t be a video decoder for that stream. In that case, the requested format might be the native video format ” probably an RGB type ” or else DirectShow might insert a color converter filter into the stream, in order to match the display mode. For example, if the source is 24-bit RGB and the display mode is 32-bit RGB, the source might be converted up to 32-bit RGB.

• The Pool field indicates which Direct3D memory pool to use.

• The MinBuffers field gives the minimum number of surfaces to create. Usually this value is the same as pNumBuffers , but it could be less. The actual number of surfaces that the allocator-presenter creates should be MinBuffers <= N <= pNumBuffers . If you always use N = pNumBuffers , you ll be safe.

• The szAspectRatio field gives the aspect ratio of the video, and szNativeSize gives the size of the original video source. These may not be the same value ” for example, DV video is natively 720 — 480 pixels, but should be displayed at a 4:3 aspect ratio (640 — 480).

Given all of these parameters, it can be tricky to allocate the surfaces correctly. Luckily, the VMR-9 provides a helper function, IVMRSurfaceAllocatorNotify9::AllocateSurfaceHelper , that does the allocation for us. This might seem kind of strange ” supposedly we re writing code to allocate Direct3D surfaces for the VMR, but then we call a method on the VMR that does the allocating. Even so, the InitializeDevice method gives us a lot of control over the allocation process, even if we use AllocateSurfaceHelper to create the surfaces.

For example, the Ducky sample uses IDirect3DDevice9::StretchRect to stretch the video onto the back-end surface. Depending on the video driver, there may be limitations as to which formats are supported for StretchRect operations. We can therefore use the InitializeDevice method to check the proposed format and verify whether the driver supports the format conversion. Also, drivers earlier than DirectX 9.0 cannot stretch a texture surface, so to be safe we can reject any requests for texture surfaces. The CAllocator class performs both of these tests before allocating the surfaces.

 STDMETHODIMP CAllocator::InitializeDevice(DWORD_PTR dwUserID,      VMR9AllocationInfo* pAllocInfo, DWORD* pNumBuffers)  {      CLock lock(&m_CritSec);      // Because we are using StretchRect, we do not want a texture surface.      if (pAllocInfo->dwFlags & VMR9AllocFlag_TextureSurface)      {          return E_FAIL;      }      // See if we can stretch this format to the back buffer.      D3DDISPLAYMODE dm;      ZeroMemory(&dm, sizeof(D3DDISPLAYMODE));      HRESULT hr = g_d3d->GetAdapterDisplayMode(D3DADAPTER_DEFAULT, &dm);      if (FAILED(hr))      {          return hr;      }      hr = g_d3d->CheckDeviceFormatConversion(D3DADAPTER_DEFAULT,          D3DDEVTYPE_HAL, pAllocInfo->Format, dm.Format);      if (FAILED(hr))      {          // This format cannot be converted, so we should reject it.          return VFW_E_INVALIDMEDIATYPE;      }      // Free existing surfaces and create a new array of surface pointers.      ReleaseSurfaces();      m_pSurf = new IDirect3DSurface9* [*pNumBuffers];      if (m_pSurf == NULL)      {          return E_OUTOFMEMORY;      }      // Let the VMR-9 do the surface allocation.      hr = m_pNotify->AllocateSurfaceHelper(pAllocInfo,          pNumBuffers, m_pSurf);      m_dwNumSurfaces = *pNumBuffers;      if (FAILED(hr))      {          ReleaseSurfaces();          *pNumBuffers = 0;      }      return hr;  } 

First, we hold the critical section with a CLock object. Then we check whether the VMR is requesting a texture surface, indicated by the VMR9AllocFlag_TextureSurface flag in the dwFlags field. If so, we reject the request by returning an error code. Next, we call IDirect3D9::CheckDeviceFormatConversion to test whether the proposed surface format can be converted to the display format.

Assuming we get past those hurdles, we create an array of size *pNumBuffers to hold the IDirect3DSurface9 pointers. Then we call AllocateSurfaceHelper on the VMR, passing in the allocation information plus the address of our array. If all goes well, we update pNumBuffers with the number of surfaces. The VMR checks this value when the method returns.

GetSurface

The VMR calls this method to retrieve a surface pointer. It will not call this method until the allocator-presenter has finished creating the surfaces.

 HRESULT GetSurface(      DWORD_PTR  dwUserID,      DWORD  SurfaceIndex,      DWORD  dwReserved  // Not used.      IDirect3DSurface9** ppSurface  ); 

The dwUserID parameter identifies the VMR instance, as before. The SurfaceIndex parameter is an index into the array of IDirect3DSurface9 pointers. The GetSurface method should copy the pointer at that index into the ppSurface parameter. The code for this method is straightforward.

 STDMETHODIMP CAllocator::GetSurface(DWORD_PTR dwUserID, DWORD SurfaceIndex,      DWORD SurfaceFlags, IDirect3DSurface9** ppSurface)  {      CLock lock(&m_CritSec);      if (!ppSurface)      {          return E_POINTER;      }      if (SurfaceIndex >= m_dwNumSurfaces)      {          return E_INVALIDARG;      }      assert (m_pSurf != NULL);      *ppSurface = m_pSurf[SurfaceIndex];      (*ppSurface)->AddRef();      return S_OK;  } 

Note that we have to AddRef the pointer before handing it to the VMR. The VMR will call Release when it s done with the pointer, as per the rules of COM.

TerminateDevice

The VMR calls this method when it s done using the allocator-presenter.

 HRESULT TerminateDevice(      DWORD_PTR  dwUserID  ); 

The only thing we need to do inside this method is release all the surfaces.

 STDMETHODIMP CAllocator::TerminateDevice(DWORD_PTR  dwID)  {      CLock lock(&m_CritSec);      ReleaseSurfaces();      return S_OK;  }  void CAllocator::ReleaseSurfaces()  {      // Private method. The caller should hold the critical section.      for (DWORD i = 0; i < m_dwNumSurfaces; i++)      {          SAFE_RELEASE(m_pSurf[i]);      }      SAFE_ARRAY_DELETE(m_pSurf);      m_dwNumSurfaces = 0;  } 

We noted earlier that the VMR calls the allocator-presenter from a separate thread. It also keeps a reference count on the allocator-presenter. Therefore, the application cannot decide when to release the surfaces, because the VMR might still be using them to decode and mix the video. Even if you stop the filter graph and release all of your DirectShow interface pointers, a race condition could still occur in which you release a surface that the VMR is still using, causing a run-time exception. Instead, the allocator-presenter must wait until TerminateDevice is called before releasing any surfaces.

Implementing the IVMRImagePresenter9 Methods

The IVMRImagePresenter9 interface controls how the allocator-presenter draws the video frames onto the back buffer. The interface has three methods, but two of them don t need any code. (Well, hardly any code.)

StartPresenting

This method is called right before the graph runs. It is just a notification from the VMR ” there s nothing in particular that the allocator-presenter needs to do in this method. But we do need to implement it, because it s part of a COM interface. Therefore, we simply return S_OK .

 STDMETHODIMP CAllocator::StartPresenting(DWORD_PTR dwUserID)  {      return S_OK;  } 

StopPresenting

This method is called right after the graph stops. Again, we don t have to do anything inside this method, so we just return S_OK .

 STDMETHODIMP CAllocator::StopPresenting(DWORD_PTR dwUserID)  {      return S_OK;  } 

PresentImage

The VMR calls this method whenever it has a new video frame for the allocator-presenter. (You should be aware that this method can be called before the first call to StartPresenting or after the last call to StopPresenting . For example, if the application pauses the filter graph while the graph is stopped , the VMR calls PresentImage to draw one video frame.)

 HRESULT PresentImage(      DWORD_PTR  dwUserID      VMR9PresentationInfo* pPresInfo  ); 

The pPresInfo parameter is a pointer to the VMR9PresentationInfo structure. This structure holds a pointer to a Direct3D surface containing the video image, plus some additional information such as the presentation time, the duration of the frame, and the correct aspect ratio. For the Ducky sample, we use the surface but ignore the other fields.

 STDMETHODIMP CAllocator::PresentImage(DWORD_PTR dwUserID,      VMR9PresentationInfo* pPresInfo)  {      CLock lock(&m_CritSec);      // Store the surface pointer in a temporary variable.      m_pVideoSurf = pPresInfo->lpSurf;      // Render one frame.      m_pGame->DoGameLoop();      // Clear the pointer.      m_pVideoSurf = NULL;      return S_OK;  } 

To draw the scene, the allocator-presenter calls the DoGameLoop method on its parent CVmrGame class. This method renders one frame in the 3-D scene, including processing the user input from the keyboard, updating the camera and model positions , and so on. Eventually, it results in a call to the CVmrGame::Render method, which does the rendering.

 HRESULT CVmrGame::Render()  {      CComPtr<IDirect3DSurface9> pRenderTarget;      m_pDevice->GetRenderTarget(0, &pRenderTarget);      // Clear the render target.      m_Device.Clear(0x00);      // Note: If the video completely overwrites the back buffer,      // you do not have to clear it. In this case, we need to clear      // the depth/stencil buffer so the duck gets rendered correctly.      HRESULT hr = m_pDevice->BeginScene();      if (FAILED(hr))      {          return hr;      }      // Stretch the video surface onto the back buffer.      hr = m_pDevice->StretchRect(m_pAlloc->m_pVideoSurf, NULL,          pRenderTarget, NULL, D3DTEXF_NONE);      if (m_pScene)      {          m_pScene->Render(m_pDevice);  // Render the scene meshes.      }      m_pDevice->EndScene();      hr = m_pDevice->Present(NULL, NULL, NULL, NULL);      return hr;  } 

In this method, we use IDirect3DDevice9::StretchRect to stretch the video surface onto the back buffer. The scene object s Render method then draws the duck mesh over the video. The IDirect3DDevice9::Present method presents the back buffer. And that s it! Next we look at a sample application that uses video on a texture surface.