Mixing Video Streams in Windowless Mode

Run the Mangler sample, which is included in the AVBook/bin directory. This application plays two video files at once, alpha-blending them in the same window. The application window also has several controls that can be used to change the video-mixing parameters (see Plate 5).

Note	Make sure that hardware acceleration is enabled in the DirectX control panel before you run the application. The VMR-9 requires hardware acceleration.

The two sliders labeled Stream 1 and Stream 2 change the width, height, and position of the two video streams, independently of each other. (Stream 1 is the dog, Stream 2 is the car.) The sliders labeled Video Source control the portion of the final image that appears in the application window. If you reduce the width of the video source, it has the effect of stretching the image horizontally, because a smaller portion of the image fills the same area of the window. Reducing the height causes the image to stretch vertically.

The check box labeled Switch Z Order flips the order of the images ” the stream on top moves to the bottom, and vice versa. At startup, the application sets the alpha value of both video streams to 0.8 (where 0.0 is translucent and 1.0 is opaque ), so that the video on the bottom layer shows through the video on the top layer.

The check box labeled Preserve Aspect Ratio controls whether the VMR-9 letterboxes the video. If this box is checked, the VMR-9 maintains the correct aspect ratio. If the box is unchecked, the image is stretched to fill the entire destination area.

Most of the code in the Mangler application is Microsoft Win32 code for the UI, which we can ignore. We ll jump right into the DirectShow code, starting with building the filter graph.

Building the Filter Graph and Configuring the VMR

In our previous DirectShow application, we called RenderFile , which built the entire filter graph all at once. However, as we noted earlier, RenderFile does not select the VMR-9. Therefore, we need to place the VMR-9 into the graph ourselves and tell the Filter Graph Manager to use it. This also gives us a chance to configure the VMR for windowless mode, which must be done before the VMR is connected to any other filters.

The following code adds the VMR to the filter graph and initializes the VMR for windowless mode.

 CComPtr<IGraphBuilder> m_pGraph;  // Filter Graph Manager.  CComPtr<IBaseFilter> m_pVMR;      // VMR-9 filter.  // Create the Filter Graph Manager.  hr = m_pGraph.CoCreateInstance(CLSID_FilterGraph);  // Add the VMR-9 filter to the graph.  hr = AddFilterByCLSID(m_pGraph, CLSID_VideoMixingRenderer9, &m_pVMR);  // Configure the VMR for windowless mode before we connect  // any video streams to it.  CComQIPtr<IVMRFilterConfig9> pConfig(m_pVMR);  hr = pConfig->SetRenderingMode(VMR9Mode_Windowless);  // Set the window where we want the VMR to paint the video.  CComQIPtr<IVMRWindowlessControl9> pWC(m_pVMR);  pWC->SetVideoClippingWindow(hVidWin);

To make the code more readable, some error-checking has been left out. Also, some code has been re-arranged to make the procedures more linear. The actual Mangler source code is organized into several C++ classes.

As in the previous application, the first step is to create the Filter Graph Manager. Then we create the VMR-9 and add it to the graph by calling an application-defined helper function, AddFilterByCLISD . We ll describe this function in a moment.

To set up windowless mode, query the VMR filter for the IVMRFilterConfig9 interface and call IVMRFilterConfig9::SetRenderingMode with the flag VMR9Mode_Windowless . After windowless mode is established, we need to tell the VMR about the application window where it should draw the video. Query the VMR for the IVMRWindowlessControl9 interface, and call IVMRWindowlessControl9::SetVideoClippingWindow with the handle of the window. In the Mangler application, the video-clipping window is a static rectangle control that belongs to the dialog box.

The order of these method calls is important, because the VMR does not expose the IVMRWindowlessControl9 interface until you specify windowless mode. Before you do that, calling QueryInterface for IVMRWindowlessControl9 will fail.

Although it is not required in the Mangler application, at this point you may want to set the number of video streams. In both windowed mode and windowless mode, the VMR-9 defaults to four input pins, meaning you can render and mix four video streams simultaneously . If you want to mix more than that, call IVMRFilterConfig9::SetNumberOfStreams and specify the number of streams that you want, up to 16. If you plan to mix fewer than four streams, you can leave the default value ” there won t be a performance penalty for having extra unconnected pins. When we get to renderless mode in the next chapter, we ll see that video mixing works somewhat differently in that mode.

(Unlike the VMR-9, the VMR-7 creates only one input pin by default. If you are using the VMR-7 to mix multiple video streams, you must call IVMRFilterConfig::SetNumberOfStreams .)

The code for the AddFilterByCLISD function is shown in the following code. It calls CoCreateInstance to create the filter, and then calls IGraphBuilder::AddFilter on the Filter Graph Manager to insert the filter into the graph. (Creating a filter does not automatically add it to the graph.) The function returns the filter s IBaseFilter interface.

AddFilterByCLSID

 // Function to add a filter to the filter graph.  HRESULT AddFilterByCLSID(      IGraphBuilder *pGraph,  // Pointer to the Filter Graph Manager.      const GUID& clsid,      // CLSID of the filter to create.      IBaseFilter **ppF)      // Receives a pointer to the filter.  {      if (!pGraph  ! ppF)      {          return E_POINTER;      }      *ppF = 0;      CComPtr<IBaseFilter> pF;      HRESULT hr = pF.CoCreateInstance(clsid);      if (SUCCEEDED(hr))      {          hr = pGraph->AddFilter(pF, NULL);          if (SUCCEEDED(hr))          {              // Return the IBaseFilter pointer to the caller.              // The caller must release it.              *ppF = pF.Detach();          }      }      return hr;  }

Now that the VMR is set up for windowless mode, we can load the video files and render the streams.

 WCHAR *wsFileName; // File name.  // Initialize wsFileName with the file name (not shown).  // Add a source filter for the source file.  CComPtr<IBaseFilter> pSource; // Source filter.  hr = m_pGraph->AddSourceFilter(wsFileName, L"Source Filter",      &pSource);  // Collect all the source filter's output pins into a list.  PinList pins;  hr = GetPinList(pSource, PINDIR_OUTPUT, pins);  // Try to render each pin, using only the renderers in the graph.  CComQIPtr<IFilterGraph2> pGraph2(m_pGraph);  for (PinIterator iter = pins.begin(); iter != pins.end(); ++iter)  {      // If any pin succeeds, treat it as a global success. Some pins      // may output non-video media types, so it's OK if they fail.      HRESULT hrTmp;      hrTmp = pGraph2->RenderEx(*iter,          AM_RENDEREX_RENDERTOEXISTINGRENDERERS, 0);      if (SUCCEEDED(hrTmp))      {          hr = S_OK;      }  }  // Release wsFileName (not shown).

We start by calling IGraphBuilder::AddSourceFilter to add a source filter for the video file. The first parameter is the file name, and the second parameter is a name for the filter. (The filter name is useful for debugging but otherwise won t be used.) The AddSourceFilter method returns a pointer to the source filter s IBaseFilter interface in the third parameter.

Next we hook up the source filter to the video renderer. For reasons that will become clear in a moment, we need to get a list of the source filter s output pins. The GetPinList function finds all of the pins on a filter that match a specified direction ” in this case, the output pins ” and collects them into an STL list object. We ll describe the code for GetPinList later in this section.

Finally, we iterate through the list of pins and call IFilterGraph2::RenderEx with each pin. The RenderEx method takes a pointer to an output pin s IPin interface. The AM_RENDEREX_RENDERTOEXISTINGRENDERERS flag tells the Filter Graph Manager to connect the pin to a renderer filter, using only the renderer filters that are already in the graph. DirectShow may add other intermediate filters to the graph, such as splitters or decoders, but it will not add any new renderers. This prevents DirectShow from using one of the other video renderer filters that we mentioned earlier. When the for loop exits, the source filter is connected to the VMR-9 through some number of intermediate filters. The resulting filter graph is shown in Figure 9.5.

Figure 9.5: AVI file playback using the VMR-9.

The source filter shown in this diagram has only one output pin, and this will be true in general for AVI files. So why go through the trouble of building a list of pins? The reason is that some source filters have multiple output pins ” for example, the source filter used to render Windows Media files creates one output pin per stream in the file.

Also, note that we did not put an audio renderer into the graph beforehand. Therefore, even if the video file has an audio stream, the audio won t be rendered, because we specified the AM_RENDEREX_RENDERTOEXISTINGRENDERERS flag. If you want to render the audio, you need to add the audio renderer to the graph first, as we did with the VMR. The CLSID for the audio renderer filter is CLSID_DSoundRender ” so named because it uses DirectSound to play the audio. You can easily modify the code in the Mangler application to include audio playback.

Here is the code for the GetPinList function.

GetPinList

 typedef CComPtr<IPin> PinPtr;  typedef std::list<PinPtr> PinList;  typedef PinList::iterator PinIterator;  HRESULT GetPinList(      IBaseFilter *pFilter,    // Pointer to the filter.      PIN_DIRECTION Direction, // The pin direction to search for.      PinList& list            // A list to collect the pins.  )  {      if (!pFilter)      {          return E_POINTER;      }      CComPtr<IEnumPins> pEnum;      HRESULT hr = pFilter->EnumPins(&pEnum);      if (FAILED(hr))      {          return hr;      }      CComPtr<IPin> pPin;      while (S_OK == pEnum->Next(1, &pPin, NULL))      {          PIN_DIRECTION ThisDirection;          hr = pPin->QueryDirection(&ThisDirection);          if (FAILED(hr))          {              return hr;          }          if (ThisDirection == Direction)          {              list.push_back(pPin);          }          pPin.Release();      }      return S_OK;  }

To enumerate a filter s pins, we use the IEnumPins interface, which we get by calling IBaseFilter::EnumPins on the filter. To iterate through all of the pins, call IEnumPins::Next until the return value is something other than S_OK . (Don t use the SUCCEEDED macro, because in this case S_FALSE does not mean the same thing as S_OK .) The Next method returns an IPin pointer for the pin.

For each pin, the GetPinList function calls IPin::QueryDirection to find the direction of the pin, either input or output. If the pin direction matches the Direction parameter, the IPin pointer is stored in the list. The STL push_back function makes a copy of the CComPtr object, which calls AddRef on the raw IPin pointer, so the reference count remains correct.

Setting the Video Mixing Preferences

Once the graph is built, we can configure how it mixes the video streams. The VMR gives you a lot of control over the mixing process. Some mixing properties are global to the VMR, and some are applied individually to each stream. Figure 9.6 shows the relationships among those settings that affect the positions of the video streams.

Figure 9.6: Video-mixing preferences in the VMR-9.

Output Rectangles

Each stream has an output rectangle . The output rectangles control how the video streams are positioned relative to each other.

Internally, the VMR creates a Direct3D surface where is mixes the videos . This surface is called the back-end surface . The size of this surface in each dimension is equal to the maximum size of all the source images in that dimension. For example, if the first video stream is 200 — 400 pixels, and the second video stream is 400 — 200 pixels, the back-end surface will be 400 — 400 pixels.

By default, the VMR positions each video in the center of the surface, so that it fills the entire surface along one dimension, while preserving the video aspect ratio in the other dimension. You can re-position a stream by changing the output rectangle for that stream. The coordinates of the output rectangle are defined so that the upper-left corner of the back-end surface has coordinates (0.0, 0.0) and the lower-right corner of the surface has coordinates (1.0, 1.0).

The output rectangle is measured relative to these coordinates, so the default position of each stream is {0.0, 0.0, 1.0, 1.0}. To place the image in the upper-left quadrant, you would set the output rectangle to {0.0, 0.0, 0.5, 0.5}. This also has the effect of shrinking the video to fit the smaller rectangle.

You are not limited to the range [0.0 ... 1.0]. Any part of the image that falls outside of this range is simply clipped, and does not get rendered. You can use this feature to create fly-in effects . For example, you could set the initial output rectangle to {-1.0, -1.0, 0.0, 0.0}, which is outside the visible area, and gradually update the rectangle to {0.0, 0.0, 1.0, 1.0} so that the video slides diagonally into the application window.

To set the output rectangle for a stream, call IVMRMixerControl::SetOutputRect .

 DWORD deStreamId = 0; // First video stream.  VMR9NormalizedRect rect;  // Set the rectangle boundaries to {0,0,1,1}.  rect.left = rect.top = 0.0f;  rect.right = rect.bottom = 1.0f;  m_pMixer->SetOutputRect(dwStreamId, &rect);

The first parameter ( dwStreamId ) specifies the video stream, indexed from zero. The index number is determined by the order in which the VMR pins are connected. The first pin to be connected is stream 0, the second is stream 1, and so on. In the Mangler application, this corresponds to the order in which the application s RenderVideoStream function is called.

Alpha

You can apply a per-stream alpha value to make a video stream transparent. The alpha can range from 0.0, which is completely transparent, to 1.0, which is completely opaque. Set the alpha value for a stream with the IVMRMixerControl::SetAlpha method.

 m_pMixer->SetAlpha(dwStreamId, 0.5f); // Semi-transparent (50% alpha).

The value is applied to the entire image area for that stream.

Background Color

In the places where the video images do not cover the back-end surface, the surface is filled with a solid background color. This color is also visible behind the image, if the image is transparent. Set the background color with the IVMRMixerControl::SetBackgroundClr method, which takes a COLORREF value. The default color is black.

 m_pMixer->SetBackgroundClr(RGB(0xFF, 0xFF, 0xC0)); // Pale yellow.

Z-Order

The z-order specifies the order in which the streams are composited onto the back-end surface. Higher z-orders are farther back ” in other words, the highest z-order is rendered first. If the stream s alpha is set to 1.0, the image obscures anything below it. Otherwise, the image is alpha-blended with layer below it. Set the z-order with the IVMRMixerControl::SetZOrder method.

 m_pMixer->SetZOrder(dwStreamId, 0); // Move this stream to the front.

In the Mangler application, the Switch Z Order check box sets the z-order of stream number 1, switching it between the values 0 and 2. Stream number 0 is kept at z-order 1.

Source and Destination Rectangles

The source rectangle is a subrectangle within the back-end surface, and the destination rectangle is a subrectangle within the application window s client area. After the VMR composites the video streams to the back-end surface, it stretches the surface s source rectangle onto the window s destination rectangle. Thus if you reduce the destination rectangle, the video occupies a smaller portion of the application window. If you reduce the source rectangle, the VMR stretches a smaller portion of the composited image into the destination rectangle, with the effect of magnifying a section of the composited image. If the source rectangle is equal to the entire back-end surface, the entire image is displayed in the destination rectangle.

To set the source and destination rectangles, call IVMRWindowlessControl9::SetVideoPosition . To find the actual size of the back-end surface, call IVMRWindowlessControl9::GetNativeVideoSize . Always set the value of the source and destination rectangles before playback starts, or the video will not be visible, because both rectangles default to {0, 0, 0, 0}. The following code sets the source rectangle equal to the entire back-end surface and the destination rectangle equal to the entire window client area.

 RECT rcSource;  RECT rcDest;  // Find the size of the composited video image.  ZeroMemory(&rcSource, sizeof(RECT));  m_pWC->GetNativeVideoSize(      &rcSource.right,  // Width.      &rcSource.bottom, // Height.      NULL, NULL        // Aspect ratio X and Y (optional).      );  // Use the entire client area for the destination rectangle.  GetClientArea(hwnd, &rcDest);  m_pWC->SetVideoPosition(&rcSource, &m_rcDest);

Letterbox Mode

By default, the VMR does not preserve the aspect ratio of the image when it stretches the source rectangle to the destination rectangle. To preserve the aspect ratio, call IVMRWindowlessControl9::SetAspectRatioMode with the flag VMR9ARMode_LetterBox .

 m_pWC->SetAspectRatioMode(VMR9ARMode_LetterBox);

Note that letterbox mode controls how the VMR stretches the back-end surface to the window, not how it draws the video streams onto the back-end surface.

Border Color

If you enable letterbox mode, you can define the color of the border where the video is letterboxed. To do so, call IVMRWindowlessControl9::SetBorderColor with a COLORREF value. The Mangler application sets the border color to orange, which is quite ugly, but makes the effect very obvious.

 m_pWC->SetBorderColor(RGB(0xFF, 0x80, 0x00)); // Orange.

The border color is not the same as the background color mentioned earlier. The background color is used to fill the areas of the back-end surface where the video does not appear, while the border color is used to fill the letterboxed edges of the client window, in letterbox mode only. The default border color is black.

Miscellaneous VMR Features

The VMR has some additional features that are not demonstrated in the Mangler application:

You can alpha-blend a static bitmap onto the video. For example, you could use this to put a logo onto the video window. Call IVMRMixerBitmap9::SetAlphaBitmap .
You can grab a copy of the most recent video frame, in the form of a device-independent bitmap (DIB). Call IVMRWindowlessControl::GetCurrentImage .
If the graphics hardware supports it, the VMR can perform hardware-accelerated deinterlacing and image adjustment. This feature is described in detail in Chapter 12.

Running the Filter Graph

With our initial mixing preferences established, we are ready to start video playback by calling IMediaControl::Run on the Filter Graph Manager.

 m_pControl->Run();

While the graph is running, each decoder in the graph delivers video frames to the VMR. The VMR runs a worker thread that picks up the decoded frames , mixes them onto the back-end surface, and stretches the composited image onto the application window. The application does not have to take any actions while this is happening ” it is all managed automatically by the VMR. However, because the VMR does not own the video window, the application must inform it whenever certain events occur:

Repaints. When the application receives a WM_PAINT message, call IVMRWindowlessControl::RepaintVideo to inform the VMR to repaint the video window. Note that you do not have to call RepaintVideo just to get the VMR to draw each frame while the video plays; that happens automatically. The purpose of calling RepaintVideo is to make sure the VMR repaints the most recent frame after the application window is invalidated. This could happen between frames while the graph is running, or while the graph is paused .
Window size changes. When the application receives a WM_SIZE message, you may need to recalculate the source and destination rectangles and call SetVideoPosition again. You can skip this call if you don t need to adjust the video position after the resize ” it will depend on your application. The Mangler application uses a non-resizable window, so it s not a issue.
Display changes. When the application receives a WM_DISPLAYCHANGE message, call IVMRWindowlessControl::DisplayModeChanged . This tells the VMR to take whatever actions are needed to respond to the display change.

As the Mangler application demonstrates , you can change any of the mixer settings while the video is playing, and thereby achieve interesting animation effects. The VMR automatically applies the new settings to the next frame that it renders .

Handling Filter Graph Events

DirectShow has an event mechanism that lets the Filter Graph Manager inform your application when interesting things happen inside the graph ” for example, when playback reaches the end of the file. In our first application, back at the start of this chapter, we waited for the end of the file by calling WaitForCompletion with a timeout value of INFINITE. That s not a good practice for a real-world application, because you don t want your application to block while the file is playing. The Mangler application shows how to perform proper event handling.

At startup, we ask the Filter Graph Manager to notify our application whenever there is a new event. It will do so by posting a message to the application s message loop. We define a private Windows message for this purpose.

 static const long WM_GRAPH_EVENT = WM_APP + 1;

To set up the notification, call IMediaEventEx::SetNotifyWindow .

 m_pEvent->SetNotifyWindow((OAHWND)hwnd, WM_GRAPH_EVENT, 0);

The first argument is the window handle, cast to an OAHWND type. The second argument is the Windows message, and the third argument is an optional value that will be returned to the application in the lParam parameter of the message. The lParam value is not used in the Mangler application, but you could use it to track events from multiple filter graphs. Whenever the Filter Graph Manager queues a new event, the designated window receives the WM_GRAPH_EVENT message (defined as WM_APP + 1 ). In your message loop, respond to the message by calling IMediaEvent::GetEvent .

 LRESULT CALLBACK WindowProc(HWND hwnd, UINT msg,      WPARAM wParam, LPARAM lParam)  {      long EventCode, param1, param2; // Event parameters.      switch (msg)      {      case WM_GRAPH_EVENT:          // Loop until no messages are left in the queue.          while (SUCCEEDED(hr = m_pEvent->GetEvent(&EventCode, &param1,                 &param2, 0)))          {              // Decide what to do based on the event code.              switch (EventCode)              {              case EC_COMPLETE: // End of file.                   // Handle the event (not shown).                  break;              }          }          // Release any resources allocated for the event parameters.          m_pEvent->FreeEventParams(EventCode, param1, param2);      break;      // Handle other Windows messages as usual (not shown).      }  }

The GetEvent method returns an event code and two event parameters. The event code defines what kind of event happened . For example, the end of the file is signalled by the EC_COMPLETE event. The event parameters are similar to the lParam and wParam values in a Windows message ” they contain additional information whose meaning depends on the event type. Sometimes you may need to cast the event parameters to another data type, such as a COM pointer. The DirectShow SDK documentation has a complete list of all the events and their parameters. After you get the event and respond to it, always call IMediaEvent::FreeEventParams . This tells the Filter Graph Manager to release any memory it may have allocated for the event parameters.

You may have noticed in the previous code example that we called GetEvent and FreeEventParams inside a while loop, which repeats for as long as GetEvent succeeds. This is done because several events might be queued by the time the application receives the WM_GRAPH_EVENT message and responds to it. Using a loop ensures that we get all the pending events.

Seeking to the Start of the File

When the Mangler application receives the EC_COMPLETE event, it stops the graph and seeks to the start of the file.

 m_pControl->Stop();  LONGLONG rtPosition = 0;  m_pSeek->SetPositions(     &rtPosition, AM_SEEKING_AbsolutePositioning,  // Current position.     NULL, AM_SEEKING_NoPositioning  // Stop position (no change).  );  m_pControl->Run();

Seeking is performed by calling IMediaSeeking::SetPositions on the Filter Graph Manager. The first parameter is the time to seek to. The units are 100 nanoseconds (10 ^-7 seconds), so 1 second is 10,000,000 units. The AM_SEEKING_AbsolutePositioning flag means the seek time is expressed relative to the start of the file, as opposed to being relative to the current playback position. The third parameter specifies a stop time. In this case, NULL is used because we do not want to modify the stop time, as indicated by the AM_SEEKING_NoPositioning flag.

When the VMR mixes multiple video streams, it has limited support for seeking. You can stop the graph and seek all the streams back to the beginning, but you cannot seek around in the file during playback. In Chapter 12, we ll show a way to get around this limitation by using several VMR instances in the same application.