Working with the VMR: A Picture-in-Picture Application

Working with the VMR: A Picture-in-Picture Application

Now that we ve done some work with broadcast TV signals, we need to take a look at how DirectShow can be used to create one of the fancy special effects available on expensive TVs: picture-in-picture. This effect takes two separate video sources (in the case of a television, two separate TV tuners) and mixes them together into the same display window, placing the thumbnail of a video image over the lower-right corner of the primary image. Picture-in-picture TVs allow you to switch between the big picture and the thumbnail, so you can monitor one channel, even as you watch another. That s great for folks who want to watch one ball game while keeping an eye on another or are channel surfing while waiting for a commercial break to end.

We could build a full picture-in-picture TV application in DirectShow, but that would require that two TV tuners be connected to the user s computer. Although this is no big deal technically, it is very rare. Instead, the Pip9 application takes two video files (which you select from your hard disk) and streams them into the same application window, creating a picture-in-picture view, as shown in Figure 9-2.

Figure 9-2. A demonstration of picture-in-picture using two AVI files

An Effects menu allows you to swap the two images, animate the swap, mirror either stream (flip it right-to-left), flip either stream up-to-down, and make ProcAmp (Process Amplification) adjustments to the brightness, hue, and saturation values on the primary stream. The Subpicture menu gives you rough control over the size and placement of the picture-in-picture thumbnail, allowing you to shrink or increase the size of the inset video, placing it into any of the four quadrants or the center of the video window. When both clips reach the end of their running time, they re rewound and begin to play again, endlessly.

Rather than a detailed line-by-line exploration of the Pip9 code, our examination will focus on four principle areas: the creation of the filter graph with the VMR, the code that creates the picture-in-picture effect, another piece of code that animates the swapping of the streams, and the dialog box that manages a virtual processing amplifier. In these examples (and the rest of the Pip9 application, which you can peruse at your leisure), you ll learn how to make the VMR work in your own DirectShow applications.

Initializing the VMR

Once the user has selected two valid movie files by selecting the Open Files menu item from the File menu, control is transferred to the InitializeWindowlessVMR function.

HRESULT InitializeWindowlessVMR(IBaseFilter **ppVmr9) { IBaseFilter *pVmr=NULL; if (!ppVmr9) return E_POINTER; *ppVmr9 = NULL; // Create the VMR and add it to the filter graph. HRESULT hr = CoCreateInstance(CLSID_VideoMixingRenderer9, NULL, CLSCTX_INPROC, IID_IBaseFilter, (void**)&pVmr); if (SUCCEEDED(hr)) { hr = pGB->AddFilter(pVmr, L"Video Mixing Renderer 9 ); if (SUCCEEDED(hr)) { // Set the rendering mode and number of streams. CComPtr <IVMRFilterConfig9> pConfig; JIF(pVmr->QueryInterface(IID_IVMRFilterConfig9, (void**)&pConfig)); JIF(pConfig->SetRenderingMode(VMR9Mode_Windowless)); JIF(pConfig->SetNumberOfStreams(2)); // Set the bounding window and border for the video. JIF(pVmr->QueryInterface(IID_IVMRWindowlessControl9, (void**)&pWC)); JIF(pWC->SetVideoClippingWindow(ghApp)); JIF(pWC->SetBorderColor(RGB(0,0,0))); // Black border // Get the mixer control interface for manipulation of video // stream output rectangles and alpha values. JIF(pVmr->QueryInterface(IID_IVMRMixerControl9, (void**)&pMix)); } // Don't release the pVmr interface because we are copying it into // the caller's ppVmr9 pointer. *ppVmr9 = pVmr; } return hr; }

This function is called once a Capture Graph Builder interface, pGB, exists to receive the instantiated VMR, which is created with a call to CoCreateInstance. The filter is immediately added to the filter graph. At this point, the VMR initialization occurs. A call to QueryInterface returns the IVMRFilterConfig9 interface, which is put into windowless mode. The number of VMR input pins is set with a call to the IVMRFilterConfig9 interface method SetNumberOfStreams.

The function then acquires the IVMRWindowlessControl9 interface, which it uses to set the display characteristics of the VMR. The video clipping window the target window for the rendered video stream is set with a method call to SetVideoClippingWindow. In your own VMR applications, that call should be supplied with a handle to a window owned by your application. The border color of the rendered output (whatever might not be filled with the video imagery) is set to black with a call to SetBorderColor. Last the function acquires the IVMRMixerControl9 interface, which will be used by other functions in Pip9 to manipulate the mixer in real time.

The video streams are brought into the filter graph in the utility function RenderFileToVMR9, which uses the IFilterGraph2 method RenderEx to build the filter graph components that connect a disk-based movie file to the VMR. The RenderEx method allows you to specify that a renderer that already exists in the filter graph (in this case, the VMR) must be used to render the output pin of another filter a File Source filter pointing to the movie. The value AM_RENDEREX_RENDERTOEXISTINGRENDERERS is passed as a parameter in the RenderEx call. (We need to be explicit about our choice of renderer because the VMR 9 is never the default renderer. In some cases, such as if the user s system has an old graphics card, the Filter Graph Manager might fall back to the old Video Renderer, which is not what you d want at all.) Both movie files are added in successive calls to RenderFileToVMR9 and connected to the first two available input pins on the VMR, which are now identifiable as stream 0 and stream 1.

Pip9 is not a very complex VMR application; only two streams are being mixed together. Even so, all the basics of VMR initialization and setup are presented in InitializeWindowlessVMR. A bit further along in application execution, the InitVideoWindow function is invoked. It uses the IVMRWindowlessControl9 interface to tie the output from the VMR to the application s main window.

HRESULT InitVideoWindow(int nMultiplier, int nDivider) { LONG lHeight, lWidth; HRESULT hr = S_OK; if (!pWC) return S_OK; // Read the default video size. hr = pWC->GetNativeVideoSize(&lWidth, &lHeight, NULL, NULL); if (hr == E_NOINTERFACE) return S_OK; // Account for requests of normal, half, or double size. lWidth = MulDiv(lWidth, nMultiplier, nDivider); lHeight = MulDiv(lHeight, nMultiplier, nDivider); int nTitleHeight = GetSystemMetrics(SM_CYCAPTION); int nBorderWidth = GetSystemMetrics(SM_CXBORDER); int nBorderHeight = GetSystemMetrics(SM_CYBORDER); // Account for size of title bar and borders for exact match // of window client area to default video size. SetWindowPos(ghApp, NULL, 0, 0, lWidth + 2*nBorderWidth, lHeight + nTitleHeight + 2*nBorderHeight, SWP_NOMOVE SWP_NOOWNERZORDER); GetClientRect(ghApp, &g_rcDest); hr = pWC->SetVideoPosition(NULL, &g_rcDest); return hr; }

The application learns the native width and height of the input streams through a call to GetNativeVideoSize. The output size is a function of the input streams. After InitVideoWindow performs some math to learn the true dimensions of the display rectangle, minus border and title bar dimensions, it assigns that value to the VMR through the IVMRWindowlessControl9 method SetVideo Position. Now when the Filter Graph Manager receives the Run command, VMR output will flow into the designated rectangle within the application s window.

When using the VMR in windowless mode, it s necessary to notify the VMR when your application receives certain Windows messages. When you receive a WM_PAINT message, call IVMRWindowlessControl::RepaintVideo. When you receive a WM_DISPLAYCHANGE message, call IVMRWindowlessControl::DisplayModeChanged. When you receive a WM_SIZE message, recalculate the position of the video and call IVMRWindowlessControl::SetVideoPosition if necessary.

Programming the VMR for Picture-in-Picture Display

To program picture-in-picture display using the VMR, all that s required is to set a display rectangle for the thumbnail picture, using a call to the IVMRMixerControl9 method SetOutputRect. If the specified rectangle has a different dimension than the destination rectangle (which is covering the entire available area inside the window), the video output from the VMR will appear to be inset within the destination rectangle. The calculation generating the coordinates for an inset rectangle within a VMR window is performed with a normalized rectangle in a data structure known as VMR9NormalizedRect. The normalized rectangle spans the coordinates from (0, 0), the upper-left corner, through (1, 1), the lower-right corner. This area is known as the composition space (shown in Figure 9-3) for the VMR.

Figure 9-3. Composition space, which creates both on-screen and off-screen drawing areas

If desired, you can perform off-screen operations with the VMR by choosing coordinates that are not within the on-screen rectangle. For example, a VMR rectangle with an upper-left coordinate of (-1, 0) and a lower-right coordinate of (-0.01, 1) would not be visible because it s not within the on-screen area of the composition space. Clever use of the VMR s composition space could allow you to construct animations that cause off-screen elements (rendered in the VMR) to fly on screen or off screen programmatically.

Here s a very simple use of the SetOutputRect method in a function that s called repeatedly in the Pip9 application to set the visible areas of both video streams supplied to the VMR:

HRESULT UpdatePinPos(int nStreamID) { HRESULT hr=S_OK; // Get a pointer to the selected stream's information. STRM_PARAM* p = &strParam[nStreamID]; // Set the left, right, top, and bottom coordinates. VMR9NormalizedRect r = {p->xPos, p->yPos, p->xPos + p->xSize, p->yPos + p->ySize}; // If mirrored, swap the left/right coordinates // in the destination rectangle. if (strParam[nStreamID].bMirrored) { float fLeft = strParam[nStreamID].xPos; float fRight = strParam[nStreamID].xPos + strParam[nStreamID].xSize; r.left = fRight; r.right = fLeft; } // If flipped, swap the top/bottom coordinates // in the destination rectangle. if (strParam[nStreamID].bFlipped) { float fTop = strParam[nStreamID].yPos; float fBottom = strParam[nStreamID].yPos + strParam[nStreamID].ySize; r.top = fBottom; r.bottom = fTop; } // Update the destination rectangle for the selected stream. if(pMix) hr = pMix->SetOutputRect(nStreamID, &r); return hr; }

UpdatePinPos is called every time there s any change to the on-screen position of either video stream. Using a VMR9NormalizedRect structure (whose fields are manipulated elsewhere in the code), the VMR is configured with a SetOutputRect command to change the visible area of the stream. In the case where Animated Stream Swap is selected from the Effects menu, a timer periodically changes the coordinates of both streams to different composition space values. UpdatePinPos is called after each coordinate change, and the stream output rectangles reflect that change in an animation that shows the streams swapping positions.

Two other, very simple functions keep the alpha (blending) and Z order of the streams updated appropriately.

HRESULT UpdatePinAlpha(int nStreamID) { HRESULT hr=S_OK; // Get a pointer to the selected stream's information. STRM_PARAM* p = &strParam[nStreamID]; // Update the alpha value for the selected stream. if(pMix) hr = pMix->SetAlpha(nStreamID, p->fAlpha); return hr; } HRESULT UpdatePinZOrder(int nStreamID, DWORD dwZOrder) { HRESULT hr=S_OK; // Update the Z order for the selected stream. if(pMix) hr = pMix->SetZOrder(nStreamID, dwZOrder); return hr; }

The function UpdatePinAlpha controls the alpha value, the transparency of the input stream when rendered into the output stream. The possible values range from 0 (completely transparent, hence invisible) to 1.0 (completely opaque). In the Pip9 application, the thumbnail stream is rendered with an alpha value of 0.6, or 60 percent opaque. This opacity means that you can see through the thumbnail stream to the stream behind it. The alpha value for a stream is set with a call to the IVMRMixerControl9 method SetAlpha.

The function UpdatePinZOrder manipulates the Z value for a given stream. As stated previously, a Z value of 0 indicates the frontmost position, while higher Z values can be thought of as receding into the display. The lowest Z value is rendered in front of all other streams. For example, the thumbnail stream in Pip9 has a lower Z value than the full-screen stream and is rendered in the foreground. The Z value for a stream is set with a call to the IVMRMixerControl9 method SetZOrder. Although it seems as though more programming should be required to create such a range of video effects picture-in-picture, animation, and transparency the VMR takes care of all these effects in just a few calls.

Programming the VMR ProcAmp

ProcAmp is a device that evolved out of necessity within the world of video hardware. In its physical manifestation, a ProcAmp is a black box that adjusts an incoming video signal so that its brightness, contrast, hue, and saturation values match those of a corresponding piece of downstream video gear. ProcAmps are often necessary when dealing with video signals because equipment produced by different manufacturers (and at different times) has differing requirements for signal strength, modulation, and so forth.

The DirectShow VMR has its own version of a ProcAmp. Rather than using an external analog device, the VMR ProcAmp manipulates the graphics hardware in the computer s display system to produce visually analogous effects if the graphics card driver supports these kinds of effects. (As of this writing, only the very newest graphics cards supported ProcAmp effects.) Within the Effects menu of Pin, there s an item named ProcAmp Adjustments that brings up a dialog box with four sliders for control of brightness, hue, saturation, and contrast, as shown in Figure 9-4.

Figure 9-4. The Pip9 dialog box that allows you to adjust the VMR ProcAmp for each stream

Here s the Windows event-handler code that handles all events from the ProcAmp dialog box. Within it, you ll see the code that initializes the slider controls using VMR ProcAmp and code that changes those values as the sliders are moved.

LRESULT CALLBACK ProcAmpAdjustProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam) { HRESULT hr; DWORD streamNum = 0; switch (message) { case WM_INITDIALOG: // Set up the slider ranges if (pMix) { DWORD streamNum = 0; VMR9ProcAmpControl myProcAmpControl; myProcAmpControl.dwSize = sizeof(VMR9ProcAmpControl); myProcAmpControl.dwFlags = 0; hr = pMix->GetProcAmpControl(streamNum, &myProcAmpControl); if (SUCCEEDED(hr)) { // For each ProcAmp control, check if it's supported. // If so, get the range and set the slider value. // Otherwise, disable the control. DWORD prop; float *pVal; for (prop = ProcAmpControl9_Brightness, pVal = &myProcAmpControl.Brightness; prop < ProcAmpControl9_Mask; prop <<= 1, pVal++) { HWND hSlider = GetDlgItem(hWnd, IDC_SLIDER_BASE + prop); // Is this property supported? if (myProcAmpControl.dwFlags & prop) { SendMessage(hSlider, TBM_SETRANGE, TRUE, MAKELONG(0, 100)); // Let's get the range for this property // to start with. VMR9ProcAmpControlRange myProcAmpControlRange; myProcAmpControlRange.dwSize = sizeof(VMR9ProcAmpControlRange); myProcAmpControlRange.dwProperty = (VMR9ProcAmpControlFlags)prop; hr = pMix->GetProcAmpControlRange(streamNum, &myProcAmpControlRange); if SUCCEEDED(hr) { // We have value and ranges, // so let's normalize for a 1-100 scale. float totalRange = myProcAmpControlRange.MaxValue - myProcAmpControlRange.MinValue; float absoluteValue = *pVal - myProcAmpControlRange.MinValue; double ratio = (absoluteValue/totalRange) * 100.0; long radical = (long)ratio; // Now we've got to set the slider to // reflect its value in the total range. SendMessage(hSlider, TBM_SETPOS, TRUE, radical); } } else { // Graphics driver does not support this // property. Disable the slider. EnableWindow(hSlider, FALSE); } } // for } else { // We failed the GetProcAmpControl call. // Disable all sliders. EnableWindow(GetDlgItem(hWnd, IDC_SLIDER_BRIGHTNESS), FALSE); EnableWindow(GetDlgItem(hWnd, IDC_SLIDER_CONTRAST), FALSE); EnableWindow(GetDlgItem(hWnd, IDC_SLIDER_HUE), FALSE); EnableWindow(GetDlgItem(hWnd, IDC_SLIDER_SATURATION), FALSE); } } return TRUE; case WM_COMMAND: switch (wParam) { case IDC_PROCAMPCLOSE: EndDialog(hWnd, TRUE); break; } break; case WM_HSCROLL: // Figure out which slider did the dirty work. HWND hCtl = (HWND)lParam; DWORD PropId = GetDlgCtrlID(hCtl) - IDC_SLIDER_BASE; DWORD dwPos = (DWORD)SendMessage(hCtl, TBM_GETPOS, 0, 0); // Now convert that to an absolute value. // Let's get the range for the property, to start with. VMR9ProcAmpControlRange myProcAmpControlRange; myProcAmpControlRange.dwSize = sizeof(VMR9ProcAmpControlRange); myProcAmpControlRange.dwProperty = (VMR9ProcAmpControlFlags)PropId; hr = pMix->GetProcAmpControlRange(streamNum, &myProcAmpControlRange); if SUCCEEDED(hr) { // OK, we have the value and the ranges for the property, // so let's normalize for a 1-100 scale. float totalRange = myProcAmpControlRange.MaxValue - myProcAmpControlRange.MinValue; float multiple = totalRange / 100.0f; float theValue = multiple * dwPos; // Add the offset back in. theValue = theValue + myProcAmpControlRange.MinValue; // And now, pass that set value back. VMR9ProcAmpControl myProcAmpControl; myProcAmpControl.dwSize = sizeof(VMR9ProcAmpControl); myProcAmpControl.dwFlags = PropId; myProcAmpControl.Brightness = theValue; hr = pMix->SetProcAmpControl(streamNum, &myProcAmpControl); } else { return FALSE; } return TRUE; break; } return FALSE; }

ProcAmpAdjustProc is a standard Windows callback function that s invoked when events are generated and passed to the ProcAmp dialog box for processing. When the dialog box is first created, a WM_INITDIALOG message is generated. At this point, the Pip9 application needs to read the current brightness, hue, and saturation values for stream 0 (the large stream) so that it can set its slider values appropriately. Before any of the values can be read, a test is performed to ensure that the video hardware attached to the system supports the ProcAmp features of the VMR. We also test individually for each property because certain video cards might support some ProcAmp features while leaving others unimplemented. These tests are performed with a call to the IVMRMixerControl9 method GetProcAmpControl, which is passed a pointer to a VMR9ProcAmpControl data structure. That structure is defined as follows:

typedef struct _VMR9ProcAmpControl { DWORD dwSize; DWORD dwFlags; float Contrast; float Brightness; float Hue; float Saturation; } VMR9ProcAmpControl;

When the call to GetProcAmpControl is made, the dwSize field of the VMR9ProcAmpControl structure must be set to the total size of the data structure, or the call will fail. On return, the dwFlags field will have flags set corresponding to the ProcAmp capabilities of the hardware.

typedef enum { ProcAmpControl9_Brightness = 0x00000001, ProcAmpControl9_Contrast = 0x00000002, ProcAmpControl9_Hue = 0x00000004, ProcAmpControl9_Saturation = 0x00000008, ProcAmpControl9_Mask = 0x0000000F } VMR9ProcAmpControlFlags;

Each of the four values (ProcAmpControl9_Brightness, ProcAmp Control9_Hue, ProcAmpControl9_Saturation, and ProcAmpControl9_Contrast) is tested against the dwFlags field to determine whether ProcAmp control is supported for that feature. If ProcAmp control is supported, a call to the IVMRMixerControl9 method GetProcAmpControlRange is made. The call receives a pointer to a VRM9ProcAmpControlRange data structure.

typedef struct _VMR9ProcAmpControlRange { DWORD dwSize; VMR9ProcAmpControlFlags dwProperty; float MinValue; float MaxValue; float DefaultValue; float StepSize; } VMR9ProcAmpControlRange;

As before, the dwSize field must be set to the size of the VMR9ProcAmpControlRange structure, while the dwProperty field is passed a value corresponding to one of the VMR9ProcAmpControlFlags, which determines which value is returned by the call. On return, the MinValue and MaxValue fields hold the lower and upper permissible range values for the property. These values are used (in conjunction with a value returned in the appropriate field by the GetProcAmpControl call) to determine the slider s position at initialization. This operation is performed three times, once each for the brightness, hue, and saturation sliders.

When ProcAmpAdjustProc receives a WM_HSCROLL message indicating that one of the sliders is processing a user-interface event, the value of the slider is translated into a value appropriate to the range of the given property. This translation is done by getting the range of the property with a GetProcAmpControlRange call. Once the range is known, the new slider position is compared to it and a new value for the property is calculated. That value is applied to the property with a call to SetProcAmpControl, which is passed a pointer to a VMR9ProcAmpControl structure. Once again, the dwSize field must be set to the size of the VMR9ProcAmpControl structure, and the dwFlags field must have the appropriate bit or bits set to indicate which properties are being changed. (You can change multiple properties with a single SetProcAmpControl call.) The appropriate field of the VMR9ProcAmpControl structure must be set to the new value for the property. When the call is made, the new value is applied to the property. ProcAmpAdjustProc is concise because all of its processing is performed in just three calls on the IVMRMixerControl9 interface: GetProcAmpControl, GetProcAmpControlRange, and SetProcAmpControl. These calls give you complete control over the VMR ProcAmp.

Using Direct3D with the VMR

Although well beyond the scope of this chapter (or this book), it s important to note that the render target of the VMR is completely under control of the programmer. You can write your own custom allocator-presenter module for the VMR, which can then be programmed to use a Direct3D object as the composition target, creating video texture maps. For this programming, you ll need to explore the IVMRSurfaceAllocator9 and IVMRImagePresenter9 interfaces, which allow access to the Direct3D data structures used by the VMR. (The DirectX SDK has a number of VMR 9 samples that also explore integration of the VMR with Direct3D.) In reality, the VMR is always drawing to a Direct3D surface, whether one is explicitly supplied to the VMR through the IVMRSurfaceAllocator9 interface or created automatically by the VMR when it s initialized. This capability gives the VMR enormous power to create sophisticated skins for 3D objects video texture maps, in the parlance of 3D with very little additional VMR programming.