Adding Data Unit Extensions to Windows Media

Adding Data Unit Extensions to Windows Media

The Windows Media Format SDK allows you to add supplemental data to the streams within a Windows Media file using data unit extensions (also known as payload extension systems). A data unit extension is simply a name that is paired with some data that s attached to a sample (or series of samples) in the Windows Media file. You can add data unit extensions while the Windows Media file is being created and then extract them from the file during playback.

The data unit extensions are sometimes confused with metadata. Like the data unit extensions, metadata is extra data added to an ASF file; unlike data unit extensions, metadata generally appears at the head of the file and is unconnected with any particular stream. Title and author information, for example, are both metadata types, while timecode information, which occurs on a per-sample basis, is a data unit extension.

In MakeASF, the author and title information are added once, to the file headers. The SMPTE timecode, on the other hand, is added on a per-sample basis, throughout the entire length of the sample. This requires a callback mechanism so the timecode information can be added to each sample just before the sample is encoded. Here s how we set up SMPTE timecoding in MakeAsfFile:

 // When adding Data Unit Extensions, // the order of operations is very important. if (TRUE == g_SMPTETimecodes) { // (1) Set the DUE on the profile stream. hr = AddSmpteDataUnitExtension(pProfile1); if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed AddSmpteDataUnitExtension! hr=0x%x\n"), hr)); return hr; } // (2) Update the filter with the new profile. hr = pConfigAsfWriter2->ConfigureFilterUsingProfile(pProfile1); if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed ConfigureFilterUsingProfile-3-! hr=0x%x\n"), hr)); return hr; } // (3) Find the video pin and register our callback. // Note here we use the same object to handle DUE callbacks // and index callbacks. So we create the object // on the heap , which is how COM objects should be created anyway. pASFCallback = new CASFCallback(); if(!pASFCallback) { return E_OUTOFMEMORY; } DbgLog((LOG_TRACE, 3, _T("About to QI for IAMWMBufferPassCallback!\n"))); hr = pASFCallback->QueryInterface( IID_IAMWMBufferPassCallback , (void**) &pSMPTECallback); if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to QI for IAMWMBufferPassCallback! hr=0x%x\n"), hr)); return hr; } // Find the video pin. CComPtr<IPin> pVideoPin; hr = GetPinByMajorType(pASFWriter, PINDIR_INPUT, MEDIATYPE_Video, &pVideoPin); if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to GetPinByMajorType(pVideoPin)! hr=0x%x\n"), hr)); return hr; } // Get its IAMWMBufferPass interface. CComQIPtr<IAMWMBufferPass, &IID_IAMWMBufferPass> pBufferPass( pVideoPin ) ; DbgLog((LOG_TRACE, 3, _T("About to set callback! hr=0x%x\n"), hr)) ; // Give it the pointer to our object. hr = pBufferPass->SetNotify( (IAMWMBufferPassCallback*) pSMPTECallback) ; if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to set callback! hr=0x%x\n"), hr)) ; return hr; } } // Now that we have set the final profile, // we can safely add the metadata. hr = AddMetadata(pASFWriter, wszAuthor, wszTitle); if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to set AddMetadata! hr=0x%x\n"), hr)); return hr; }

In the first step, the profile is updated with the SMPTE timecode information in a call to AddSmpteDataUnitExtension. The profile must be updated because the SMPTE timecode information is being added to the stream data, and this will affect the overall bandwidth requirements of the stream:

HRESULT AddSmpteDataUnitExtension(IWMProfile *pProfile) { HRESULT hr; DWORD dwStreams = 0; DWORD dwMediaTypeSize = 0; hr = pProfile->GetStreamCount(&dwStreams); if ( FAILED( hr ) ) { DbgLog((LOG_TRACE, 3, _T("Failed GetStreamCount (hr=0x%08x)!\n"), hr )); return hr; } // First, find the profile's video stream. for(WORD j = 1; j <= dwStreams ; j++) { CComPtr<IWMStreamConfig> pWMStreamConfig; hr = pProfile->GetStreamByNumber(j, &pWMStreamConfig); if ( FAILED( hr ) ) { DbgLog((LOG_TRACE, 3, _T("Failed GetStreamByNumber (hr=0x%08x)!\n"), hr )); return hr; } // Get the stream's major type. Note that in this example we assume // that there is only one video stream in the file. GUID guidStreamType; hr = pWMStreamConfig->GetStreamType(&guidStreamType); if ( FAILED( hr ) ) { DbgLog((LOG_TRACE, 3, _T("Failed GetStreamType (hr=0x%08x)!\n"), hr )); return hr; } // If this is the video stream, then set the DUE on it. if(IsEqualGUID(WMMEDIATYPE_Video, guidStreamType)) { CComQIPtr<IWMStreamConfig2, &IID_IWMStreamConfig2> pWMStreamConfig2 (pWMStreamConfig); hr = pWMStreamConfig2->AddDataUnitExtension( WM_SampleExtensionGUID_Timecode, WM_SampleExtension_Timecode_Size, NULL, 0); if ( FAILED( hr ) ) { DbgLog((LOG_TRACE, 3, _T("Failed to set SMPTE DUE (hr=0x%08x)!\n"), hr )); return hr; } else { DbgLog((LOG_TRACE, 3, _T("AddDataUnitExtension for SMPTE succeeded (hr=0x%08x)!\n"), hr )); } // Don't forget to call this, // or else none of the changes will go into effect! hr = pProfile->ReconfigStream(pWMStreamConfig); if ( FAILED( hr ) ) { DbgLog((LOG_TRACE, 3, _T("Failed to reconfig stream (hr=0x%08x)!\n"), hr )); return hr; } return S_OK; } } // We didn't find a video stream in the profile // so just fail without trying anything heroic. return E_FAIL; }

AddSmpteDataUnitExtension walks through the profile (a technique you saw in SetNativeVideoSize) looking for the video stream. (If you need to handle profiles with more than one video stream, you have to do some extra work here.) Once it locates the video stream, it acquires the IWMStreamConfig2 interface for the stream and calls its AddDataUnitExtension method. The method is passed a GUID indicating that a timecode will be added, and it passes a size for the timecode data. (A list of the permissible GUIDs for data unit extensions can be found in the Windows Media Format SDK.) Add DataUnitExtension adjusts the profile to reflect the timecode data now inserted into the stream. The changes are committed to the profile with another call to the IWMProfile method ReconfigStream.

Adding data unit extensions to a stream can make the stream much larger, depending on how much data you carry with every sample. This data will need to be streamed along with any audio and video streams to the client for playback. If you re not careful, you can add so much data using data unit extensions that it becomes impossible to play your streams over low-bandwidth connections.

Upon the return to MakeAsfFile, the new profile is applied to the filter with another call to ConfigureFilterUsingProfile. (Using the Windows Media Format SDK, we could have added these data unit extensions to the profile when we created it, but the Windows Media Profile Editor doesn t let you do this. Now you know how to add data unit extensions to a profile on the fly.)

A callback method must be registered on the video input pin of the WM ASF Writer so the SMPTE timecode can be updated as each media sample arrives at the filter. A callback object is instantiated and is queried for its IAMWMBufferPassCallback interface, which it inherits as a descendent. The video input pin on the WM ASF Writer filter is located with a call to the local function GetPinByMajorType, and using the returned IPin pointer, its IAMWMBufferPass interface is acquired. This interface is implemented only on the input pins of the WM ASF Writer and output pins of the WM ASF Reader; it is used to register callbacks on those pins. It has one method, SetNotify, which is passed a pointer to the IAMWMBufferPassCallback object. (The implementation of the callback object is discussed in a later section.) The callback has now been set; on every video sample received by the WM ASF Writer, the callback will be invoked. With the SMPTE timecode metadata callback installed, we can go through the simpler process of adding some author and title metadata to the output file with a call to the local function AddMetadata:

HRESULT AddMetadata(IBaseFilter* pFilter, WCHAR* pwszAuthor, WCHAR* pwszTitle) { HRESULT hr = S_OK; CComQIPtr<IServiceProvider, &IID_IServiceProvider> pServiceProvider(pFilter); if(!pServiceProvider) { return E_FAIL; } CComPtr<IWMWriterAdvanced2> pWriterAdvanced2; hr = pServiceProvider->QueryService(IID_IWMWriterAdvanced2, &pWriterAdvanced2); if(FAILED(hr)) { return hr; } CComQIPtr<IWMHeaderInfo3, &IID_IWMHeaderInfo3> pWMHeaderInfo3 (pWriterAdvanced2); if (!pWMHeaderInfo3) { return E_FAIL; } // If we wanted the ability to modify these attributes later // in our session, we would store this value. WORD pwIndex = 0; DWORD length = (DWORD) (wcslen(pwszAuthor)* sizeof(WCHAR)) ; //Set the author that the user entered. hr = pWMHeaderInfo3->AddAttribute(0, g_wszWMAuthor, &pwIndex, WMT_TYPE_STRING, 0, (BYTE*) pwszAuthor, length); if(FAILED(hr)) { return hr; } //Set the title that the user entered. length = (DWORD)(wcslen(pwszTitle)* sizeof(WCHAR)) ; hr = pWMHeaderInfo3->AddAttribute(0, g_wszWMTitle, &pwIndex, WMT_TYPE_STRING, 0, (BYTE*) pwszTitle, length); return hr; }

AddMetadata acquires the IServiceProvider interface on the WM ASF Writer filter. This interface offers a QueryService method, which is used to acquire the IWMWriterAdvanced2 interface. That interface is queried for its IWMHeaderInfo3 interface; it has the AddAttribute method, which is used to add metadata to a Windows Media file. The method is passed a stream number (stream 0, as passed here, means file-level metadata, but metadata can be applied on a per-stream basis), the name of the metadata attribute (Author or Title in this case), a data type, the data itself, and a length value for the data. The possible data types for metadata are listed in Table 15-1.

Table 15-1. Data Types for Metadata

Metadata Type	Description
WMT_TYPE_DWORD	32-bit DWORD value
WMT_TYPE_STRING	Null-terminated Unicode string
WMT_TYPE_BINARY	An array of bytes of arbitrary length
WMT_TYPE_BOOL	4-byte Boolean value
WMT_TYPE_QWORD	64-bit QWORD value
WMT_TYPE_WORD	16-bit WORD value
WMT_TYPE_GUID	128-bit GUID value

The File Source and WM ASF Writer filters have been added to the filter graph and interconnected properly. A callback for SMPTE metadata has been added to the WM ASF Writer, if needed, and any author and title metadata has been added to the output file. Now MakeAsfFile is ready to run the filter graph:

 // Now we are ready to run the filter graph and start encoding. // First we need the IMediaControl interface. CComQIPtr<IMediaControl, &IID_IMediaControl> pMC(pGraph); if(!pMC) { DbgLog((LOG_TRACE, 3, _T("Failed to QI for IMediaControl! hr=0x%x\n"), hr)); return hr; } hr = pMC->Run(); if(FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to run the graph! hr=0x%x\nCopy aborted.\n\n"), hr)); DbgLog((LOG_TRACE, 3, _T("Please check that you have selected the correct profile for copying.\n") _T("Note that if your source ASF file is audio-only, then selecting a\n") _T("video profile will cause a failure when running the graph.\n\n"))); return hr; } // Wait for the event signaling that we have reached the end // of the input file. We listen for the // EC_COMPLETE event here rather than in the app's message loop // in order to keep the order of operations // as straightforward as possible. The downside is that // we cannot stop or pause the graph once it starts. int nEvent = WaitForCompletion(g_hwnd, pGraph); // Stop the graph. If we are doing one-pass encoding, then we are done. // If doing two-pass, then we still have work to do. hr = pMC->Stop(); if (FAILED(hr)) DbgLog((LOG_TRACE, 3, _T("Failed to stop filter graph! hr=0x%x\n"), hr)); // We should never really encounter these two conditions together. if (g_fMultipassEncode && (nEvent != EC_PREPROCESS_COMPLETE)) { DbgLog((LOG_TRACE, 3, _T("ERROR: Failed to receive expected EC_PREPROCESSCOMPLETE.\n"))); return E_FAIL; } // If we're using multipass encode, run again. if (g_fMultipassEncode) { DbgLog((LOG_TRACE, 3, _T("Preprocessing complete.\n"))); // Seek to beginning of file. CComQIPtr<IMediaSeeking, &IID_IMediaSeeking> pMS(pMC); if (!pMS) { DbgLog((LOG_TRACE, 3, _T("Failed to QI for IMediaSeeking!\n"))); return E_FAIL; } LONGLONG pos=0; hr = pMS->SetPositions(&pos, AM_SEEKING_AbsolutePositioning , NULL, AM_SEEKING_NoPositioning); // Run the graph again to perform the actual encoding based on // the information gathered by the codec during the first pass. hr = pMC->Run(); if (FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to run the graph! hr=0x%x\n"), hr)); return hr; } nEvent = WaitForCompletion(g_hwnd, pGraph); hr = pMC->Stop(); if (FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to stop filter graph after completion! hr=0x%x\n"), hr)); return hr; } DbgLog((LOG_TRACE, 3, _T("Copy complete.\n"))); // Turn off multipass encoding. hr = pConfigAsfWriter2->SetParam(AM_CONFIGASFWRITER_PARAM_MULTIPASS, FALSE, 0); if (FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("Failed to disable multipass encoding! hr=0x%x\n"), hr)); return hr; } } // end if g_fMultipassEncode

The filter graph interface IMediaControl receives the Run message, and the filter graph begins execution. The WaitForCompletion local function interprets messages sent to the application. If single-pass encoding was specified, the function should return the value EC_COMPLETE. If multipass encoding was specified, the function returns EC_PREPROCESS_COMPLETE, indicating that the first of the two passes required to encode the streams has completed successfully.

If multipass encoding has been enabled, we need to rewind the source to its beginning so it can play a second time through the filter graph. We do this by acquiring the IMediaSeeking interface from the IMediaControl interface. We invoke its SetPositions method with parameters that specify a rewind to the beginning of the stream. This forces the File Source filter to seek to the beginning of the input file. Once the stream is rewound, the Run method is called again, and the filter graph again executes. This time, WaitForCompletion should return EC_COMPLETE, indicating that the multipass encoding process has completed successfully. When the encoding has completed (successfully or not) the multipass parameter on the WM ASF Writer is turned off with a call to the IConfigAsfWriter2 method SetParam.

There s only a little bit of MakeAsfFile left; it handles the frame indexing if that feature was enabled in the application s UI:

 // Finally, if frame-based indexing was requested, // this must be performed manually after the file is created. // Theoretically, frame based indexing // can be performed on any type of video data in an ASF file. if (g_fFrameIndexing) { if(!pASFCallback) // We didn't ask for SMPTE // so we need to create the callback object here { DbgLog((LOG_TRACE, 3, _T("Creating a new callback obj for Frame Indexing\n"))); pASFCallback = new CASFCallback(); if(!pASFCallback) { return E_OUTOFMEMORY; } } hr = IndexFileByFrames(wszTargetFile, pASFCallback); if (FAILED(hr)) { DbgLog((LOG_TRACE, 3, _T("IndexFileByFrames failed! hr=0x%x\n"), hr)); return hr; } OutputMsg(_T("A frame-based index was added to the file.\r\n")); } // Note: Our callback object is automatically cleaned up // when we exit here because we only addref'd // using CComPtrs in our app, // which call Release automatically when they go out of scope. The // WM ASF Writer filter also AddRef's the object // when it gets the SetNotify call, and it correctly // releases its pointers when its done with them. return hr; }

The application s UI permits two types of indexing: SMPTE indexing, which is implemented in a callback on the WM ASF Writer filter s input pin, and frame indexing. Once again, a callback object is instantiated, and it is passed to the local function IndexFileByFrames:

HRESULT IndexFileByFrames(WCHAR *wszTargetFile, CASFCallback* pCallback) { HRESULT hr; CComPtr<IWMIndexer> pIndexer; hr = WMCreateIndexer(&pIndexer); CASFCallback myCallback2; if (SUCCEEDED(hr)) { // Get an IWMIndexer2 interface to configure for frame indexing. CComQIPtr<IWMIndexer2, &IID_IWMIndexer2> pIndexer2(pIndexer); if(!pIndexer2) { DbgLog((LOG_TRACE, 3, _T("CopyASF: Failed to QI for IWMIndexer2! hr=0x%x\n"), hr)); return hr; } // Configure for frame-based indexing. WORD wIndexType = WMT_IT_NEAREST_CLEAN_POINT; hr = pIndexer2->Configure(0, WMT_IT_FRAME_NUMBERS, NULL, &wIndexType); if (SUCCEEDED(hr)) { HANDLE hIndexEvent = CreateEvent( NULL, FALSE, FALSE, WMVCOPY_INDEX_EVENT ); if ( NULL == hIndexEvent ) { DbgLog((LOG_TRACE, 3, _T("Failed to create index event!\n"))); return E_FAIL; } HRESULT hrIndex = S_OK; //CASFCallback pCallback->hEvent = hIndexEvent; pCallback->phr = &hrIndex; DbgLog((LOG_TRACE, 3, _T("About to QI for IWMStatusCallback\n"))); CComPtr<IWMStatusCallback> pIndexCallback; pCallback->QueryInterface(IID_IWMStatusCallback, (void**) &pIndexCallback); if (fVerbose) DbgLog((LOG_TRACE, 3, _T("\nStarting the frame indexing process.\n"))); hr = pIndexer->StartIndexing(wszTargetFile, pIndexCallback, NULL); if (SUCCEEDED(hr)) { // Wait for indexing operation to complete. WaitForSingleObject( hIndexEvent, INFINITE ); if ( FAILED( hrIndex ) ) { DbgLog((LOG_TRACE, 3, _T("Indexing Failed (hr=0x%08x)!\n"), hrIndex )); return hr; } //else DbgLog((LOG_TRACE, 3, _T("Frame indexing completed.\n"))); } else { DbgLog((LOG_TRACE, 3, _T("StartIndexing failed (hr=0x%08x)!\n"), hr)); return hr; } } else { DbgLog((LOG_TRACE, 3, _T("Failed to configure frame indexer! hr=0x%x\n"), hr)); return hr; } } return hr; }

The Windows Media Format API function WMCreateIndexer returns a pointer to an IWMIndexer interface, which is immediately queried for its IWMIndexer2 interface. IWMIndexer2 is configured with a call to its Configure method, specifying the stream to be indexed (only video streams can be indexed), the indexer type, the desired interval between index entries (NULL means use the default), and a pointer to the type of object associated with the index. Table 15-2 lists the possible values for the indexer type. Table 15-3 lists the permissible values for the index object type.

Table 15-2. Values for the Indexer Type

Indexer Value	Description
WM_IT_PRESENTATION_TIME	Indexer will build its index using presentation times as indexes.
WMT_IT_FRAME_NUMBERS	Indexer will build its index using frame numbers as indexes.
WM_IT_TIMECODE	Indexer will build its index using SMPTE timecodes as indexes.

Table 15-3. Values for the Index Object Type

Index Object Value	Description
WMT_IT_NEAREST_DATA_UNIT	The index will associate indexes with the nearest data unit, or packet, in the Windows Media file.
WMT_IT_NEAREST_OBJECT	The index will associate indexes with the nearest data object, or compressed sample, in the Windows Media file.
WMT_IT_NEAREST_CLEAN_POINT	The index will associate indexes with the nearest keyframe in the Windows Media file. This is the default index type.

Once the IWMIndexer2 interface has been configured, a callback is set up using the IWMStatusCallback interface, which the callback object inherits. The IWMIndexer method StartIndexing is invoked. The indexing takes place on its own thread, so the call to the Windows function WaitForSingleObject allows us to wait until the indexing operation is complete. During the indexing process, every time an index point is located by the indexer, the callback method is called. When complete, the entire index will have been added to the Windows Media file.