7.10 Postprocessing

7.10 Postprocessing

7.10.1 Editing

Editing of a video sequence is best performed before compression, but situations may arise where only the coded bit stream is available. One possible method would be to decode the bit stream, perform the required editing on the pixels and recode the bit stream. This usually leads to a loss in video quality, and it is better, if possible, to edit the coded bit stream itself.

Although editing may take several forms, the following discussion pertains only to editing at the picture level, that is deletion of the coded video material from a bit stream and insertion of coded video material into a bit stream, or rearrangement of coded video material within a bit stream.

If a requirement for editing is expected (e.g. clip video is provided analogous to clip art for still pictures), then the video can be encoded with well defined cutting points. These cutting points are places at which the bit stream may be broken apart or joined. Each cutting point should be followed by a closed group of pictures (e.g. a GOP that starts with an I-picture). This allows smooth play after editing.

To allow the decoder to play the edited video without having to adopt any unusual strategy to avoid overflow and underflow, the encoder should make the buffer fullness take the same value at the first I-picture following every cutting point. This value should be the same as that of the first picture in the sequence. If this suggestion is not followed, then the editor may make an adjustment either by padding (stuffing bits or macroblocks) or by recording a few images to make them smaller.

If the buffer fullness is mismatched and the editor makes no correction, then the decoder will have to make some adjustment when playing over an edited cut. For example, consider a coded sequence consisting of three clips, A, B and C, in order. Assume that clip B is completely removed by editing, so that the edited sequence consists only of clip A followed immediately by clip C, as illustrated in Figure 7.18.

Figure 7.18: Edited sequences

Assume that in the original sequence the buffer is three-quarters full at the beginning of clip B, and one-quarter full at the beginning of clip C. A decoder playing the edited sequence will encounter the beginning of clip C with its buffer three-quarters full, but the first picture in clip C will contain a buffer fullness value corresponding to a quarter full buffer. In order to avoid buffer overflow, the decoder may try to pause the input bit stream, or discard pictures without displaying them (preferably B-pictures), or change the decoder timing.

For another example, assume that in the original sequence the buffer is one-quarter full at the beginning of clip B, and three-quarters full at the beginning of clip C. A decoder playing the edited sequence will encounter the beginning of clip C with its buffer one-quarter full, but the first picture in clip C will contain a buffer fullness value corresponding to a three-quarters full buffer. In order to avoid buffer underflow, the decoder may display one or more pictures for longer than the normal time.

If provision for editing was not specifically provided in the coded bit stream, or if it must be available at any picture, then the editing task is more complex, and places a greater burden on the decoder to manage buffer overflow and underflow problems. The easiest task is to cut at the beginning of a group of pictures. If the group of pictures following the cut is open (e.g. GOP starts with two B-pictures), which can be detected by examining the closed GOP flag in the group of pictures header, then editing must set the broken link bit to 1 to indicate to the decoder that the previous group of pictures cannot be used for decoding any B-pictures.

7.10.2 Resampling and up conversion

The decoded sequence may not match the picture rate or the spatial resolution of the display device. In such situations (which occur frequently), the decoded video must be resampled or scaled. In Chapter 2 we saw that CCIR-601 video was subsampled into SIF format for coding, hence for display it is appropriate to upsample it back into its original format. Similarly, it has to be temporally converted for proper display, as was discussed in Chapter 2. This is particularly important for cases where video was converted from film.