10.7 Coding of the background

An important advantage of the content-based coding approach in MPEG-4 is that the compression efficiency can be significantly improved by not coding the video background. If the background is completely static, all the other noncontent-based codecs would not code the background either. However, due to noise, camera shaking, or deliberate camera movement, such as panning, zooming, tilt etc., the background cannot be entirely static. One of the attractive features of content-based or object-based coding is that the background movement does not need to be represented in its exact form. Viewers normally do not pay attention to the accuracy of the background video, unless it is badly coded, such that the coding distortion distracts the viewer from the main scene.

An efficient way of coding the background is to represent the background movement with a global motion model. In section 9.4.5 we saw that motion compensation with the spatial transform was capable of dealing with complex motions such as translation, sheering, zooming etc. The same concept can be used to compensate for the global motion of the background. Here the whole background VOP is transformed to match against the VOP of the previous frame. The transformation parameters then represent the amount of information required to code the background VOP, which corresponds to an extremely low bit rate.

In MPEG-4, the global motion is either represented by six motion parameters (three motion vectors) through the affine transform, defined as:

(10.13)

or, with eight parameters (four motion vectors), using the perspective transform, defined by:

(10.14)

In both cases, similar to the bilinear transform of section 9.4.5, the coordinates (u, v) in the current frame are matched against the coordinates (x, y) in the previous frame. The best matching parameters then define the motion of the object (in this case, the background VOP).

In MPEG-4, global motion compensation is based on the transmission of a static sprite. A static sprite is a (possibly large) still image, describing panoramic background. For each consecutive image in a sequence, only eight global motion parameters describing camera motion are coded to reconstruct the object. These parameters represent the appropriate perspective transform of the sprite transmitted in the first frame.

Figure 10.22 depicts the basic concept for coding an MPEG-4 video sequence using a sprite panorama image. It is assumed that the foreground object (tennis player, top right image) can be segmented from the background and that the sprite panorama image can be extracted from the sequence prior to coding. (A sprite panorama is a still image that describes the content of the background over all frames in the sequence.) The large panorama sprite image is transmitted to the receiver only once as the first frame of the sequence to describe the background. The sprite is stored in a sprite buffer. In each consecutive frame only the camera parameters relevant for the background are transmitted to the receiver. This allows the receiver to reconstruct the background image for each frame in the sequence based on the sprite. The moving foreground object is transmitted separately as an arbitrary-shape video object. The receiver composes both the foreground and background images to reconstruct each frame (bottom picture in Figure 10.22). For low delay applications it is possible to transmit the sprite in multiple smaller pieces over consecutive frames or to build up progressively the sprite at the decoder.

click to expand
Figure 10.22: Static sprite of Stefan (courtesy of MPEG-4)

Global motion compensation can also be applied to the foreground objects. Here, for each VOP, a spatial transform, like the perspective transform, is used to estimate the transformation parameters. These are regarded as the global motion parameters that are used to compensate for the global motion of the foreground object. Globally motion compensated VOP is then coded by motion compensated texture coding, where it is once more motion compensated, but this time with a conventional block matching technique.