Susie Wee, Bo Shen and John Apostolopoulos
Streaming Media Systems Group
Palo Alto, CA, USA
Video compression algorithms are being used to compress digital video for a wide variety of applications, including video delivery over the Internet, advanced television broadcasting, video streaming, video conferencing, as well as video storage and editing. The performance of modern compression algorithms such as MPEG-1, MPEG-2, MPEG-4, H.261, H.263, and H.264/MPEG-4 AVC is quite impressive -- raw video data rates often can be reduced by factors of 15–80 or more without considerable loss in reconstructed video quality. This fact, combined with the growing availability of video encoders and decoders and low-cost computers, storage devices, and networking equipment, makes it evident that between video capture and video playback, video will be handled in compressed video form.
End-to-end compressed digital video systems motivate the need to develop algorithms for handling compressed digital video. For example, algorithms are needed to adapt compressed video streams for playback on different devices and for robust delivery over different types of networks. Algorithms are needed for performing video processing and editing operations, including VCR functionalities, on compressed video streams. Many of these algorithms, while simple and straightforward when applied to raw video, are much more complicated and computationally expensive when applied to compressed video streams. This motivates the need for developing efficient algorithms for performing these tasks on compressed video streams.
In this chapter, we describe compute- and memory-efficient, quality-preserving algorithms for handling compressed video streams. These algorithms achieve efficiency by exploiting coding structures used in the original compression process. This class of efficient algorithms for handling compressed video streams is called compressed-domain processing (CDP) algorithms. CDP algorithms that change the video format and compression format of compressed video streams are called compressed-domain transcoding algorithms, and CDP algorithms that perform video processing and editing operations on compressed video streams are called compressed-domain editing algorithms.
These CDP algorithms are useful for a number of applications. For example, a video server transmitting video over the Internet may be restricted by stringent bandwidth requirements. In this scenario, a high-rate compressed bitstream may need to be transcoded to a lower-rate compressed bitstream prior to transmission; this can be achieved by lowering the spatial or temporal resolution of the video or by more coarsely quantizing the MPEG data. Another application may require MPEG video streams to be transcoded into streams that facilitate video editing functionalities such as splicing or fast-forward and reverse play; this may involve removing the temporal dependencies in the coded data stream. Finally, in a video communication system, the transmitted video stream may be subject to harsh channel conditions resulting in data loss; in this instance it may be useful to create a standard-compliant video stream that is more robust to channel errors and network congestion.
This chapter presents a series of compressed-domain image and video processing algorithms that were designed with the goal of achieving high performance with computational efficiency. It focuses on developing transcoding algorithms for bitstreams that are based on video compression algorithms that rely on the block discrete cosine transform (DCT) and motion-compensated prediction. These algorithms are applicable to a number of predominant image and video coding standards including JPEG, MPEG-1, MPEG-2, MPEG-4, H.261, H.263, and H.264/MPEG-4 AVC. Much of this discussion will focus on MPEG; however, many of these concepts readily apply to the other standards as well.
This chapter proceeds as follows. Section 2 defines the compressed-domain processing problem. Section 3 gives an overview of MPEG basics and it describes the CDP problem in the context of MPEG. Section 4 describes the basic methods used in CDP algorithms. Section 5 describes a series of CDP algorithms that use the basic methods of Section 4. Finally, Section 6 describes some advanced topics in CDP.