5.5 Content Adaptation

---

5.5 Content Adaptation

The advances in network technology, together with novel communication protocols and the considerably enhanced throughput bandwidths of networks, attracted more and more consumers to load or stream multimedia data to their PCs. Despite all these technological progresses, to date, nobody can guarantee for an end-to-end QoS over a variety of heterogeneous network domains and devices. For instance, when trying to connect the local network of the University of Klagenfurt (Institute of Information Technology) to that of the University of Krakow (Computer Science Department) we observed at least four different domains having different capacities and characteristics; for example, different link bandwidths and error rates. ^[120]

One promising solution is to resort to the adaptation principle; for example, dynamic content adaptation of the media quality to the level admitted by the network. ^[121], ^[122], ^[123], ^[124] The requirement for the use of effective content adaptation is accentuated by the client constraints; for example, a user requests for an HDTV quality video object, but the access constraint of the terminal forces the user to view a low-resolution video. In this regard, content must also include personalization and user profile management. These issues are tackled in the MPEG-21 ^[125] DI adaptation part, as described in Chapter 3.4.

Content adaptation is achieved by modifying the quality of a media object so that it can be delivered over the network with the available bandwidth and can then be presented at the terminal satisfying its access and user constraints. The quality of a media object, usually considered adaptable, refers to the resolution of an object and its play rate. ^[126] Exhibit 5.8 reconsiders the C-S Multimedia System of Exhibit 5.2 for content adaptation. The server adapts the stored MPEG-4 videos (Main Profile) per request so that they may be displayed on the terminals. For the mobile phone, the video object with the highest priority among the objects compliant to the simple profile will be retained. For the PDA, all video objects not compliant to MPEG-4 Core are removed.

Exhibit 5.8: Client-server multimedia system with content adaptation.

Content adaptation is related to the widely known Universal Multimedia Access (UMA) framework, which means universal access to rich multimedia. This includes methods for scalable coding and content adaptation. Research on UMA is pursued, for example, by projects at EPFL Lausanne (http://ltswww.epfl.ch/~newuma); at NTNU Trondheim (http://www.midgardmedia.net/); ^[127] at Columbia University, New York (http://www.ctr.columbia.edu/~ywang/Research/UMA/); and at Siemens, Munich (Multimedia Message Box) and in the ADMITS project (http://www-itec.uniklu.ac.at/~harald/research.html). ^[128], ^[129], ^[130] These UMA projects provide important results and insights into media adaptation and supply terminal and network capability descriptions as well as user preference descriptions to steer the adaptation process. The latter also holds for the current activities in MPEG-21 DI Adaptation, ^[131] as described in detail in Chapter 3.4.

Content Adaptation is supported by scalable coding. For instance, in MPEG-4, ^[132] content adaptation can effectively drop video objects (VO) in the MPEG-4 encoded video by attributing priorities to the different VOs and adapting the stream by discarding VOs by their priority information.

Content adaptability depends on the type of media encoding technique employed. In the paragraphs that follow, we will describe content adaptation possibilities for MPEG-4 AV streams. Scalable coding is equally considered for scene descriptions, two- and three-dimensional graphics, and avatars, which are important for distributed games. Research involves here the selection of scalable coding for scene descriptions and vector graphics, focusing on complexity-optimized decoding, composition, and rendering algorithms allowing implementation on terminal devices with low computational power.

Considering MPEG-4 AV streams, content adaptation can be exercised either on single or composed elements of the syntactical structure of an MPEG-4 AV streams by the following means ^[133], ^[134], ^[135]:

• Translation: Translation involves the conversion from one modality (image, video, text, audio, synthetic model) to another. Examples of translation are text-to-speech, speech-to-text (speech recognition), video-to-image (video mosaicking), image-to-text (embedded caption recognition) conversions, and three-dimensional model rendering.

• Summary: This involves the reduction of information details (AV abstract). Examples of summaries include the MPEG-7 summary DS and its subtypes. ^[136] For example, the subtype SequentialSummary DS can be used for representing multiple key frames extracted from a single Internet streaming video, ^[137] as it provides the possibility of linking a summary to a VideoSegment.

• Scaling: Scaling involves operations of data transcoding, manipulation, and compression, which result in the reduction of size and quality. Examples of scaling include image, video, and audio transcoding; image size reduction; video frame dropping; color conversion; and Discrete Cosine Transform coefficient scaling.

• Extraction: This involves the extraction of information from the input. Examples of extraction include key frame extraction from video; audio-band and voice extraction from audio; paragraph and key term extraction from text, region, segment, and object; and event extraction from audio and video.

• Substitution: Substitution indicates that one representation can be used to substitute another. Examples of substitution include a text passage that replaces a photographic image when a photographic image cannot be handled by a terminal device, or an audio track that replaces a chart in a presentation.

• Revision: This indicates that the AV program was revised in some way, such as through editing or postprocessing, to achieve the quality adaptation.

Content adaptation can successfully be employed at different levels of the network. For instance, a proxy cache may change the compression ratio of the streamed data to adapt to a changing network bandwidth. An active router may discard an enhancement layer to act against sudden network congestion. A proxy cache can enhance its cache replacement policy by applying filter algorithms to obtain more space instead of replacing an object and, therefore, enhance its cache hit-rate.

There are some initial works considering these issues on proxy caches, ^[138], ^[139], ^[140] but to the best of our knowledge, there is not any work done for other network components. In this context, our ADMITS project ^[141], ^[142], ^[143] proposes to encode the adaptation capability information of multimedia data into MPEG-7 descriptors to provide this information to the components on the delivery path to the client. This information may then be used to improve the performance of these components; for example, they may govern their replacement policies better. MPEG-7 is well suited in this context, as it proposes special descriptors to describe content adaptation (e.g., the Variation DS).

For instance, the following spatialReduction variation of the first frame in our pisa.mpg would lead to the following MPEG-7 fragment (Exhibit 5.9).

Exhibit 5.9: MPEG-7-based content adaptation: spatialReduction example.

In this sense, the ADMITS project aims at global adaptation based on properly distributed individual adaptation measures. This is in accordance with newer approaches in peer-to-peer networking (in comparison to the more C-S-oriented UMA architectures; see also the architectural consideration at the beginning of this Chapter 5), such as the Media Accelerating Peer Services, as described in Lienhart et al. ^[144] More information on the ADMITS is given below.

Another important issue is the support of the MMDBMS in the content adaptation process. From the point of view of a MMDBMS, the proposed multimedia data models are not able to satisfy the requirements of content adaptation. The implemented data models contain actually only very rudimentary information about the delivery of data, for example, frame rate, but more advanced features such as the quality adaptation capabilities of the streams are not considered. This information would be of interest to the user. For instance, a mobile client is only interested in videos being adaptable to the special requirements of the mobile terminal. Therefore, it is necessary to propose a common framework for modeling and querying content information and quality adaptation capabilities of AV data. ^[145]

The modeling of the perceptual quality of AV data, with respect to the adaptation possibilities, is the other important issue for effective content adaptation. There have been recent considerable advances in the domain of video quality modeling. Examples are the works of Winkler ^[146], ^[147], ^[148] as well as the techniques worked out in the Video Quality Experts Group. ^[149] The proposed methods focus on the effect of variable bit rates on perceptual quality. For instance, the ITU-R Recommendation 500 ^[150] proposes several standards for perceptual quality testing, including viewing conditions, criteria for observer selection, assessment procedures, and analysis methods. However, conceptualizing and validating models for the perceptual quality of a video with respect to the content adaptation on single or composed elements, as described above, is an open research issue.

5.5.1 Use Case: Metadata-Driven Adaptation in the ADMITS Project

The ADMITS project ^[151], ^[152], ^[153],^[154] realizes an end-to-end multimedia delivery chain as shown in Exhibit 5.10. The exhibit gives a walkthrough of the metadata-driven adaptation in the end-to-end multimedia scenario. The numbers in the shaded circles indicate the sequence of events in the adaptation process. The individual components play different roles in the adaptation process. They are connected physically by the network and semantically by MPEG-7 metadata that flow over the network in conjunction with the media data. The major components, together with the metadata life cycle, are discussed in the sequel.

Exhibit 5.10: End-to-end metadata-driven adaptation scenario.

5.5.2 Metadatabase

The metadatabase supports all kinds of adaptations by supplying metadata. Queries formulated by the clients are forwarded to the metadatabase, containing data about the AV data stored on the media server. The metadata enables multiple functionality:

• Support of content-based queries, based on low- or high-level indexing information

• Support of adaptation based on metainformation, such as the definition of transcoding procedures with the help of MPEG-7 variation descriptors

5.5.3 Adaptive Virtual Video Server

The adaptive virtual video server provides a means for component adaptation. The video server stores the raw AV data. It has a distributed architecture containing a number of nodes. In particular, it is composed of a media storage server and a metadatabase (Exhibit 5.10). Content indexing for obtaining relevant metadata is done in two ways: the media is analyzed for semantic and structural content (Step 1a) and for its adaptation capabilities (Step 1b). Therefore, we extract two categories of metadata in two steps:

• Step 1a: In this step, video segmentation is carried out, and semantically meaningful information is extracted; for example, which events, persons, and objects are the important entities of a segment. Furthermore, low-level descriptors are extracted; for instance, the MPEG-7 DominantColor and the ScalableColorType. MPEG-7 Semantic and structural Descriptors enable content-based queries. ^[155]

• Step 1b: In this step, MPEG-7 Variation Descriptors are produced. These descriptors express the relationship between the available variations of the media and their characteristics in terms of media information (e.g., file size and frame rate) and quality.

The ADMITS server provides delivery functionality for media and its descriptive metadata. Therefore, on a media request, the media server contacts the metadatabase for descriptions (Step 2a). These descriptions are attached to the media stream and delivered to the client (Step 2b).

The server is called virtual because it is able to change the set of actually allocated physical nodes on demand. The infrastructure recommends nodes to be allocated to implement adaptation. For example, in a specific scenario, a component called the host recommender might notice that a client is "far" from the nodes storing the stripe units (e.g., because of a slow connection) and that, therefore, it would be desirable to allocate a new node near the client to act as a data collector and streaming proxy. The proxy functionality is loaded by the so-called application loader. After that, the other nodes of the server push their stripe units to the new node, which collects and streams them to the client.

5.5.4 Adaptive Proxy Cache

This component supports media adaptation. The proxy cache implements improved, quality-aware versions of the well-known LRU and greedy dualsize algorithms. The cache initially stores full videos with their respective metadata descriptions in the form of MPEG-7 variation descriptors, and it reduces their quality (and thus their size) in integral steps before fully deleting them. Fortunately, the relation between size and quality reduction is usually nonlinear up to a certain limit, and hence, large size reduction usually causes a moderate quality loss. Thus, the cache can offer both short start-up delay and acceptable quality. The essential new idea is that the quality reduction process is driven by metadata in a standardized format (MPEG-7) that is readily provided in ADMITS by the metadatabase (Step 3 of Exhibit 5.10). For each adaptable video, the cache expects just the simple triple parameter (transcoding operation, resulting size, resulting quality).

5.5.5 Adaptive Routers

Routers with enriched functionality may support media adaptation as well. In contrast to a proxy cache, the operations that a router can perform on a media stream are fairly limited, as the forwarding speed must not be compromised significantly. In contrast, a router is best positioned, for example, to cope judiciously with dynamically varying network load (most importantly, congestion) or to multicast media streams along heterogeneous links or toward clients with different capabilities. Typical adaptation operations of routers would be dropping of video frames, of certain enhancement layers, or of entire MPEG-4 elementary streams. The challenge in this context is to provide and efficiently encode and communicate metadata for routers that convey this information and enable them to perform effective media adaptations.

5.5.6 Adaptive Query and Presentation Interface

The adaptive query and presentation interface supports the specification of search criteria for media resources, displays results from the database (metadata), proposes means for selecting the media from the database results, and opens players for the selected media. It is the final adaptation level in the end-to-end scenario as depicted in Exhibit 5.10, Step 4. It is adaptive in the sense that all interface components adjust dynamically to the usage environment. The usage environment comprises the client's terminal capabilities (e.g., hardware, software) and the usage preferences (e.g., user prefers only audio files).

^[120]Kosch, H., Sota, R., Kitowski, J., Nikolow, D., Podlipnig, S., and Breidler, K., MMSRS—multimedia storage and retrieval system for a distributed medical information system, in High Performance Computing and Networking (HPCN) 2000 Conference, Amsterdam, May 2000, LNCS 1823, Springer-Verlag, New York, pp. 517–524.

^[121]Gecsei, J., Adaptation in distributed multimedia systems, Proc. IEEE MultiMedia, 4, 58–66, 1997.

^[122]Rejaie, R., On design of Internet multimedia streaming applications: an architectural perspective, in Proceedings of the IEEE International Conference on Multimedia and Exhibition, New York, July 2000, pp. 327–330.

^[123]Mohan, R., Smith, J.R., and Li, C.-S., Adapting multimedia internet content for universal access, IEEE Trans. Multimedia, 1, 104–114, 1999.

^[124]Naghshineh, M. and Willebeek-LeMair, M., End-to-end QoS provisioning in multimedia wireless/mobile networks using an adaptive framework," IEEE Comm. Mag., 35, 72–81, 1997.

^[125]Hill, K. and Bormans, J., Overview of the MPEG-21 Standard. ISO/IECJTC1/SC29/WG11 N4041 (Shanghai Meeting), October 2002, http://www.chiariglione.org/mpeg/.

^[126]Prabhakaran, B., Adaptive multimedia presentation strategies, Multimedia Tools and Appl., 12, 281–298, 2000.

^[127]Perkis, A., Abeljaoued, Y., Cristopoulos, C., Ebrahimi, T., and Chicaro, J.F., Universal multimedia access from wired and wireless systems, Transact. Circuits Syst.Signal Process., Special Issue on Multimedia Commun., 20, 387–402, 2001.

^[128]Heuer, J., Casas, J.L., and Kaup, A., Adaptive multimedia messaging based on MPEG7—the M3 box, in Proceedings of the Second International Symposium on Mobile Multimedia Systems and Applications, Delft, November 2000.

^[129]Böszörményi, L., Hellwagner, H., Kosch, H., Libsie, M., and Podlipnig, S., Metadata driven adaptation in the ADMITS Project, Image Comm., 18(8), 749–766, 2003.

^[130]Döller, M. and Kosch, H., An MPEG-7 multimedia data cartridge, in SPIE Conference on Multimedia Computing and Networking 2003 (MMCN 2003), Santa Clara, January 2003.

^[131]Hill, K. and Bormans, J., Overview of the MPEG-21 Standard. ISO/IECJTC1/SC29/WG11 N4041 (Shanghai Meeting), October 2002, http://www.chiariglione.org/mpeg/.

^[132]Koenen, R., MPEG-4 overview. ISO/IEC JTC1/SC29/WG11 N4668 (Jeju Meeting), March 2002, http://www.chiariglione.org/mpeg/.

^[133]Prabhakaran, B., Adaptive multimedia presentation strategies, Multimedia Tools and Appl., 12, 281–298, 2000.

^[134]Ramanujan, R.S., Newhouse, J.A., Ahamad, A., Kaddoura, M.N., Chartier, E.R., and Thurber, K.J., Adaptive streaming of MPEG video over IP networks, in Proceedings of the 22nd IEEE Conference on Computer Networks, November 1997.

^[135]Rejaie, R., Handley, M., and Estrin, D., Quality adaptation for congestion controlled video playback over the Internet, in Proceedings of ACM SIGCOMM 99, Cambridge, MA, September 1999, pp. 189–200.

^[136]van Beek, P., Benitez, A.B., Heuer, J., Martinez, J., Salembier, P., Smith, J., and Walker, T., MPEG-7: multimedia description schemes, ISO/IEC FDIS 15938-5:2001, October 2001.

^[137]Rehm, E., Representing internet streaming media with MPEG-7, in Proceedings of the ACM Multimedia 2000 Workshop Standards, Interoperability and Practice: Who Needs Standards Anyway?, New York, November 4, 2000, ACM Press, pp. 93–106.

^[138]Rejaie, R., Yu, H., Handley, M., and Estrin, D., Multimedia proxy caching mechanism for quality adaptive streaming applications in the Internet, in Proceedings of the 2000 IEEE Computer and Communications Societies Conference on Computer Communications (INFOCOM-00), Los Alamitos, March 26–30, 2000, IEEE CS Press, pp. 980–989.

^[139]Rejaie, R., Yu, H., Handely, M., and Estrin, D., Multimedia proxy caching mechanism for quality adaptive streaming applications in the Internet, in Proceedings of IEEE Infocom'2000, Tel-Aviv, March 2000, pp. 980–989.

^[140]Podlipnig, S. and Böszörményi, L., Quality-aware proxy caching for web videos, in Proceedings of the 3rd Austrian-Hungarian Workshop on Distributed and Parallel Systems DAPSYS'2000, Balatonfüred, September 2000.

^[141]Kosch, H., Böszörményi, L., and Hellwagner, H., Modeling quality adaptation capabilities of audio-visual data, in Proceedings of the International DEXA'2001 Workshops, Munich, September 2001, pp. 141–145.

^[142]Böszörményi, L., Döller, M., Hellwagner, H., Kosch, H., Libsie, M., and Schojer, P., Comprehensive Treatment of Adaptation in Distributed Mul-timedia Systems in the ADMITS Project, in Proceedings of the 10th ACM International Conference on Multimedia, Antibes, November–December 2002, ACM Press.

^[143]Böszörményi, L., Hellwagner, H., Kosch, H., Libsie, M., and Podlipnig, S., Metadata driven adaptation in the ADMITS Project, Image Comm., 18(8), 749–766, 2003.

^[144]Lienhart, R., Holliman, M., Chen, Y.-K., Kozintsev, I., and Yeung, M., Improving media services on P2P networks, IEEE Internet Comput., 6, 73–77, 2002.

^[145]Döller, M. and Kosch, H., An MPEG-7 multimedia data cartridge, in SPIE Conference on Multimedia Computing and Networking 2003 (MMCN 2003), Santa Clara, January 2003.

^[146]Winkler, S., Visual fidelity and perceived quality: towards comprehensive metrics, in SPIE Human Vision and Electronic Imaging Conference,Vol. 4299, San Jose, January 21–26, 2001.

^[147]Winkler, S., van den Branden Lambrecht, C.J., and Kunt, M., Vision and video: models and applications, in Vision Models and Applications to Image and Video Processing, van den Branden Lambrecht, C.J., Ed., Kluwer Academic Publishers, Dordrecht, 2001, chap. 10.

^[148]Winkler, S., Issues in vision modeling for perceptual video quality assessment, Signal Process., 78, 231–252, 1999.

^[149]Rohaly, A.M., Corriveau, P., Libert, J., Webster, A., Baroncini, V., Beerends, J., Blin, J.-L., Contin, L., Hamada, T., Harrison, D., Hekstra, A., Lubin, J., Nishida, Y., Nishihara, R., Pearson, J., Pessoa, A.F., Pickford, N., Schertz, A., Visca, M., Watson, A., and Winkler, S., Video Quality Experts Group: current results and future directions, in Proceedings of SPIE Visual Communications and Image Processing, Perth, June 21–23, 2000, pp. 742–753.

^[150]Switzerland ITU, Methodology for the Subjective Assessment of the Quality of Television Pictures, Geneva, 1998.

^[151]Kosch, H., Böszörményi, L., and Hellwagner, H., Modeling quality adaptation capabilities of audio-visual data, in Proceedings of the International DEXA'2001 Workshops, Munich, September 2001, pp. 141–145.

^[152]Böszörményi, L., Döller, M., Hellwagner, H., Kosch, H., Libsie, M., and Schojer, P., Comprehensive Treatment of Adaptation in Distributed Mul-timedia Systems in the ADMITS Project, in Proceedings of the 10th ACM International Conference on Multimedia, Antibes, November–December 2002, ACM Press.

^[153]Böszörményi, L., Hellwagner, H., Kosch, H., Libsie, M., and Podlipnig, S., Metadata driven adaptation in the ADMITS Project, Image Comm., 18(8), 749–766, 2003.

^[154]This research project is funded in part by FWF (Fonds zur Förderung der wissenschaftlichen Forschung), under the project numbers P14788 and P14789, and KWF (Kärntner Wirtschafts-förderungsfonds). http://www-itec.uni-klu.ac.at/~harald/research.html.

^[155]Döller, M. and Kosch, H., An MPEG-7 multimedia data cartridge, in SPIE Conference on Multimedia Computing and Networking 2003 (MMCN 2003), Santa Clara, January 2003.

---