4.6 Multimedia Database Management System Products

---

4.6 Multimedia Database Management System Products

The first MMDBMSs rely mainly on the operating system for storing and querying files. These were ad hoc systems that served mostly as repositories.

The mid-1990s saw a first wave of commercial, implemented from scratch, and full-fledged MMDBMS. Some of them were MediaDB, now MediaWay, ^[224] JASMINE, ^[225] and ITASCA, which is the commercial successor of ORION. ^[226] They were all able to handle diverse kinds of data and provided mechanisms for querying, retrieving, inserting, and updating data. Most of these products disappeared from the market after some years of existence, and only a few of them continued and adapted themselves successfully to the hardware and software advances as well as to application changes. For instance, MediaWay provided early very specific support for a wide variety of different media types. Specifically, different media file formats varying from images, and video to PowerPoint documents, ^[227] can be managed segmented, linked, and searched. Information on the database management system available can be found at http://www.mediaway.com.

In a second wave, commercial systems were proposed that handle multimedia content by providing complex object types for various kinds of media. The object-oriented style provides the facility to define new data types and operators appropriate for the new kinds of media. ^[228] Therefore, broadly used commercial MMDBMSs are extensible ORDBMSs.

The most advanced solutions are marketed by Oracle 9i, IBM DB2, and IBM Informix. We already discussed the main characteristics of the Oracle interMedia and the related ORDImage, ORDAudio, and ORDVideo object types at several places in this book and we, therefore, refer the reader to Section 4.5 for further details.

The IBM DB2 Universal Database Extenders extend the ORDBMS management to images, video, audio, and spatial objects. All these data types are modeled, accessed, and manipulated in a common framework. Features of the multimedia extenders include importing and exporting multimedia objects and their attributes into and out of a database, controlling access to nontraditional types of data with the same level of protection as traditional data, and browsing or playing objects retrieved from the database.

For instance, the DB2 Image Extender defines the distinct data type DB2IMAGE with associated user-defined functions for storing and manipulating image files (http://www-3.ibm.com/software/data/db2/extenders/image.htm). The actual contents of the image file that DB2Image describes can be stored as binary large objects (BLOBs) or outside of the database in a file system. The following SQL-insert statement shows how an image is stored into a column named "image," in a table named "Picture." In this example, the content of the image comes from a server file and stored as a BLOB in the database:

  INSERT INTO Picture (image) VALUES(   DB2IMAGE (   CURRENT SERVER, /* database server name located in a       DB2 environment variable */   'pisa.jpg', /* source_file */   'JPG',             /* source_format */   1,                 /* 1=BLOB, 2=file pointer */   'my Image File'    /* comment */ ) ) 

The DB2 Image Extender provides similarity search functionality based on the QBIC technology ^[229] (in detail in Section 4.4) for images stored in the DB2IMAGE type. The QBIC technology provides the ability to query, or search, for images based on their content. Using this query mechanism, one may specify image content features, such as color values, and another image file as input to a query. These features are matched against the contents of the images stored in the database, and a score is assigned to each image. A score is a double-precision floating-point value between 0 and 1 that indicates how closely an image's features match those specified in the QBIC query. The lower the score, the closer the match.

DB2IMAGE columns must be explicitly enabled for QBIC queries by specifying on which image features one wants to be able to query. For each column enabled, the DB2 Image Extender analyzes the images in that column and stores their features in a QBIC catalog. These features are stored as additional administrative files that are maintained by the DB2 Image Extender.

The image features that can be used in QBIC queries are average color, histogram color distribution, positional color values, and the texture of an image. This is in line with the SQL/MM standard, as described in Section 4.3; however, different query syntax has been defined by IBM. For instance, the following SQL statement shows an example of executing a QBIC query that ranks each image in the image column based on how closely its average color matches red:

 SELECT CONTENTS(image),  QBScoreFROMStr('averageColor=<255,0,0>', image) AS SCORE  FROM signs ORDER BY SCORE 

The IBM Informix DataBlades provides similar multimedia search functionality as the IBM DB2 Multimedia Extenders, and details are not given here. Note that IBM Informix is the commercial development of the Postgres Database, which is the result of a research project at the University of California-Berkeley that was initiated by Stonebraker and Rowe. ^[230]

In addition to the commercial products, several research projects have implemented full-fledged multimedia database systems. Recently, successfully terminated projects include the following:

MIRROR, ^[231] an acronym for multimedia information retrieval reducing information overload, developed at the University of Twente, is a research MMDBMS that is developed to better understand the kind of data management that is required in the context of multimedia digital libraries. Its main features are an integrated approach to both content and traditional structured data management. MIRROR provides probabilistic inference mechanisms during the interaction with the user, ^[232] which has been adopted from cognitive theories. MIRROR is implemented on top of the ORDBMS Monet database system. More information on MIRROR may be obtained at http://wwwhome.cs.utwente.nl/~arjen/mmdb.html. On top of MIRROR runs the Acoi system (http://monetdb.cwi.nl/acoi/), which is a platform for the indexing and retrieval of video and image data. The system provides a plug-in architecture to subsequently index multimedia objects using various feature extraction algorithms. ^[233] Acoi relies on the COBRA (content-based retrieval) video data model (only low-level descriptors). COBRA introduces a feature grammar to describe the low-level persistent metadata and the dependencies between the extraction mechanisms. ^[234] Exhibit 4.13 shows the system architecture of the MMDBMS product.

Exhibit 4.13: Architecture of the Acoi/MIRROR System. ^[235]

In addition, the system distinguishes four distinct layers within video content: the raw data, the feature, the object, and the event layer. The object and event layers are concept layers consisting of entities characterized by spatial and temporal dimensions, respectively. This goes in line with the concepts of the MPEG-7 standards; however, no standardized entities are employed, and interoperable exchange of metadata is therefore not enabled. A description of the whole system, including a Web demonstration, may be found at http://wwwhome.cs.utwente.nl/~blokh/VLDB2001/.

DISIMA, ^[236], ^[237] developed at the University of Alberta, is an image database system that enables content-based querying.

Exhibit 4.14 shows the system architecture of DISIMA.

Exhibit 4.14: Architecture of the Distributed Multimedia Database Management System (DISIMA). ^[238], ^[239]

The prototype was implemented on top of the OODBMS ObjectStore. Queries are specified in the query language MOQL (or Visual MOQL for images only), which relies on a new type of concept model for both image and spatial applications. The associated query languages (MOQL and Visual MOQL), extending OQL, allow spatio-temporal querying as well as the definition of a presentation specification. For instance, the following simple query allows us to find all images in which a person appears (assuming that an Image table m and Person table p is defined beforehand).

 SELECT m  FROM Images m, Persons p  WHERE m contains p 

The distributed architecture of DISIMA involves different image sources including servers and file systems. Please refer to http://db.uwaterloo.ca/~ddbms/projects/multimedia/ for more information on DISIMA, a Web demonstration, and related projects.

The third wave, that is, currently running projects, addresses the needs of applications for richer semantic content. Most of them rely, thereby, on the new MPEG standards, MPEG-4 and MPEG-7. Representative running projects include, among others, the European Project ASSAVID, described in Section 4.1; the MARS project carried out at the University of Illinois at Urbana-Champaign; and the SMOOTH multimedia database and the MPEG-7 MDC developed by us.

MARS ^[240], ^[241], ^[242] realizes an integrated multimedia information retrieval and database management system that supports multimedia information as first-class objects suited for storage and retrieval based on their semantic content. MARS proposes a set of tools for an MMDBMS back end, as shown in Exhibit 4.15.

Exhibit 4.15: Set of tools proposed in the Multimedia Analysis and Retrieval System Projects (MARS). ^[243], ^[244], ^[245]

The MARS project includes the conception of a multimedia data model for content indexing and retrieval and for database management. The presented multimedia data model influenced the development of the descriptors in the MDS of MPEG-7. ^[246] MARS is a from-scratch management system, based on a query refinement processing. ^[247] Furthermore, a table of contents (ToC) extraction mechanism for videos has been developed. ^[248] A hybrid tree data structure to support high-dimension feature indexing in multimedia databases ^[249], ^[250] has also been realized. For the multimedia information retrieval, an adapted relevance feedback approach that is able to learn users' information needs in image databases is proposed. ^[251], ^[252] More information on MARS may be obtained from http://www-mars.ics.uci.edu. A Web demo page is available at http://www-db.ics.uci.edu/pages/demos/index.shtml.

Here we will detail two research MMDBMS developed by us: first, the SMOOTH multimedia database that relies on a metadatabase implementation, and second, the MDC that extends Oracle 9i with multimedia extensions relying on the MPEG-7 standard.

4.6.1 Use Case Study 1: The SMOOTH Multimedia Database

The SMOOTH ^[253], ^[254] prototype of a distributed multimedia database system integrates a query, navigation, and annotation framework for video media material driven by the content of a metadatabase. This metadatabase implements the semantic and structural part of our multimedia data model VIDEX, as presented in Section 4.3.

Exhibit 4.16 displays the general systems' architecture of SMOOTH. The video server provides selective access to the physical video streams (the Oracle Video Server with the supported protocol types UDP [User Datagram Protocol] and RTP [Real-Time Transport Protocol] was used).

Exhibit 4.16: SMOOTH distributed multimedia system scenario.

As described in Section 4.3 and Exhibit 4.9, the core of the metadatabase are the base classes: events, objects, persons, and locations. These classes are subclasses of a general ContentObject, which may refer to low-level motion objects. Application-specific classes are added by declaring subclasses (content classes) of the base classes. Furthermore, the metadatabase contains classes for a detailed structuring of video streams and for relating the instances of the semantic classes to the media segments.

The interfaces provide means to annotate, query, and navigate through video material. The core components of the client include the annotator, querier, presentation manager, and navigator. The annotator allows the structuring of the video into segments and the annotation of high-level content objects. The querier follows a text-based, structured query specification technique. It enables the definition of video queries by specifying conditions on content objects and the specification of semantic and temporal relations to other content objects. The results of a query are presented in a compact form; that is, server address, length, video description, and so forth. In the presentation manager, the user can compose presentations, with temporal constraints, from the query results. The navigator allows the navigation through the contents of the metadatabase.

Querier: The user is able to submit query conditions through a combination of fill-in forms, menu selections, and text fields. Furthermore, query refinement, query storage, and a replay mechanism are provided. For instance, in our soccer application, queries such as, "Find all video shots, where the player Lothar Matthäus scores a goal" can be specified. Exhibit 4.17 shows an extract of the SMOOTH query interface that allows the specification of this query. We need to declare two conditions, one for the subclass "Player" of "Person" and one for the subclass of "Goal" of "Event." Results may be displayed directly, as shown in Exhibit 4.17 or through a presentation dialog, seen later. In many cases, it is useful to specify a temporal constraint between two events; for example, a goal should strictly occur after the pass. This temporal constraint between the two events "Goal" and "Pass" is entered in the "Event" dialog, shown in Exhibit 4.18.

Exhibit 4.17: SMOOTH query dialog.

Exhibit 4.18: Specification of a temporal constraint between two events in SMOOTH.

Annotator: The underlying multimedia data model is generic and could lead to complex annotation processes. Therefore, SMOOTH provides an annotation tool to help the user annotate the videos and refer to already annotated units for reuse of information. For instance, Exhibit 4.19 shows a typical annotation situation. The already annotated segments are displayed in the upper panel, allowing the definition of larger segments in the bottom panel; for example, a "Scene" can be built from two "Shots." In addition, text fields and menu selections are provided to enter the content information for the events.

Exhibit 4.19: Extract of the annotator frame in SMOOTH.

Presentation Manager: The query results can be composed into a presentation in the dialog presentation manager shown in Exhibit 4.20. In the upper-left corner, the query results are displayed as beams with respective length in a pool ready to be used for a presentation composition. These query results are then dragged and dropped from the pool to be arranged along a time chart. Users may specify complex presentation scenarios by imposing temporal constraints on the video delivery. Temporal constraints comprise precedence constraints related to the ordering of the videos and delay constraints related to the waiting time between the videos. For instance, in Exhibit 4.20, we specify that the second video starts in Window 1 immediately after the completion of the first video. Then, upon the completion of the second video (after 28 seconds), two new videos are displayed in parallel in Windows 2 and 3. Exhibit 4.20 snapshots the situation after 32 seconds with two running videos in the bottom Windows 2 and 3, whereas the upper Widow 1 shows the last frame of the second video. The server addresses, ports, stream identifiers and time intervals of the requested video units are obtained from the result sets of the video queries.

Exhibit 4.20: Presentation manager in SMOOTH.

4.6.2 Use Case Study 2: The MPEG-7 MDC

The MPEG-7 Multimedia Data Cartridge (MDC) ^[255], ^[256] is a system extension of the Oracle 9i DBMS providing a multimedia query language, access to media, processing and optimization of queries, and indexing capacities relying on a multimedia database schema derived from MPEG-7.

The MDC builds on two main concepts (see Exhibit 4.21). At first, the Multimedia Data Model is the database schema that is derived from MPEG-7 descriptions. It is realized with the help of the extensible type system of the cartridge environment; that is, descriptors in the MPEG-7 schema are mapped to object types and tables. Second, the Multimedia Indexing Framework (MIF), which provides an extensible indexing environment for multimedia retrieval. The indexing framework is integrated into the query language and enables efficient multimedia retrieval.

Exhibit 4.21: Architecture of the MPEG-7 multimedia data cartridge. ^[257]

The Multimedia Schema of MDC relies on the one hand on the structural and semantic parts of the MPEG-7 standard (high-level descriptions). On the other hand, object types for the MPEG-7 low-level descriptors, such as color, shape, and texture, are provided and linked to the high-level descriptions. This enables one to retrieve multimedia data not only by low-level features but also by semantics in combination with low-level characteristics.

Exhibit 4.22 displays an extract of the multimedia schema derived from MPEG-7. It shows the mapping of an MPEG-7 StillRegionType, which is a delegate for images in MPEG-7 (i.e., StillRegion denotes complete images and parts of them) to object-relational database types. In the database schema, an object type of the same name (StillRegionType) is created. Some of its attributes are declared as separate object types, and some are defined by the specific SYS.XMLType.

Exhibit 4.22: Multimedia database schema of the MPEG-7 multimedia data cartridge. ^[258]

The decision of which type to use was carried on importance of the description for a possible query. For instance, the attribute with type TextAnnotationType was chosen to be detailed further because it is of importance for free-text search in the database, whereas the UsageInformation was chosen to be declared as SYS.XMLType. The latter type is suspected to contain only a little information for a concrete image, because meta-information on the usage may already be declared at the MPEG-7 root level; furthermore, it spans a DS that might contain many different descriptions with similar content. This would lead to many database object and tables, probably containing little content. Therefore, it was decided to store the subtree of the MPEG-7 document containing UsageInformation directly in the database type or table using the XMLType. Note that all information not retained in object types is not lost for querying. The reason is that the XMLType provides XPATH query functionality that enables one to reach elements and attributes of this document by XPATH. In other words, with a combined SELECT and XPATH query, any information may be reached.

However, as pointed out earlier, important information can be reached directly through object navigation, as, for instance, from StillRegion to TextAnnotation.

Other explicitly used object types include MediaInformation (MediaInformationType, MediaProfile, MediaFormat, etc.). These types describe information on the coding, media attributes, locations, and physical structure of the data.

Semantic content of an image may be obtained through the semantic reference linking to a SemanticBagType, which is an abstract root object type for concrete semantic indexing classes, such as events, places, and time.

Finally, the decomposition of the image (structural aspect) is specified by following the reference of the SpatialDecomposition attribute in the StillRegionType. Further important object types are ScalableColorType and ColorStrucutureType. They are used to store the feature vectors of the color histograms extracted from the images described by the StillRegionType and are indexed with the help of our MIF extension.

The MIF offers advanced indexing services to the MMDBMS. It is generic in a way that new index types may be added without changing the interface definitions. The MIF is divided into three modules (as seen in Exhibit 4.21). Each module, especially the GistService and the Oracle Enhancement, may be used on its own and may be distributed over the network.

The GistService is the main part and is realized in the external address space. It offers services for index management based on the Generalized Search Trees (GiST) ^[259],¹⁶⁸ (e.g., X-tree and SR-tree; see Section 4.2) and on access methods not relying on balanced trees (VA files). This service is split into two components: GistCommunicator and GistHolder. The GistCommunicator is a COM-object (component object model) used for interprocess communication between the database (by passing through the GistWrapper shared library) and the implemented access methods. Thus, the GistCommunicator supplies the necessary functionality (e.g., creating, inserting, deleting) for accessing the index structures. The GistHolder manages all currently running index trees and the accesses to them. Each index tree is identified through a global and unique identifier, which is forwarded to the accessing process.

The integration of MIF into the MDS is done via the index extension mechanisms of Oracle 9i. For each new index, a new Oracle indextype has to be defined, but the interface to the Gist remains unchanged.

^[224]Phillips, B., Mediaway presses access to multimedia database. PC Week, 13, 39–40, 1996.

^[225]Khoshafian, S., Dasananda, S., and Minaasian, N., The Jasmine Object Database: Multimedia Applications for the Web, Morgan Kaufmann Publishers, 1998.

^[226]Woelk, D., Kim, W., and Luther, W., An object-oriented approach to multimedia databases, in SIGMOD Conf., Washington, D.C., 1986, pp. 311–325.

^[227]Phillips, B., Mediaway presses access to multimedia database. PC Week, 13, 39–40, 1996.

^[228]Subrahmanian, V.S., Principles of Multimedia Database Systems, Morgan Kaufman Press, 1998.

^[229]Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D., Steele, D., and Yanker, P., Query by image and video content: the QBIC system. IEEE Comput., 28, 23–32, 1995.

^[230]Stonebraker, M.R. and Rowe, L.A., The Design of POSTGRES, in Proceedings of the ACM SIGMOD International Conference on Management of Data, Washington, DC, May 1986, pp. 340–355.

^[231]de Vries, A.P., van Doorn, M.G.L.M., Blanken, H.M., and Apers, P.M.G., The MIRROR MMDBMS architecture, in Proceedings of the International Conference on Very Large Databases, Edinburgh, 1999, pp. 758–761.

^[232]van Doorn, M.G.L.M. and de Vries, A.P., The psychology of multimedia databases, in Proceedings of the Fifth ACM Conference on Digital Libraries, San Antonio, June 2000, pp. 1–9.

^[233]de Vries, A.P., Windhouwer, M.A., Apers, P.M.G., and Kersten, M.L., Information access in multimedia databases based on feature model, New Generation Comput., 18, 323–339, 2000.

^[234]Blok, H.E., Windhouwer, M.A., van Zwol, R., Petkovic, M., Apers, P.M.G., Kersten, M.L., and Jonker, W., Flexible and scalable digital library search, in Proceedings of the International Conference on Very Large Data Bases (VLDB), Rome, September 2001, pp. 705–706.

^[235]de Vries, A.P., Windhouwer, M.A., Apers, P.M.G., and Kersten, M.L., Information access in multimedia databases based on feature model, New Generation Comput., 18, 323–339, 2000.

^[236]Oria, V., Özsu, M.T., Xu, B., Cheng, L.I., and Iglinski, P., VisualMOQL: the DISIMA visual query language, in IEEE International Conference on Multimedia Computing and Systems, Florence, Italy, June 1999, Vol. 1, pp. 536–542.

^[237]Oria, V., Özsu, M.T., Iglinski, P., Lin, S., and Ya, B., DISIMA: a distributed and interoperable image database system, in Proceedings of the ACM SIGMOD International Conference of Management of Data, Dallas, May 2000, p. 600.

^[238]Oria, V., Özsu, M.T., Xu, B., Cheng, L.I., and Iglinski, P., VisualMOQL: the DISIMA visual query language, in IEEE International Conference on Multimedia Computing and Systems, Florence, Italy, June 1999, Vol. 1, pp. 536–542.

^[239]Oria, V., Özsu, M.T., Iglinski, P., Lin, S., and Ya, B., DISIMA: a distributed and interoperable image database system, in Proceedings of the ACM SIGMOD International Conference of Management of Data, Dallas, May 2000, p. 600.

^[240]Porkaew, K., Ortega, M., and Mehrotra, S., Query reformulation for content based multimedia retrieval in MARS, in IEEE International Conference on Multimedia Computing and Systems, Florence, June 1999, Vol. 2.

^[241]Chakrabarti, S., Porkaew, K., and Mehrotra, S., Efficient query refinement in multimedia databases, in Proceedings of the IEEE International Conference on Data Engineering (ICDE), San Diego, February–March 2000, p. 196.

^[242]Rui, Y., Huang, T., and Mehrotra, S., Relevance feedback: a powerful tool for contentbased image retrieval. IEEE Trans. Circuits Syst. Video Technol., 8, 25–36, 1998.

^[243]Porkaew, K., Ortega, M., and Mehrotra, S., Query reformulation for content based multimedia retrieval in MARS, in IEEE International Conference on Multimedia Computing and Systems, Florence, June 1999, Vol. 2.

^[244]Chakrabarti, S., Porkaew, K., and Mehrotra, S., Efficient query refinement in multimedia databases, in Proceedings of the IEEE International Conference on Data Engineering (ICDE), San Diego, February–March 2000, p. 196.

^[245]Rui, Y., Huang, T., and Mehrotra, S., Relevance feedback: a powerful tool for contentbased image retrieval. IEEE Trans. Circuits Syst. Video Technol., 8, 25–36, 1998.

^[246]Rui, Y., Huang, T.S., and Mehrotra, S., MARS and its applications to MPEG-7. Technical Report TR-DB-97-05, Department of Computer Science, University of Illinois, Chicago, 1998.

^[247]Chakrabarti, S., Porkaew, K., and Mehrotra, S., Efficient query refinement in multimedia databases, in Proceedings of the IEEE International Conference on Data Engineering (ICDE), San Diego, February–March 2000, p. 196.

^[248]Rui, Y., Huang, T.S., and Mehrotra, S., Constructing table-of-content for videos. ACM Multimedia Syst., 7, 359–368, 1999.

^[249]Chakrabarti, K. and Mehrotra, S., The hybrid tree: an index structure for high dimensional feature spaces, in Proceedings of the International Conference on Data Engineering (ICDE), Sydney, March 1999, pp. 440–447.

^[250]Chakrabarti, K. and Mehrotra, S. Efficient concurrency control in multidimensional access methods, in Proceedings of the International ACM SIGMOD Conference on Management of Data, Philadelphia, June 1999, pp. 25–36.

^[251]Porkaew, K., Ortega, M., and Mehrotra, S., Query reformulation for content based multimedia retrieval in MARS, in IEEE International Conference on Multimedia Computing and Systems, Florence, June 1999, Vol. 2.

^[252]Rui, Y., Huang, T., and Mehrotra, S., Relevance feedback: a powerful tool for contentbased image retrieval. IEEE Trans. Circuits Syst. Video Technol., 8, 25–36, 1998.

^[253]Kosch, H., Tusch, R., Böszörményi, L., Bachlechner, A., Dörflinger, B., Hofbauer, C., Riedler, C., Lang, M., and Hanin, C., SMOOTH—a distributed multimedia database system, in Proceedings of the International VLDB Conference, Rome, Italy, September 2001, pp. 713–714.

^[254]Tusch, R., Kosch, H., and Böszörményi, L., VideX: an integrated generic video indexing approach, in Proceedings of the ACM Multimedia Conference, Los Angeles, October–November 2000, pp. 448–451.

^[255]Döller, M., Kosch, H., Dörflinger, B., Bachlechner, A., and Blaschke, G., Demonstration of an MPEG-7 Multimedia Data Cartridge, ACM Multimedia 2002, Juan-Les-Pins, December 2002, pp. 429–430.

^[256]Döller, M. and Kosch, H., An MPEG-7 Multimedia Data Cartridge, SPIE Conference on Multimedia Computing and Networking 2003 (MMCN 2003), Santa Clara, January 2003.

^[257]Döller, M. and Kosch, H., An MPEG-7 Multimedia Data Cartridge, SPIE Conference on Multimedia Computing and Networking 2003 (MMCN 2003), Santa Clara, January 2003.

^[258]Döller, M. and Kosch, H., An MPEG-7 Multimedia Data Cartridge, SPIE Conference on Multimedia Computing and Networking 2003 (MMCN 2003), Santa Clara, January 2003.

^[259]Minka, T.P. and Picard, R.W., Interactive learning using a "Society of Models," Pattern Recognition, 30, 1997.

---