12.4 Asset Management Formats

SMPTE has been working with users, manufacturers, and various industry groups to provide some universal standardized asset management solutions. Most of this work is based on the SMPTE Data Model that grew out of SMPTE's seminal 1998 cooperation with the European Broadcasting Union (EBU) on digital program interchange.

A simplified asset management system streams the content of individual material media onto a centralized asset storage facility indexed by a catalog database (see Figure 12.10). consists of storage media. Compressed stream formats, such as MPEG program streams and MPEG transport streams, are well suited for the transport and storage of uncut shots or for emission; DVDs are based on the MPEG Program format. The DV and DVCPRO formats are successful for camcorders and news applications. Long-GoP MPEG is heavily used in emission systems.

Figure 12.10. A simplified view of an Asset Management System.

The Universal Material IDentifier (UMID) [UMID] and Key Length Value (KLV) [KLV] standards are key elements in the effort to reduce the cost and increase effectiveness of asset management. To facilitate interpretability, each data component is labeled by a UMID, and all equipment and system components work with standard data wrapping formats. UMID and KLV were among the key building blocks of the Material eXchange Format (MXF) standard developed by the Pro-MPEG forum [PROMPEG] and the Advanced Authoring Format Association (AAF).

12.4.1 Universal Material Identifiers (UMID)

UMID is the SMPTE 330M standard and the RP205 recommended practice, both available at the SMPTE site. The work on UMID took into account numbering schemes previously standardized by the IEEE and the IETF. Related standards are the SMPTE 298M standard, that specifies UMID of digital data, and the SMPTE 335M, that specifies metadata dictionary structure [METADIC]. Also related is the International Standard Book Number (ISBN) and the International Standard Audiovisual Number (ISAN). Whereas other unique program identifiers such as the ISBN and ISAN are simple labels that identify only completed works, UMIDs can uniquely identify every component of a program and provide linkage between the various pieces of essence and their associated metadata.

The purpose of the UMID is to provide a standard means of identifying unique instances of media-essence throughout the production process of audiovisual productions and throughout the media-asset management systems within this process. They provide a reference for all captured content units, so that a particular content unit can always be located either locally, remotely or on a archive storage medium. Essentially , the UMID is a method for distinguishing all copies and versions of a unit. In addition, UMIDs can provides the link between the essence (video, audio, graphics, stills) and the metadata, including time-axis information (e.g., time-codes and dates) for synchronization.

The concepts of a content unit and a clip are essential to understanding the UMID. A content unit is the smallest practical unit to work with under a given set of circumstances. For digital audio, it could be the length of one audio frame, perhaps 4 ms. For video, it might be 1/60 of a second for an NTSC field or as much as half a second if the video was encoded as a long MPEG-2 group of pictures. A clip is a sequence of one or more of these content units.

It is expected that UMIDs are assigned automatically at the time of authoring or capturing, and ties ownership, rights, edit decisions, pan and scan vectors, virtually anything of importance, to its associated essence regardless of where or how it is stored. UMIDs are intended to be automatically generated by the systems and support granularity up to the level of shots and video and audio frames .

The UMID structure contains a set of components that each represent a key aspect of the audio- or video essence. There are two types of UMIDs:

• Basic UMID : It is a 32-byte globally unique identification for a clip. The first 12 bytes are the key identifying it as a UMID. They are followed by one byte to indicate length and 19 bytes of value. The first three value bytes are an instance number, the remaining 16 are the material number.

• Extended UMID : A 64-byte ID that extends the basic UMID with 32 bytes specifying the creation time and date (8 bytes), recording location (12 bytes), and the signature metadata of the creating organization: 4 bytes for the country, 4 bytes for the organization and 4 bytes for the user .

Country Code

A 4-byte country code is used which is an abbreviated alpha-numeric string according to the set defined in ISO 3166-1 using use alpha-numeric characters from the Latin set 1 as defined by ISO/IEC 8859-1 (alphanumeric character values may only lie in the range 20h to 7Fh.). Countries that are not registered may obtain a registered alphanumeric string from the SMPTE Registration Authority. Where the country code is less than 4 bytes, the active part of the code occupies the first part of the 4 bytes and the remainder is filled with the space character (0x20).

Organization Code

An organization code is used which is an abbreviated 4-byte alpha-numeric string limited to values between 20h and 7Fh of the Latin set 1 as defined by ISO/IEC 8859-1. Organization codes are defined in relation to their registered country code, allowing organization codes to have the same value in different countries. When the organization code is less than 4 bytes, the active part of the code occupies the first part of the 4 bytes and the remainder is filled with the space character (0x20). Organization codes may not use the ~ symbol (ISO/IEC 8859-1 character number:0 0x7E) as the first character. This character is reserved for freelance operator registration. Organizations without an organization code can obtain a registered alphanumeric string from the SMPTE Registration Authority.

User Code

he user code is used which is a 4-byte alphanumeric string limited to values between 20h and 7Fh of the Latin set 1 as defined by ISO/IEC 8859-1. When the user code is less than 4 bytes, the active part of the code occupies the first part of the 4 bytes and the remainder is filled with the space character (20h).

User codes are assigned locally by each organization and are not globally registered. User codes are defined in relation to their registered organization and country codes so that user codes may have the same value in different organizations and countries.

Freelance Operators

Freelance operators may use their country of domicility for the country code and organization code if appropriate. However, if freelance operators have no registered organization code, they may concatenate the 4-byte organization code with the 4-byte user code to form an 8-byte operator code. To ensure a unique code, freelance operators are encouraged to obtain a registered alphanumeric string from the SMPTE Registration Authority.

All operator codes start with the ~ symbol (ISO/IEC 8859-1 character number, 7Eh). The remaining 7 alphanumeric characters are all filled with characters from the range between 20h and 7Fh of the Latin set 1 as defined by ISO/IEC 8859-1. When the operator code is less than 8 bytes (including the ~ character), the active part of the code occupies the first part of the 8 bytes and the remainder is filled with the space character (20h).

12.4.2 KLV

The KLV, SMPTE 336M, is a data encoding protocol. The key is a unique registered sequence of bits, which conform to the ISO rules for an object identifier (in this case a SMPTE 298M UMID), and specify a type (e.g., video, audio, Edit Decision List (EDL), or UMID). This means that a compliant system receiving the key can trace its meaning to the appropriate registry and determine what to do with the data. The length element indicates how long the payload is, and the value , is the payload.

Almost all data formats could be mapped to the KLV data structure, which is streamable and provides capabilities such as playing while recording, and operation with isolated sections of streams. It supports frame-accurate metadata enabling video mapping. These metadata fields can carry anything from external positioning information through to editing and EDL history for the footage. The end-result is fast integration into live video systems and good synchronization between metadata and video.

KLV's success relies on its registries. They facilitate interchange by providing a common repository for definitions (data types, signal formats, effect names , plug-in types, content identifiers, etc.). The registries may be public and global (e.g., SMPTE UMID and the SMPTE Metadata Dictionary [METADIC]), or they may be private for such things as contract provisions and billing data. To avoid having to query a global registry for every action, local registries provide temporary storage of the most frequently used identifiers for a particular application.

12.4.3 Digital Picture Exchange (DPX)

The DPX format was developed several years ago to support the transfer of uncompressed images between telecine machines. It was later used for synthetic image file transfers. DPX was standardized in Fertuary 18, 1994 as SMPTE 268M [DPX].

The images that make up a DPX file are uncompressed video. The format uses metadata to specify the details of the image size, image shape, sample (pixel) information, and some fixed metadata. It is extensible and allows insertion of user-defined metadata. DPX does not support compressed video types or other features that are important parts of some of the new and proposed formats.

12.4.4 General Exchange Format (GXF)

The GXF, SMPTE 360M, is in use in many broadcast facilities today. The active audio and compressed video material is sent in a time-multiplexed stream. A frame lookup table was designed to optimize partial file retrieve and frame lookup operations. An EDL, user data (metadata) and a frame lookup table precede the content. Material that is currently not in use can be included and transferred after the active material.

GXF user data transports metadata that can be bound to the entire composition, a track or a segment within a track. The metadata is encoded in a manner that allows the encapsulation of KLV, XML, or other metadata formats.

This format is based on packets each containing a frame of video or about a half second of audio or several seconds of time codes. Each packet has a short header. This header specifies the packet type, identifies the track, and specifies a frames location within the track.

12.4.5 Advanced Authoring Format (AAF)

The AAF, is an industry-driven, cross-platform, file format that allows the interchange of data between multimedia authoring tools. It simplifies project management, saves time and preserves valuable information that was often lost in the past when transferring content between applications (see Figure 12.11).

Figure 12.11. The scope of the Advanced Authoring Format (AAF).

AAF can be used to interchange essence data and metadata. Essence data consists of picture, sound, and other forms of data that can be directly perceived. Metadata is data that describes essence data, performs some operations on essence data, or provides supplementary information about the essence data. Much of the creative effort that goes into a multimedia program is represented by metadata. This includes section transitions, special effects, and synchronization of those transitions and effects with images and sounds.

The post production process frequently uses many different devices. Video editing is performed on one system, special effects may be rendered on a different device, and audio work uses additional systems. AAF files can be interchanged among all of these devices. AAF's ability to save and interchange post-production projects as work in progress is key. In addition to its rich editing features, AAF supports the SMPTE metadata model.

12.4.5.1 AAF Association

The AAF Association is an independent nonprofit organization incorporated January 2000 to foster implementation of AAF. It is open to all users and manufacturers, broadcasters and post-production organizations [AAFASSOC].

The AAF Association has released a file format, an API specification and an SDK. The format specification and SDK were derived from the OMF initiative. The AAF SDK includes a reference implementation of the AAF API. This implementation can be used as an implementation baseline that promises to achieve interpretability between derived implementations . Adoption has been progressing swiftly. As of April 7th, 2000, adopters included BBC, CNN, Turner, 4MC and US NIMA. Manufacturers include Avid, Discreet, Matrox, Microsoft, Pinnacle, Sony, Quantel, Panasonic, AIST and Leitch.

12.4.5.2 AAF Specifications

The major parts of AAF are as follows :

• Object Specifications : It defines a structured container for storing essence data and metadata, using an object-oriented model. The object specification defines the logical contents of the objects and the rules for how the objects relate to each other.

• Container Specifications : It describes how each object is stored on disk. It uses Structured Storage, a file storage system developed by Microsoft, to store the objects on disk.

• SDK and Reference Implementation : It is an object-oriented programming toolkit and documentation which allow client applications to access the data stored in an AAF file. The It is a platform-independent toolkit provided in source form. The AAF SDK has been built and tested on several reference platforms including Windows 2000, MacOS, Irix and Linux. The SDK is housed on SourceForge.net, a large open source development Web site where the SDK can be freely downloaded.

12.4.5.3 AAF Software Architecture

The AAF SDK has a layered design, consisting of a public API, a reference implementation of the AAF object model (known as the data model manager), an object manager, and a storage system (see Figure 12.12). This design allows the possibility of alternative public APIs or storage systems in the future.

• Public API : The public API is the aspect of the Data Model Management (DMM) that all client applications see, and that treats all potential clients equally [AAFAPI]. It is written in IDL, to permit bindings to different languages (e.g., C, C++) and object brokers . It provides basic services: persistence (save and restore), transaction (add, modify, delete), accessories (get, set), and navigation (traversal, iteration, query). It provides clear mechanisms for extension of the data model, so that new object types can be linked into the API without causing revision or recompilation of the kernel software. Although Microsoft's COM is employed by the SDK, it also includes a minimal implementation of COM for use on non-Microsoft platforms.

• Data Model Manager : Beneath the public API there are various interfaces and implementation helper functions that are not expected to be directly called by the client. It is here that much of the design value of the DMM is concentrated. One of the benefits of using IDL to define the public API is that the unpublished implementation details are defined separately, reducing the temptation for clients to use an internal function and risk less than full error checking.

• Object Manager : The Object Manager (OM) provides the basic functions of Saving and Restoring objects and sub-objects and maintaining the relationships between them. The architecture separates DMM from the generic OM. The interface between these two subsystems is not public; the DMM interface exposes the OM interface polymorphically through the DMM API.

• Storage System : The Storage System underlying the OM is normally one of the file systems provided by the OS. In the AAF SDK, this function is provided by Microsoft Structured Storage (MSS). MSS refers to a data storage architecture that uses a file system within a file architecture. This container format is a public domain format, allowing interested parties to add future developments or enhancements in a due process environment. Microsoft has specifically upgraded the core technology compound file format on all platforms (Windows, MacOS, UNIX variants) to address the needs of AAF.

• Operating System : Underlying all the other subsystems is the Operating System. One of the challenges in designing the DMM and OM was to keep them separable from the Operating System, to serve the cross-platform interpretability requirements of the clients.

Figure 12.12. AAF SDK Layered Architecture.

12.4.5.4 AAF Object Model Capabilities

The AAF defines a number of object. Table 12.31 summarizes common AAF classes. The object model has the following capabilities:

• It encapsulates essence and metadata in objects. The model defines objects that allow an application to determine the format of the essence and to determine what conversions, if any, it needs to apply to the essence to process it.

• It provides a synchronization framework in which the essence and metadata are kept aligned. It describes the format of essence that contains interleaved streams enabling applications to synchronize separate streams of essence that were originally created in synchronization and derived from a single source, such as film, audio tape, and videotape.

• It describes the derivation of essence from the original media sources. For example, this mechanism allows applications referencing tape time-code and film edge-code that correspond to the essence, and enables regenerating essence from the original media.

• It describes compositions and transformations. Compositions contain information about how sections of essence should be combined in sequence, how to synchronize parallel tracks of sequences, and how to alter sections of essence or combine sections of essence by performing effects on the essence.

• It provides a mechanism to define new proprietary classes or to add optional information to existing classes. This mechanism allows applications to store additional information in an interchange file without restricting the interchange of the information specified by AAF.

12.4.5.5 AAF Built-In Classes

The AAF API defines a base set of built-in classes (see Table 12.31). These built-in classes can be used to interchange a broad range of data between applications, but applications may have additional forms of data that cannot be described by the basic set of built-in classes. The AAF extension mechanism enables defining new effects, new kinds of metadata, and new kinds of essence data.

Table 12.31. Common AAF Object Model Built-in Classes

AAF class	Function
Header	Provides file-wide information and contains the Identification(s), Dictionary and ContentStorage. There is one Header per file.
Identification	Provides information about the application(s) that created or modified the file.
Dictionary	Contains DefinitionObjects, that is definitions of the Classes, Types, Effects and Parameters used in the file.
Package	Describes composition of multiple material essence or physical media. Packages have names descriptions, and a unique ID.
PluginDefinition	Identifies code objects that provide an implementation for a DefinitionObject, e.g., a codec providing an implementation for a CodecDefinition.
ContentStorage	Contains the Packages and EssenceData objects in the file. There is one ContentStorage object per file.
EssenceData	Contains essence associated with a Package.
EssenceDescriptor	Describes the format of essence associated with a File Source Package or media associated with a Physical Source Package.
Locator	Provides information to help find a file that contains the essence.
TaggedValue	Specifies a user-defined tag, key and value.
KLVData	Specifies user data with a Key (SMPTE label), Length and Value.
Transition	Specifies that the two adjacent Segments should be overlapped when they are played and the overlapped sections should be combined using the specified Effect.
Parameter	Specifies a control argument for an effect.
ControlPoint	Specifies a value and a time point and is used to specify an effect control point.

The package object is a key pillar for a top-down understanding of the AAF model. It is an object with a universal (globally unique) identifier. Packages describe composition, essence or physical media. Packages have names and descriptions, but are primarily identified by a unique identifier, which is called a PackageID (may also be a basic SMPTE UMID). Table 12.32 lists four kinds of packages that are commonly used.

A package can describe more than one kind of essence. For example, a Package can have audio, video, still image and time-code data. Composition packages do not directly reference the essence data that they combine to form a program. Composition packages reference the basic essence data with source clips that identify the material package and file source package that describe the essence data.

A package has one or more slots. Each slot can describe only one kind of essence data. A slot can be referenced from outside of the package. For example, a package can have two Slots with audio, one slot with video, three Slots with still images, and two slot with time-code. Each slot in a package has a SlotID that is unique within the package. To reference the essence data in a slot, the PackageID and the SlotID are used.

Table 12.32. AAF Package Types

Package Type	Function
Composition	Describes creative decisions on how to combine or modify essence: Decisions on order of essence data; Decisions on placement of essence data; Decisions on effects that modify or combine essence data.
Material	Collect and possibly synchronize related essence data; provides indirect access to essence data, which is independent of storage details.
File Source	Provides direct access to and describes the format of digital essence data that is (or can be) stored in a computer file.
Physical Source	Describes physical media such as a videotape or film.

Table 12.33 lists three kinds of slot commonly used in the AAF object model. A slot has a segment describing an essence element. The segment subclasses include the following:

• Source clip which references a section of a slot in another Package; for example a source clip in a timeline slot can describe video data.

• Sequence which specifies that its set components are arranged in a sequential order; in a timeline slot, the components are arranged in sequential time order.

• Effect which specifies that either two or more segments should be combined using a specified effect or that one Segment should be modified using a specified effect.

• Filler which defines an unspecified value for its duration.

Table 12.33. AAF Package Slot Types

Slot Type	Function
Static	Describes essence data that has no specific relationship to time, such as static images or static text.
Timeline	Describes essence data that has a fixed or continuous relationship with time, such as audio, film, video, time-code, and edge-code.
Event	Describes essence data that has an irregular relationship with respect to time, such as GPI events, MIDI, interactive events, and user annotation associated with specific times.

12.4.6 Material Exchange Format (MXF)

In 1999, the Pro-MPEG forum recognized that the server industry had no common file format; each company developed their own and most were proprietary. Further, AAF introduced client complexity which can have a significant impact within embedded systems, such as cameras , where processing and memory resources may be scarce . An MXF task force was established to address this issue through the development of a file format that is intended to interoperate with AAF. The Pro-MPEG forum2 submitted MXF to SMPTE for standardization. The first letter ballot was started in March 2001. As of September 2001, MXF has not successfully completed a Technology Committee letter ballot. This is not unusual for a complex standard.

Addressing this issue is the MXF, which is being developed jointly by the Pro-MPEG forum and the AAF Association. MXF reuses a subset of the AAF object model. The material related components are included in the MXF but the components dealing with compositions, effects and the in-file dictionary are removed [MXF].

12.4.6.1 Features and Applicability

MXF is streamable. By using SMPTE 336M KLV coding, instead of Structured Storage, and applying other rules on placement of data within the stream, MXF provides capabilities such as playing while recording and operation with isolated sections of streams. By replacing Structured Storage however, the AAF feature of in place editing of existing files is lost.

MXF is designed as the basic architecture and structure for a wide base of broadcast applications including acquisition, contribution, operations, archives, and digital cinema. Although file exchanges (transfers) between AAF devices usually use the native AAF file format., the key application of MXF is to transfer of files into and out of AAF-based editing systems.

12.4.6.2 MXF File Structure

MXF is designed for servers, tape streamers and digital archives. An MXF file is a wrapper with one or more audio visual containers that contains a complete material sequence such as clips and program segments. Essentially, MXF is a wrapper of stream-based essence containers having comprehensive metadata header (see Figure 12.13). Each essence container is an interleaved streamable file comprised of audio, video and data essence plus temporal metadata essence. Essence containers can be based on one of several basic kinds, including MPEG, digital video and uncompressed formats.

Figure 12.13. MXF File Structure.

Starting an MXF file is the file header which provides information about the file as a whole. The header contains structural description metadata, which describes the contents of the MXF body and allows MXF files to be directly read and written by AAF compliant systems. It also contains descriptive metadata, which provides editorial metadata. An optional Index Table may be used to provide random access capabilities and improve access times. The file's body, usually contains more than 99% of the data, and comprises the essence components in a stream data container. Each of the containers can contain a single essence stream or an interleave of multiple essence streams. Finally, the footer terminates the file. The essence containers may be playable without either the header or footer. Each of which can be a single essence stream or an interleave of multiple essence streams.

12.4.6.3 Interoperability

To facilitate interpretability, a common wrapper is used to access essence container streams. An essence format-independent wrapper preserves production metadata for any compression format (if compression is used). It supports random access and partial file transfers (e.g., moving a sports or news highlight without transferring the whole file). It enables access and use before completion of transfers (e.g., for playing, accessing and editing). It supports cuts-only edits for versioning for library and archive applications.

12.4.6.4 Operational Patterns

The MXF specification also includes operational patterns that describe several layers of features. The lowest layer patterns are simple streams and higher levels add editing features. One of the patterns is a metadata-only file.

12.4.7 ZIP

The ZIP file format is a data integration format that supports a number of compression methods . It does not specify any particular interface to a file system or character sets or encodings (except for file names and comments, which are optional). For an official specification, see ATSC DASE-1 Part 5 [25] (which is essentially a reproduction of APPNOTE.TXT, the ZIP File Format Specification, Version 4.0, May 1, 2001, PKWARE, Inc.).

A ZIP archive resource consists of a list of file entries followed by a directory. Each entry in the list describes a file and comprises a header, data and trailer (see Table 12.34). The directory is a list of entries describing the directory structure of the files listed (see Table 12.35).

Table 12.34. Zip File Structure Including Header, Body, and Trailer

Field Name	Bits	Description
HeaderSignature	32	Indicates the start of the entry header; it is set to 0x04034b50.
headerVersion	16	A field that indicates the version of the archive decoder needed to decode the file. The lower byte indicates the version of the encoding system, with the major version being the integer value divided by 10, and minor version being value modulo 10.
headerFlags	16	A general purpose bit flag.
headerMethod	16	The compression method applied for the file referenced by this entry (see Table 12.32).
headerLastModTime	16	The last modified time of the entry as follows: bits 4 “0 denote seconds divided by two (0 “29), bits 10 “5 denote minutes (0 “59), and bits 15 “11 denote hours (0 “23).
headerLastModDate	16	The last modified date of the entry as follows: bits 4 “0 denote day of month (1 “31), bits 8 “5 denote month (1 “12), and bits 15 “9 denote years relative to 1980. The value of this field conforms to the format used by the MS-DOS FILEINFO.fiFileDate field (the MS DOS date format).
headerCRC32	32	If bit 3 of the headerFlags is set, then this field is set to zero; otherwise , it is set to a value representing a checksum of the uncompressed entry data.
headerCompressedSize	32	If bit 3 of the headerFlags is set, then this field is set to zero; otherwise, it is set to the length of the compressed entry data.
HeaderUncompressedSize	32	If bit 3 of the headerFlags is set, then this field is set to zero; otherwise, it is set to the length of the uncompressed entry data.
headerPathnameLength	16	The length of the headerPathnameBytes field.
headerhExtraLength	16	The length of the headerExtensionBytes field.
HeaderPathnameBytes	*	This field specifies the name of the entry optionally preceded by a relative path . The value of this field consists of a non- terminated character string (its length is specified by headerPathnameLength) that conforms to the following syntax: pathname : [ rel_path ] name rel_path : path_segment [ "/" path_segment ]* "/" path_segment : pchar+ name : pchar+ pchar : { any non-control character except '/' }
HeaderExtraBytes	*	Used for extension fields. If present (i.e., headerhExtraLength is non-zero ), the value of this field is exposed to an application for application-defined usage; otherwise, the field is ignored.
DataBytes	*	The entry data after having been compressed in accordance with the compression method specified by the headerMethod field.
TrailerSignature	32	Indicates the start of the entry header; it is set to 0x08074b50. This field is not defined by the specification, but it is commonly used as a reliable means of locating the beginning of the trailer.
trailerCRC32	32	A field whose value represents a checksum of the uncompressed entry data.
trailerCompressedSize	32	The length of the compressed entry data.
trailerUncompressed Size	32	The length of the uncompressed entry data.

Table 12.35. Zip Directory Structure

Field Name	Bits	Description
dirMarker	32	Indicates the start of the entry header; it is set to 0x02014b50.
dirVersionEncoder	16	A field that indicates the version of the archive encoder needed to decode the file. The lower byte indicates the version of the encoding system, with the major version being the integer value divided by 10, and minor version being value modulo 10.
dirVersionDecoder	16	A field that indicates the version of the archive decoder needed to decode the file. The lower byte indicates the version of the decoding system, with the major version being the integer value divided by 10, and minor version being value modulo 10.
dirFlags	16	A general purpose bit flag.
dirMethod		The compression method applied for the file referenced by this entry (see Table 12.32).
dirLastModTime	16	The last modified time of the entry as follows: bits 4-0 denote seconds divided by two (0-29), bits 10-5 denote minutes (0-59), and bits 15-11 denote hours (0-23).
dirLastModDate	16	The last modified date of the entry as follows: bits 4-0 denote day of month (1-31), bits 8-5 denote month (1-12), and bits 15-9 denote years relative to 1980. The value of this field conforms to the format used by the MS-DOS FILEINFO.fiFileDate field.
dirCRC32	32	If bit 3 of the headerFlags is set, then this field is set to zero; otherwise, it is set to a value representing a checksum of the uncompressed entry data.
dirCompressedSize	32	If bit 3 of the headerFlags is set, then this field is set to zero; otherwise, it is set to the length of the compressed entry data.
dirUncompressedSize	32	If bit 3 of the headerFlags is set, then this field is set to zero; otherwise, it is set to the length of the uncompressed entry data.
dirPathnameLength	16	The length of the dirPathnameBytes field.
dirExtraLength	16	The length of the dirExtensionBytes field.
dirCommentLength	16	The length of the dirCommentBytes field.
dirSegment	16	The number of the archive segment, e.g., disk, within that this entry starts.
dirInternalFileAttribs	16	This field is used to indicate attributes about the file. If bit 0 is set, then the entry data is believed by the encoder to be of textual nature (i.e., encoded characters); if it is clear, then the entry data is believed by the encoder to be of another binary form. All other bits are reserved.
DirExternalFileAttribs	16	These attributes vary as the encoder environment varies. The value of this field may be zero, indicating that no external attributes were specified. Encoders may assign this field a value of zero, and decoders may ignore this field. If the encoding environment is MS-DOS, then the lower byte corresponds to the value of the FILEINFO.fiAttribute byte.
dirOffset	*	The offset from the start of the segment, e.g., disk, in which the entry appears to the entry's header.
dirPathnameBytes	*	This field specifies the name of the entry optionally preceded by a relative path. The value of this field consists of a non-terminated character string (its length is specified by headerPathnameLength) which conforms to the following syntax: pathname : [ rel_path ] name rel_path : path_segment [ "/" path_segment ]* "/" path_segment : pchar+ name : pchar+ pchar : { any non-control character except '/' }
dirExtraBytes	*	Used for extension fields. If present (i.e., headerhExtraLength is non-zero), the value of this field is exposed to an application for application-defined usage; otherwise, the field is ignored.
dirCommentBytes	*	Used for specifying a comment on that entry.
dataBytes	*	The entry data after having been compressed in accordance with the compression method specified by the headerMethod field.
dirTrailerSignature	32	Indicates the start of the entry header; it is set to 0x06054b50. This field is not defined by the specification, but it is commonly used as a reliable means of locating the beginning of the trailer.
dirSegmentCount	16	The number of the archive segments (e.g., disks) within which the directory trailer starts.
dirStartSegment	16	The index segment (e.g., disk) within which the directory starts.
dirEntriesCount	32	The total number of directory entries in the archive, namely the number of files archived.
dirSize	32	The length of the directory data.
dirOffset	32	The offset from the stard of the segment for this directory.
DirTrailerCommentLength	16	The length of the dirTrailerCommentBytes field.
dirTrailerCommentBytes	*	Used for specifying a comment applicable to the entire directory.

Manipulating ZIP files could be done using the Java JDK 1.2.2 APIs. The ZIP class is used to access a ZIP file, and the ZIPEntry class is used to access individual entries. It is important to note that the ZIP file format is extensible through ZipEntry.getExtra() and ZipEntry.setExtra() , which enable accessing and updating the headerExtraBytes and dirExtraBytes fields.

A directory may be signed by a digital signature. The signature structure starts with 0x05054b50, followed by the sigLength field specifying the length of the signature, followed by the signature data bytes.

Table 12.36. The List of Compression Methods

Method	Description
	Entry is stored without compression.
1	Entry is shrunk.
2	Entry is reduced with compression factor 1.
3	Entry is reduced with compression factor 2.
4	Entry is reduced with compression factor 3.
5	Entry is reduced with compression factor 4.
6	Entry is imploded.
7	Reserved for use with tokenizing compression algorithm.
8	Entry is deflated.
9	Reserved for use with enhanced deflation algorithm.
10	Reserved for use with date compression library imploding algorithm.
11 “65535	Reserved.

Method 1, shrinking, is a Dynamic Ziv-Lempel-Welch compression algorithm with partial clearing. The initial code size is 9 bits; the maximum code size is 13 bits. Shrinking differs from conventional Dynamic Ziv-Lempel-Welch implementations as follows:

1. The code size is controlled by the compressor, and is not automatically increased when codes larger than the current code size are created (but not necessarily used). When the decompressor encounters the code sequence 256 (decimal) followed by 1, it should increase the code size read from the input stream to the next bit size. No blocking of the codes is performed, so the next code at the increased size should be read from the input stream immediately after where the previous code at the smaller bit size was read. The decompressor should not increase the code size used until the sequence {256, 1} is encountered .

2. When the table becomes full, total clearing is not performed. Rather, when the compressor emits the code sequence {256, 2} (decimal), the decompressor should clear all leaf nodes from the Ziv-Lempel tree, and continue to use the current code size. The nodes that are cleared from the Ziv-Lempel tree are then reused, with the lowest code value reused first, and the highest code value reused last. The compressor can emit the sequence {256, 2} at any time.

Methods 2 through 5 indicate that a reduction compression is applied to the entry data. The reduction algorithm is a combination of two distinct algorithms. The first algorithm compresses repeated octet sequences, and the second algorithm takes the compressed stream from the first algorithm and applies a probabilistic compression method. The probabilistic compression stores an array of 255 sets, corresponding to each possible ASCII character. Each of these sets contains between 0 and 32 characters. The sets are stored at the beginning of the data area for a reduced entry, in reverse order.

Method 6, implosion, is a combination of two distinct algorithms. The first algorithm compresses repeated octet sequences using a sliding dictionary. The second algorithm is used to compress the encoding of the sliding dictionary output, using multiple Shannon-Fano prefix-code trees. These trees are stored at the start of the compressed entry. The number of trees stored is defined by bit 2 of the entry flags field; a 0 bit indicates two trees stored, a 1 bit indicates three trees are stored. If three trees are stored, the first Shannon-Fano tree represents the encoding of the literal characters, the second tree represents the encoding of the Length information, and the third represents the encoding of the distance information. When two Shannon-Fano trees are stored, the Length tree is stored first, followed by the Distance tree.

Although Shannon-Fano and Huffman coding produce similar results, the Huffman coding is guaranteed to be at least as efficient as Shannon-Fano method, and thus has become the preferred coding method.

12.4.8 JAR

iTV applications can be packaged as Java Archive (JAR) files. The JAR file format is essentially the ZIP file format extended with a manifest file containing metadata about each file in the archive. The manifest is a special file that can contain information about the files packaged in a JAR file. There can be only one manifest file in an archive, and it always has the pathname META-INF/MANIFEST.MF .

12.4.8.1 JAR for JDK 1.1

The JAR file format was introduced JDK 1.1 whose components often supported by iTV execution environment. With the JDK 1.1, the manifest file would look something like this:

 Manifest-Version: 1.0 Name: java/math/BigDecimal.class SHA1-Digest: TD1GZt8G11dXY2p4olSZPc5Rj64= MD5-Digest: z6z8xPj2AW/Q9AkRSPF0cg== Name: java/math/BigInteger.class SHA1-Digest: oBmrvIkBnSxdNZzPh5iLyF0S+bE= MD5-Digest: wFymhDKjNreNZ4AzDWWg1Q== 

The manifest has entries for each file contained in the archive, including the files' pathnames and digest values, according to the following rules: Any headers that immediately follow a name header without any intervening blank lines pertain to the file specified by the Name header. The Name: header contains a relative path to the file being signed within the archive, or an absolute URL referencing data outside the archive. Multiple digest algorithms may be listed (and corresponding (*)-Digest: lines must exist for each one listed); at least one must be present. Two entries for the same file may not appear in the manifest file. When duplicate entries do occur, only the first is recognized. Headers that are not understood are ignored. Such headers may include policy information used by applications.

12.4.8.2 JAR for JDK 1.2

The format of the manifest file changed between versions 1.1 and 1.2. JDK 1.2 extends the JDK 1.1 format with the features of download extensions, package sealing , and versioning.

Download Extensions

Version 1.2 of the JDK introduced download extensions These are JAR files that are referenced by the manifest files of other JAR files. In a typical situation, an iTV Xlet (i.e., Java TV Applet) is bundled in a JAR file with a manifest references a JAR file (or several JAR files) that serves as an extension for the purposes of that Xlet. Extensions may reference each other in the same way.

Download extensions are specified in the Class-Path header field in the manifest file of an applet, application, or another extension. A Class-Path header might look like this, for example:

 Class-Path: application1.jar images.jar beans.jar 

With this header, the classes in the files application1.jar , info .jar , and beans.jar serve as extensions for purposes of the Xlet. The URLs in the Class-Path header are given relative to the URL of the JAR file of the Xlet.

Package Sealing

Version 1.2 of the JDK introduced the package sealing option. A package sealed within a JAR file, should have all classes defined in that package are archived in the same JAR file. It is appropriate to seal packages to ensure version consistency among the classes, or for security reasons. To seal a package, there is a need to add a Name header for the package, followed by a Sealed header, similar to this:

 Name: myCompany/myPackage/ Sealed: true 

The Name header's value is the package's relative path name, which ends with a '/' to distinguish it from a filename. Any headers following a Name header, without any intervening blank lines, apply to the file or package specified in the Name header. In the this example, because the Sealed header occurs after the Name: myCompany/myPackage header, with no blank lines between, the Sealed header is interpreted as applying (only) to the package myCompany/myPackage.

Package Versioning

The package versioning feature introduces new manifest headers capable of specifying versioning information. One set of such headers can be assigned to each package. The versioning headers should appear directly beneath the Name header for the package. This example shows all the versioning headers:

 Name: broadcast/util/ Specification-Title: "Broadcast Utility Classes" Specification-Version: "2.1" Specification-Vendor: "ACME, Inc.". Implementation-Title: "broadcast.util" Implementation-Version: "build1381" Implementation-Vendor: "ACME, Inc." 

12.4.8.3 JAR Signature Files

Each signer is represented by a signature .SF file that starts with the version number of the Java manifest signature standard. Next the file specifies information supplied by the signer but not specific to any particular path name should follow the version information.

The format of the JAR signature file is similar to the format of the manifest file. It consist of sections that are essentially lists of names, all of which must be present in the manifest file. Only the 'Name' headers are required, and extra headers may be included. A digest, if present, is a digest of the entry in the manifest file. To allow subsets of the JAR to be signed, paths or URL's appearing in the manifest file but not in the signature file are not used in calculation of the digest.

The following is a simple example of a signature .SF file:

 Signature-Version: 1.0 Name: common/class1.class MD5-Digest: (base64 representation of MD5 digest) Name: common/class2.class MD5-Digest: (base64 representation of MD5 digest) 

A digital signature is a signed version of the .SF (signature instructions) file; it is a binary files not intended to be interpreted by humans . Digital signature files have the same filename as the .SF file but different extension. The extension varies depending on the type of digital signature:

 .RSA      (PKCS7 signature, MD5 + RSA) .DSA      (PKCS7 signature, DSA) .PGP      (Pretty Good Privacy Signature) 

A signature is typically verified when the manifest is parsed. This verification can be remembered , for efficiency. This only validates the signature directions themselves , not the actual archive files.

To validate a file, a digest value in the signature file is compared against a digest calculated against the corresponding entry in the manifest file. Then, a digest value in the manifest file is compared against a digest calculated against the actual data referenced in the Name: header.

Interestingly, it is technically possible that different entities may use different signing algorithms to share a single signature file. This, however, may result in predictable behavior as it violates the specification published by Sun Microsystems.