9.9 Synchronized Multimedia Integration Language (SMIL)

SMIL was designed for visually annotating videos , adding layers of content customized to the specific locale and preferences of each viewer without altering the video itself [SMIL].

Work on SMIL dates back to 1998. After initial enthusiasm in multimedia circles about developing kiosks and similar applications, it virtually disappeared from the center stage. SMIL 2.0, released in August 2001, specifies a collection of reusable modules of animation, layout, and synchronization, which can be independently implemented and used in other languages.

In SMIL 2.0, the root element is <smil/> , and its children are <head/> and <body/> . The <head/> element needs to contain the content insertion layout, whereas the <body/> element contains a tree whose nodes may reference the content to be inserted. Each element is part of exactly one module. For example, <region/> is part of the layout module, and <animateMotion/> is part of the animation module. Two profiles are defined, the language profile, and the base profile (for small devices). The basic layout-only SMIL controls the layout of the elements <ref/> , <animation/> , <audio/> , <img/> , <video/> , <text/> and <textstream/> media elements. It is consistent with the visual rendering model defined in CSS2. It reuses the formatting properties defined by the CSS2 specification, and newly introduces the fit attribute.

SMIL 2.0 modules are designed to be reusable as parts of other XML vocabularies, allowing implementing vendors as well as standards initiatives to decide to implement only parts of SMIL. Examples include the merging of XHTML and the SMIL timing module with the declarative animation in Scalable Vector Graphics (SVG), implemented by Internet Explorer 6.0 and Adobe SVG Viewer 2+ respectively.

9.9.1 Annotating Footage

9.9.1.1 Layout

Effective layout arranges the annotations to direct the attention of the viewer to the annotated information. Layout could be accomplished in a variety of ways, including the introduction of a sense of depth, establishment of contrast between elements, or sequencing elements in an intuitive manner. Figure 9.25 illustrates a layout and the code implementing it.

Figure 9.25. Layout areas for the boxing video annotation example.

9.9.1.2 Adding Annotation Elements

Consider starting with the addition of opening credits in the form of two simple GIF files. Each GIF should appear for 3 seconds, and the two GIFs should be sequenced . To achieve this the media is referenced using <img/> elements, and grouped in a <seq/> (for sequence) element (see Example 9.1). The elements that can be used within the sequence are as follows :

<img/> : JPEG or GIF images work on all current players. GIF89 transparency is supported in any current player, non-interlaced GIF preferred in RealPlayer.
<video/> : MPEG, AVI, RealVideo and other formats for motion clips are included using this element. The support for different video formats is specially dependent on the player.
<text/> : Static text. HTML is not supported in any SMIL 1.0 players.
<audio/> : Used for audio clips including WAV, AU and RealVideo.
<animation/> : Used for animation clips; limited support.
<ref/> : Any clip not covered by other elements but supported by the player.

It is important to realize that not all SMIL players support all those tags. The incomplete support for some media types in many players is one of the reasons for the slow adoption of SMIL. For example, many players do not support GIF transparency correctly. Similarly, all SMIL 1.0 players and some SMIL 2.0 players exclude support for HTML as a media element. RealPlayer and QuickTime support additional elements for including vendor-specific "smart text" for effects like tickers and basic formatting.

Example 9.1 Coding a GIF sequence for presenting credits.

  <smil>   <head>   <!-- Layout exactly as in Figure 9.25 -->   </head>   <body>   <seq>   <img src="Intro-Names.gif" region="video" dur="3s"/>   <img src="Intro-Date.gif" region="video"  dur="3s"/>   </seq>   </body>   </smil>

9.9.2 Media Objects

Media objects are classified into continuous media, having an intrinsic duration, and discrete media, that do not have intrinsic duration (see Table 9.7). When playing back a media object, the player does not derive the exact type of the media object from the name of the media object element. Instead, it relies solely on other sources about the type, such as type information contained in the type attribute, or the type information communicated by a server or the operating system. Authors should make sure that the group into which the media object falls (animation, audio, img, video, text, or textstream) is reflected in the element name to further increase readability. When in doubt about the group of a media object, authors should use the generic <ref/> element.

Table 9.7. Media Objects Supported by SMIL 1.0 Basic Layout

Media Object	Classification	Media Object	Classification
Audio	Continuous	Text	Discrete
Video	Continuous	Textstream	Continuous
Animation	Continuous	Ref	Continuous, discrete
Img	Discrete

9.9.3 Flow Control Elements

The flow control elements are specified in Table 9.8. Authors need to ensure that their child elements are supported by SMIL, the media-type can be decoded, and all the test attributes (whose name start with 'system-' as described in Table 9.9) evaluate to true.

Table 9.8. Flow Control Elements

Media Object	Classification
`seq`	The children of a "seq" element form a temporal sequence.
`par`	The children of a par element can overlap in time. The textual order of appearance of children in a par has no significance for the timing of their presentation.
`switch`	The switch element allows an author to specify a set of alternative elements from which only one acceptable element should be chosen .

9.9.3.1 The Switch Element `<switch/>`

The switch element allows an author to specify a set of alternative elements from which only one acceptable element should be chosen. This capability subsumed the http content-negotiation capabilities as it enables the use of test attributes with names that start with 'system-'.

A child element of <switch/> is selected by evaluating the elements in the order in which they occur in the switch element. The first acceptable element is selected at the exclusion of all other elements within the switch. Authors should order the alternatives from the most desirable to the least desirable. Further, authors should place a relatively fail-safe alternative as the last item in the <switch/> so that at least one item within the switch is chosen (unless this is explicitly not desired). Unpredictable behavior may occur when the test-attributes of all children of the <switch/> element fail.

In one common scenario, implementations may wish to allow for selection via a system- bitrate attribute on elements (see Example 9.2). The media player evaluates each of the options, looking for an acceptable bitrate given the known characteristics of the link between the media player and media server. In another common scenario, an audio resource is available both in French and in English. Based on the user 's preferred language, the player can choose one of these audio resources (see Example 9.3). The presentation may contain alternative parts designed for screens with different resolutions and bit-depths (see Example 9.4). The player can choose one of the alternatives. It is also possible to distinguish between captions and stock tickers (see Example 9.5).

Example 9.2 Choosing between content with different bitrate.

  ...   <par>   <text .../>   <switch>   <par system-bitrate="40000"> ... </par>   <par system-bitrate="24000"> ... </par>   <par system-bitrate="10000"> ... </par>   </switch>   </par>   ...

Example 9.3 Choosing between audio resources in different languages.

  ...   <switch>   <audio src="joe-audio-french" system-language="fr"/>   <audio src="joe-audio-english" system-language="en"/>   </switch>   ...

Example 9.4 Choosing between content written for different screens.

  ...   <par>   <text .../>   <switch>   <par system-screen-size="1280X1024" system-screen-depth="16">   ...   </par>   <par system-screen-size="640X480" system-screen-depth="32">   ...   </par>   <par system-screen-size="640X480" system-screen-depth="16">   ...   </par>   </switch>   </par>   ...

Example 9.5 Distinguishing caption tracks from stock tickers.

  ...   <seq>   <par>   <audio      src="audio.rm"/>   <video      src="video.rm"/>   <textstream src="stockticker.rtx"/>   <textstream src="closed-caps.rtx" system-captions="on"/>   </par>   </seq>   ...

9.9.4 The `<a/>` Element

The functionality of the SMIL <a/> element is very similar to the functionality of the <a/> element in HTML 4.0 [HTML40] . It has an href attribute. For synchronization purposes, the <a/> element is transparent, i.e., it does not influence the synchronization of its child elements. <a/> elements may not be nested. A show attribute is used with one of the following values:

Replace : This is the default value. The current presentation is paused at its current state and is replaced by the destination resource. If the player offers a history mechanism, the source presentation resumes from the state in which it was paused when the user returns to it.
New : The presentation of the destination resource starts in a new context, not affecting the source.
Pause : The source presentation is paused at its current state, and the destination resource starts in a new context. When the display of the destination resource ends, the source presentation resumes from the state in which it was paused.

Example 9.6 illustrates a link from an element in Presentation A to the middle of another Presentation B , that plays Presentation B starting at the element with id=" next " .

Example 9.6 Usage of the Ancho Element.

  <!-- Presentation A: -->   ...   <a href="http://www.xyz.tv/presentationB#next">   <video src="rtsp://foo.com/graph.imf"/>   </a>   <!-- Presentation B: at http://www.xyz.tv/presentationB#next -->   ...   <seq>   <video src="rtsp://foo.com/graph.imf"/>   <par>   <video src="rtsp://foo.com/timbl.rm" region="l_window"/>   <video  id="next"  src="rtsp://foo.com/v1.rm" region="r_window"/>   <text src="rtsp://foo.com/caption1.html" region="l_2_title"/>   <text src="rtsp://foo.com/caption2.rtx" region="r_2_title"/>   </par>   </seq>

Table 9.9. SMIL Element Attributes

Attrbute Name	Description
`abstract`	A brief description of the content contained in the element.
`alt`	For players that cannot display a particular media-object, this attribute specifies alternate text. All media object elements should have an `alt` attribute with a meaningful description. Authoring tools should ensure that no element can be introduced into an SMIL document without this attribute.
`author`	Information about the content author.
`begin`	The time for the explicit begin of an element, and may contain two values: A delay value is a clock value measuring presentation time. Presentation time advances at the speed of the presentation. It behaves like the time-code shown on a counter of a tape-deck. An event value indicate that the element begins playing when a certain event occurs; the element generating the event is be in scope.
`clip-begin`	The clip-begin attribute specifies the beginning of a subclip of a continuous media object as offset from the start of the media object. The value consists of a metric specifier, followed by a time value whose syntax and semantics depend on the metric specifier . Allowed formats include SMPTE time stamp and Normal Play Time (NPT).
`clip-end`	The `clip-end` attribute specifies the end of a subclip of a continuous media object (such as audio, video or another presentation) that should be played . It uses the same attribute value syntax as the `clip-begin` attribute. If the value of `clip-end` exceeds the duration of the media object, the value is ignored, and the clip end is set equal to the effective end of the media object.
`copyright`	The copyright notice of the content contained in the element.
`dur`	This attribute specifies the explicit duration of an element. The attribute value can be a clock value, or the string "indefinite".
`end`	This attribute specifies the explicit end of an element. The attribute can contain the same types of attribute values as the `begin` attribute.
`fill`	The attribute can have the values "remove" and "freeze".
`id`	The element's identifier.
`longdesc`	This attribute specifies a link (URI) to a long description of a media object. This description should supplement the short description provided using the alt attribute. When the media object has associated anchors, this attribute should provide information about the anchor's contents.
`region`	This attribute specifies an abstract rendering surface (either visual or acoustic) defined within the layout section of the document. Its value is an XML identifier. If no rendering surface with this id is defined in the layout section, the formatting properties of this element are determined by the default layout.
`src`	The value of the src attribute is the URI of the media object
`system-bitrate`	This attribute specifies the approximate bandwidth, in bits per second available to the system. The measurement of bandwidth is application specific, meaning that applications may use sophisticated measurement of end-to-end connectivity, or a simple static setting controlled by the user. In the latter case, this could for instance be used to make a choice based on the users connection to the network. Typical values for modem users would be 14400, 28800, 56000 bit/s etc. The attribute can assume any integer value greater than 0.
`system-captions`	This attribute allows authors to distinguish between a redundant text equivalent of the audio portion of the presentation (intended for audiences such as those with hearing disabilities or those learning to read who want or need this information) and text intended for a wide audience. The attribute has the value "on" if the user has indicated a desire to see closed-captioning information, and it has the value "off" if the user has indicated that they don't wish to see such information. Evaluates to true if the value is "on", and evaluates to false if the value is "off".
`system-language`	The attribute value is a comma-separated list of language names as defined in [RFC1766]. Multiple languages may be listed for content that is intended for multiple audiences. Evaluates to "true" if one of the languages indicated by user preferences exactly equals one of the languages given in the value of this parameter, or if one of the languages indicated by viewer preferences exactly equals a prefix of one of the languages given in the value of this parameter such that the first tag character following the prefix is "-". Evaluates to "false" otherwise .
`system- overdub -or-caption`	This attribute is a setting which determines if users prefer overdubbing or captioning when the option is available. The attribute can have the values "caption" and "overdub". Evaluates to true if the user preference matches this attribute value. Evaluates to false if they do not match.
`system-required`	This attribute specifies the name of an extension. Evaluates to true if the extension is supported by the implementation, otherwise, this evaluates to false . In a future version of SMIL, this will be an XML namespace.
`system-screen-size`	Attribute values have the following syntax: screen-size-val ::= screen-height"X"screen-width Each of these is a pixel value with an integer value greater than 0. Evaluates to true if the SMIL playback engine is capable of displaying a presentation of the given size. Evaluates to false if the SMIL playback engine is only capable of displaying a smaller presentation.
`system-screen-depth`	This attribute specifies the depth of the screen color palette in bits required for displaying the element. The value is greater than 0. Typical values are 1, 8, 24 .... This attribute evaluates to true if the SMIL playback engine is capable of displaying images or video with the given color depth. It evaluates to false if the SMIL playback engine is only capable of displaying images or video with a smaller color depth.
`title`	It is strongly recommended that all media object elements have a `title` attribute with a meaningful description. Authoring tools should ensure that no element can be introduced into a SMIL document without this attribute.
`type`	MIME type of the media object referenced by the `src` attribute.

9.9.5 Conclusion

Media streaming is associated with a wide range of technologies: At the transport layer, there are MPEG-based and IP-based transports, at the media layer there are various decoding and encoding technologies (e.g., MP3), at the synchronization layer there are time models, and at the application layer there are various integration APIs (e.g., JMF, SMIL).

This chapter presented some of the issues involved. However, each aspect of the discussion could be further expanded deeper and deeper into an entire subfield. It was not the intention of this chapter to answer all the inevitable questions that arise when inquiring about those aspects. This chapter was only intended to create awareness of many of the issues and enable readers to perform effective subsequent search.