9.8 Java Media Framework (JMF)

JMF, currently at version 2.1, is Sun's initiative to bring time-based media processing to Java [JMF]. The JMF API enables JavaTV Xlets (possibly embedded within XHTML pages) to:

Play various multimedia files or streams acquired from broadcasts or a return channel.
Capture audio and video with a microphone and video camera, then store the data.
Provide Audio on Demand (AOD) and VOD features.
Support the formats of AU, AVI, MIDI, MPEG, QuickTime, and WAV.

Consider a scenario in which a stereo system player plays music from a CD. In this scenario, the CD provides the data encoding of the music. The signal generated from the decoded data is sent the speakers. The data stored on the music CD has been previously captured in a studio using microphones and other audio capture devices. The CD itself is a data source to the stereo system. The speakers serve as an output device. The JMF programming models for this scenario uses the object classes of DataSource , Capture Device , Format , DataSink , Player , Processor and Manager .

9.8.1 Data Source

A data source represents the media stream much like a music CD. In JMF, a DataSource object encapsulates the audio media, video media, DSM-CC data stream, or a combination of the two. A DataSource can be a file or an incoming stream; it encapsulates both the media location, e.g., file name or URL, the protocol, e.g., HTTP, and the software used to deliver the media, e.g., MP3 decoder. Once created, a DataSource can be fed into a Player to be rendered, with the Player unconcerned about where the DataSource originated or what its format was.

Media data can be obtained from various sources, such as local or network files, or live broadcasts. As such, DataSource s can be classified according to how a data transfer initiates:

Pull data source : The consumer of the data initiates the transfer and controls the data flow. The http:// and file:// protocols serve as examples for this type of data source.
Push data source : The producer of the data, i.e., capture device, initiates the data transfer and controls the data flow from a push data source. Push data source examples include MPEG-2 DSM-CC transport streams used in DTV broadcasts and cable programming.

9.8.2 Capture Device

A capture device represents the hardware used to capture data, such as a microphone or a camera. Capture devices can be categorized as either push or pull sources. For example, a still camera is a pull data source because it does not capture, record or otherwise consume the data until a shot is taken. In contrast, a microphone and a video camera transmits data continuously regardless of whether the data is recorded or otherwise consumed in any way.

9.8.3 Player

A Player takes as input a data source and renders it to a speaker or a screen, much like a CD Player reads a CD and outputs music to the speaker. A Player can have states, which exist naturally because a Player has to prepare itself and its data source before it can start playing the media. When a CD is inserted into the Player , the CD player cannot instantly play the song. It first has to start spinning the CD, then search the track where the first song begins and perform some other preparations . After about a few seconds the music could be heard . Likewise, the JMF Player does some preparation. A Player steps through a sequence of states until it reaches the final state. The following six Player states are defined by JMF:

Unrealized : This is the first state which a Player object enters as soon as it is instantiated. A newly instantiated Player does not yet know anything about its media.
Realizing : A Player moves from the unrealized state to the realizing state on invocation of the Player.realize() interface method. In the realizing state, the Player is in the process of determining its resource requirements. A realizing Player often downloads files from a broadcaster over a network and loads them into memory.
Realized : Transitioning from the realizing state, the Player comes into the realized state on completion of an initialization procedure; this transition is not triggered by invoking an interface function, but instead, it occurs as soon as the implementation of the Player is able to access the data and perform some minimal decoding to determine which resources it needs and the type of media it is to present. At this state, some Player implementations may provide visual components and controls, and establish links to other objects in the system that are in place.
Prefetching : A Player may transition from the realized state into the prefetching state on invocation of the prefetch() interface method. A prefetching Player is preparing to present its media. During this phase, the Player preloads its media data, obtains exclusive-use resources, and does whatever else is needed to play the media data.
Prefetched : On completion of the prefetching operation, a Player transitions from the prefetching state into the prefetched state; it is now ready to start. The transition into this state cannot be controlled by invoking an interface method; it occurs as soon as the Player 's implementation determines that prefetching was completed.
Started : This state may be entered either from the realized or prefetched states. It is entered when on invocation of the start() interface method. In case the start() method is invoked before the Player is in realized or prefetched states, it automatically tries to complete the tasks required prior to transition into the started state. During this state, the Player is rendering the data onto an output device such as speakers or graphics display.

9.8.4 Processor

A Processor is a type of Player which has control on the specific processing performed on the input media. In the JMF API, a Processor interface extends Player . As such, a Processor supports the same presentation controls as a Player , with the addition of some processing controls. In addition to rendering a DataSource , a Processor can generate output media data through a DataSource so it can be presented or otherwise consumed by another Player or Processor , or converted to some other format. This feature enables cascading Processor in a tree structure whose root is a single data source and leaves are the output media.

In addition to the six aforementioned Player states, a Processor includes two additional states that occur before the Processor enters the realizing state but after the unrealized state:

Configuring : A Processor enters the configuring state from the unrealized state on invocation of the configure() method. A Processor remains in the configuring state when it connects to the DataSource , demultiplexes the input stream, and accesses information about the format of the input data.
Configured : From the configuring state, a Processor transitions into the configured state on connection to a DataSource and after the data format has been determined. It is not possible to control the decision to transition from the Configuring into the Configured state using an interface method invocation; this transition is at the discretion of the Processor implementation.

As with a Player , a DataSource transitions to the Realizing state on invocation of the realize() method and subsequently transitions into the Realized state as soon as the initialization procedure.

9.8.5 DataSink

The DataSink is a base interface for objects that read media content delivered by a DataSource and render the media to some destination. As an example consider a file-writer object that stores the media in a file could serve as a DataSink . Intuitively, a DataSink is a simplified Processor that is not performing complex rendering operations and is not required to maintain states. An object can implement both the Processor and DataSink classes, and determine dynamically whether to behave as a sophisticated Processor or a DataSink .

9.8.6 Format

A Format object represents an object's exact media format. The format itself carries no encoding-specific parameters or global-timing information; it describes the format's encoding name and the type of data the format requires. Format subclasses include AudioFormat and VideoFor mat . In turn , VideoFormat contains six direct subclasses: H261Format , H263Format , IndexedColorFormat , JPEGFormat , RGBFormat , YUVFormat .

9.8.7 Manager

A JMF Manager is an intermediary object integrating the implementations of key interfaces that can be used seamlessly with existing classes. No real-world equivalent exists in a home-entertainment system; it can be found in the HAVi specification described in the previous chapter. The role of such a manager is to automate the interfacing between various objects. For example, with Manager one can create a Player from a DataSource s. The four JMF managers are:

Manager : This Manager is used to create Players , Processor s, DataSource s, and DataSink s.
PackageManager : This Manager maintains a registry of packages that contain JMF classes, such as custom Player s, Processor s, DataSource s, and DataSink s.
CaptureDeviceManager : This Manager maintains a registry of available capture devices.
PlugInManager : This Manager maintains a registry of available JMF plug-in components.

9.8.8 Player Construction

With JMF multimedia programming, one key operation is creating a Player . A Player is created by calling the Manager.createPlayer() method giving it a URL for the media (i.e., containing the protocol, domain and path ). The Manager uses the URL of the media or MediaLocator to create an appropriate Player . Once a Player is created, one can obtain the visual components where a Player renders a visual representation of its media. Those components could be added to an application's window. The following is required to display a Player object's visual component:

Obtain the visual component by calling the getVisualComponent() method
Add the visual component to the an HComponent of the JavaTV Xlet.

A Player may create and subsequently utilize a control panel with buttons to start, stop, and pause the media stream, as well as control the volume, just like the similar buttons on a CD player. Many of the Player methods can be called only when the Player is in the realized state. To guarantee that a Player has been realized, one can call Manager.createRealizedPlayer() method to create the Player . This method provides a convenient way to create and realize a Player in a single step. When it is called, it blocks (i.e., doesn't return) until the Player is realized. The start() method can be invoked after a Player is created but before it reaches the prefetched state. In this case, start() attempts to transition the Player to the started state from whatever state it's currently in. The start() method implicitly invokes all necessary methods to bring the Player into the started state.

9.8.9 Processing Streams

JMF enables controlling the transmission and reception of media streams, such as live radio and TV broadcasts. Processing media streams differs from processing data files in several ways:

Flow-Control : Processing of media streams is flow-controlled whereas processing files is rate-controlled. Processing media files allows controlling the rate at which data is processed , but with media streams there is no control over that rate; there is a need to take data chunks out of a buffer and process them to allow placing subsequent chunks in that buffer. There is a need to ensure sthat a decoding buffer neither overflows or underflows. This is achieved through the use of a buffer model.
Error Correction : With media streaming there are no guarantees. With all media streaming protocol there are guarantees that all the packets arrive , that they are in the correct order, and that their data is intact. A key challenge, with MPEG and RTP, is to compensate for lost data.
Quality of Service (QoS) : With media streaming, the processor makes QoS assumptions. It needs to know at what bandwidth the data is received and what error rates to expect. Even with the presence of such information, most software-based implementations of media players have a difficulty playing streams smoothly; without QoS guarantees, it is practically certain that the viewer's experience is optimized compared to traditional nondigital broadcasting techniques.
Partial Download : A media stream can be rendered as soon as a frame or some other basic data unit is received, and there is no need to wait for the entire data to download. MPEG audio and video comprise a sequence of packets with no particular entry point. Files generated by recording MPEG streams can be truncated or otherwise cut in the middle and still play correctly.

9.8.9.1 Transport Protocols

JMF supports both MPEG-based and IP-based media streaming. The JMF was originally specified with API supporting the RTP delivering media streams over UDP/IP. The MHP extended the JMF API to support MPEG-based streaming,

As opposed to TCP-based protocols, RTP, which is UDP-based, does not guarantee that data packets arrive in the order in which they were sent, and further, there is no guarantee that all packets arrive to their destination. In fact, there's no guarantee they arrive at all. It's up to the receiver to reconstruct the sender's packet sequence and detect lost packets using the information provided in the packet header. The RTCP protocol addresses resource reservation and QoS guarantees for real-time services. RTCP also allows data delivery monitoring in a manner scalable to large multicast networks.

JMF provides the APIs defined in the javax.media.rtp, javax.media.rtp.event, and javax.media.rtp.rtcp packages for RTP stream playback and transmission. JMF RTP APIs can work seamlessly with JMF's capture device, players, processors, and processing capabilities. There is a SessionManager that is dedicated to coordinating RTP sessions and managing the RTCP control channel.

9.8.9.2 Transmission of RTP Media Streams

There are two ways to transmit RTP streams:

To transmit RTP data by calling the Manager.createDataSink() method using as parameters the MediaLocator combined with the RTP session parameters to construct an RTP DataSink .
To use a SessionManager to create and send streams for the content and control the transmission. After retrieval of the output DataSource from a Processor , there is a need to invoke the SessionManager.createSendStream() and startSession() methods.

9.8.9.3 Receive and Play RTP Media Streams

A MediaLocator is used to create a Player using the Manager.createPlayer() method, passing a MediaLocator as the argument. As an alternative, when the media location is not available and the DataSource can be accessed directly, a Player object is constructed by retrieving the DataSource from the stream and passing it to the Manager.createPlayer() method. In both scenarios, the DataSource comes from the SessionManager controlling the retrieval of the media stream; for each stream a separate player is constructed.