Section 2.10. Application Services

2.10. Application Services

This layer could be perceived as providing two types of services: those specialized for graphics and multimedia applications and those usable by any kind of an application.

2.10.1. Graphics and Multimedia Services

The graphics and multimedia services layer provides APIs for using 2D graphics, 3D graphics, video, and audio. Figure 219 shows how this layer fits in the overall graphics and multimedia architecture.

Figure 219. The Mac OS X graphics and multimedia architecture

2.10.1.1. Quartz

The core of the Mac OS X imaging model is called Quartz, which provides support for rendering 2D shapes and text. Its graphics-rendering functionality is exported via the Quartz 2D client API, which is implemented in the Core Graphics framework (CoreGraphics.framework)a subframework of the Application Services umbrella framework. Quartz is also used for window management. It provides a lightweight window server, the Quartz Compositor, which is implemented partly in the WindowServer application^[39] and partly in the Core Graphics framework. Figure 220 shows a conceptual view of Quartz's constituents.

^[39] The WindowServer application resides in the Resources subdirectory of the Core Graphics framework.

Figure 220. The key constituents of Quartz

Quartz 2D

Quartz 2D uses the Portable Document Format (PDF) as the native format for its drawing model.^[40] In other words, Quartz stores rendered content internally as PDF. This facilitates features such as automatically generating PDF files (so you can save a screenshot "directly" to PDF), importing PDF data into native applications, and rasterizing PDF data (including PostScript and Encapsulated PostScript conversion). Quartz 2D is also responsible for device- and resolution-independent rendering of bitmap images, vector graphics, and anti-aliased text.

^[40] We saw in Chapter 1 that the windowing system in NEXTSTEP and OPENSTEP used Display PostScript for its imaging model.

Vector and Raster Graphics

PDF is a vector image file type. PDF imagesand vector graphics in generalcan be created through a sequence of mathematical statements that specify placement of geometric objects in a 2D or 3D vector space. Starting from a simple image (say, one consisting only of a straight line), a complex image can be drawn by adding more shapesmuch like how one would draw such a picture on a piece of paper. Moreover, the various elements of the picture are stored as individual objects. This makes it easy to alter the picture without loss of information.

Another approach to digital imaging is known as raster graphics. When a digital camera or a scanner is used to capture a picture, the resulting image is a raster image. A raster is a coordinate grid that resides in the display space. A raster image is a set of samples of this space. The image file's contents contain color (or monochrome) values that apply to each of the coordinates. Since the file's "bits" map to the display grid, raster images are commonly known as bitmap images. Unlike a vector image, it is difficult to alter a raster image without loss of information. One could think of a vector image as "the formula to generate image data," whereas a raster image would be "image data." Consequently, vector image files are typically smaller in size than raster images.

Examples of vector image formats include PDF, EPS (Encapsulated PostScript), and SVG (Scalable Vector Graphics). Examples of raster image formats include BMP (Windows Bitmap), GIF (Graphics Interchange Format), JPEG (Joint Photographic Experts Group), and TIFF (Tagged Image File Format).

Beginning with Mac OS X 10.4, Quartz includes the PDF Kit (PDFKit.framework)a Cocoa framework containing classes and methods for accessing, manipulating, and displaying PDF files.

Quartz Compositor

The Quartz Compositor is so called because of how it operates. Compositing refers to the process of overlaying independently rendered images into a single final image, while taking into account aspects such as transparency. Quartz implements layered compositing whereby an on-screen pixel can be shared between multiple windows in real time. The Quartz Compositor can composite pixels belonging to content from sources such as Quartz 2D, OpenGL, and QuickTime. In this sense, it follows a video-mixer model. Its implementation consists of the WindowServer program, which is an OpenGL application, and some private Apple libraries. Note that the Compositor does not perform any rendering itselfrendering is OpenGL's responsibility and is hardware accelerated when possible.

Quartz Services

Whereas the window server APIs are private, certain low-level features of the window server are exposed through the Quartz Services API. In particular, the API provides functions for accessing and manipulating display hardware. It also provides functions to access low-level window server events, thus allowing a program to "drive" the system remotely.

Quartz Extreme

Quartz has an integrated hardware acceleration layer called Quartz Extreme, which automatically becomes active if suitable hardware is available. Specific hardware features required for Quartz Extreme include a minimum of 16MB video memory and a graphics card that supports arbitrary texture sizes, multitexturing, and pixel formats used by Quartz.

Quartz Extreme is an implementation technique that uses OpenGL support in the graphics card so that the graphics processing unit (GPU)rather than the CPUdoes the compositing of window backing-stores^[41] to the framebuffer. Consequently, the CPU is freer, leading to improvements in system responsiveness and performance. Figure 221 shows an overview of Quartz Extreme.

^[41] A window backing-store is essentially a bitmap for the window. It contains information saved by the window system to track the window's contents.

Figure 221. An overview of Quartz Extreme

As Figure 221 shows, even when Quartz Extreme is active, the CPU is used to place pixels into window backing-stores, which reside in main memory. The GPU uses direct memory access (DMA) to asynchronously transfer the backing stores. It was not until Mac OS X 10.4 that Quartz 2D gained the capability to use the GPU for the entire drawing pathfrom the application to the framebuffer. This feature, called Quartz Extreme with Accelerated 2D (QE2D), is an evolution of Quartz Extreme. It can move data from an application to a window backing-store using DMA, while keeping track of frequently used data sets in an adaptively purged in-memory cache. QE2D strives to make rendering much faster than software rendering, while maintaining almost software-like quality. Common operations such as drawing glyphs, images, lines, and rectangles are implemented using OpenGL. Uncommon drawing operations, or those that cannot be handled well without software rendering,^[42] are handled through an optimized CPU drawing path. QE2D requires a GPU with the ARB_fragment_program OpenGL extension. Figure 222 shows an overview of QE2D. Note that backing stores are now cached in video memory. Moreover, graphics resources such as colors, images, layers, and patterns are also cached. For this scheme to work well, the programmer must judiciously retain references to resources.

^[42] Drawing a complex path with shading or stroking is an example of such an operation.

Figure 222. An overview of Quartz Extreme with Accelerated 2D

Quartz also uses hardware vector processing, when available on the CPU, to enhance performance.

2.10.1.2. QuickDraw 2D

QuickDraw 2D is a deprecated API for creating, manipulating, and displaying 2D shapes, text, and pictures. QuickDraw on Mac OS X is a reimplementation of the legacy QuickDraw API. It exists for the benefit of older projects as they transition to Quartz. QuickDraw makes use of the Velocity Engine but does not use graphics hardware acceleration like Quartz 2D.

You can also make Quartz rendering calls from within QuickDraw, but mixing QuickDraw and Quartz 2D code in an application will result in hardware acceleration being disabled altogether for the rendering.

2.10.1.3. OpenGL

Mac OS X includes an implementation of OpenGLa cross-platform graphics API standard for creating 3D and 2D graphics. OpenGL works with Quartz 2D and the Quartz Compositor, enabling system-wide visual effects and the graphical aspects of features such as Exposé and Dashboard.

As Figure 223 shows, there are several interfaces to OpenGL in Mac OS X.

Figure 223. Interfaces to OpenGL in Mac OS X

2.10.1.4. Core Image and Core Video

Core Image is a high-level image-processing interface to a GPU-based media technology. Using Core Image, application developers can harness the GPU without needing to resort to low-level programming. Depending on available hardware, Core Image uses GPU-based acceleration and vector processing to improve performance. It supports 32-bit floating-point pixels for enhanced accuracy. It also uses other, more fundamental graphical technologies such as OpenGL,^[43] Quartz, and QuickTime for optimal image processing.

^[43] Even when you use Core Image, it is OpenGL that eventually rasterizes the data.

Core Image uses a plug-in architecture^[44] for accessing filters, transitions, and effects packagesthese are called Image Units. Developers can use the various Image Units shipped with Core Image or create their own by describing filters and effects using dynamically compiled expressions. Examples of bundled Image Units include filters for blurring and sharpening, color adjustment, compositing, distortion, gradients, halftones, tiling, and transitions. Figure 224 shows a conceptual view of image processing with Core Image. The key components of Core Image include the following:

^[44] The Core Image plug-in architecture is inspired by the Core Audio plug-in architecture.

• Context (CIContext) a destination for images to be drawn, say, one created from an OpenGL context

• Image (CIImage) an image that theoretically extends to infinity, with a subrectangle as the region of interest

• Filter (CIFilter) an object that encapsulates an image-processing kernel and represents an effect or transition, with a parameter-passing interface

• Sampler (CISampler) an accessor object that is used (typically by a kernel) to retrieve samples (pixels) from an image

• Kernel (CIKernel) an object that lives within a filter and contains the per-pixel instructions that actually apply the filter's image-processing effect

Figure 224. Image processing with Core Image

Core Video applies concepts similar to Core Image to video, allowing video filters and effects to benefit from hardware acceleration. It could be seen as a bridge between QuickTime and the graphics hardware for processing video data by enhancing performance and lowering CPU requirements. Figure 225 shows a conceptual overview of using both Core Image and Core Video in an instance of the QuickTime video-rendering pipeline.

Figure 225. Core Image and Core Video in the QuickTime video-rendering pipeline

Core Image and Core Video are both parts of the Quartz Core framework.

Quartz Composer

Mac OS X 10.4 introduced a visual development tool called Quartz Composer, which allows rapid creation of graphical applications using several Mac OS X graphics technologies such as Core Image, OpenGL, Quartz 2D, and QuickTime. Besides graphical content, Quartz Composer can also use Rich Site Summary (RSS) content and MIDI System Services. For example, by simply dragging and dropping graphical building blocks into a grid, and "connecting" the blocks appropriately, you can rapidly create, say, a rotating cube with a different image rendered onto each of its faces, with custom lighting effects.

2.10.1.5. QuickTime

QuickTime is both a graphics environment and an application environment. It provides features for working with interactive multimedia. Depending on the media type, QuickTime allows you to manipulate media in several ways, such as:

• Accessing media (open, play, or display)

• Capturing media from an external device

• Compressing media

• Creating certain types of mediafor example, author panorama movies, object movies, and scenes using QuickTime Virtual Reality (QTVR)

• Editing and enhancing media, including synchronizing multiple media to a single timeline

• Streaming media over a local network or the Internet using protocols such as HTTP, RTP (Real-Time Transport Protocol), and RTSP (Real-Time Streaming Protocol)

• Translating between media formats

QuickTime works with several types of media such as video, graphics, animation, virtual reality, audio, and textin a variety of file and streaming formats. The media could reside on a local disk, be accessed over the network in its entirety, or be streamed in real time.

The QuickTime architecture is modular and extensible. QuickTime components can be written to implement support for new media types, to implement a new codec, and to interact with custom media capture hardware. As indicated in Figure 225, beginning with Mac OS X 10.4, QuickTime cooperates with Core Image and Core Video to improve performance.

Besides being an integrated technology in Mac OS X, QuickTime is available for other platforms such as Java, Mac OS 9, and several versions of Microsoft Windows.

2.10.1.6. Core Audio

The Core Audio layer (see Figure 219) allows management of audio software and hardware. It uses a plug-in architecture in which a plug-in can perform software operations on audio data or interact with audio hardware. Core Audio's hardware abstraction layer hides unnecessary details of the underlying hardware from applications. Core Audio APIs provide functionality for performing operations such as the following:

• Accessing and manipulating audio files

• Aggregating multiple audio devices into a single "virtual" device, which can be seamlessly made available to all applications

• Working with multichannel audio, including channel mixing

• Converting audio data between various formats

• Developing audio codecs

• Providing low-level access to audio hardware, including device sharing between multiple applications

• Synthesizing audio using software

• Using MIDI hardware and software

Apple's AU Lab digital mixing application (AULab.app) allows blending audio from multiple sources: the input of an audio device, audio generated by an Audio Unit Instrument, and audio generated from an Audio Unit Generator. It supports multiple outputs.

OpenAL

The Open Audio Library (OpenAL) is a cross-platform 3D audio API for use in games or other applications that require high-quality spatialized audio. It is available for systems such as BSD, IRIX, Linux, Solaris, Microsoft Windows, Mac OS 8, Mac OS 9, Sony PlayStation 2, Microsoft Xbox/Xbox360, and Nintendo GameCube. On Mac OS X, OpenAL is implemented using Core Audio as the underlying device. In the OpenAL specification, which is inspired by the OpenGL specification for graphics, a device is defined as an implementation-dependent entity that can be a hardware device, a daemon, or a system service.

2.10.2. Other Application Services

The Application Services umbrella framework contains subframeworks that facilitate development of many types of applicationshence the name. The following are examples of this framework's subframeworks.

• AE allows creation and manipulation of events in the Apple Events mechanism for interapplication communication.

• ATS allows font layout and management using Apple Type Services.

• ColorSync is used for color matching using Apple's integrated color management system, which is also called ColorSync.

• CoreGraphics provides the Core Graphics API.

• FindByContent provides an interface for searching specific volumes or directories for files containing specified content.

• HIServices provides human-interface services such as Accessibility, Icon Management, Copy and Paste, Process Management, Translation Management, and Internet Configuration.

• LangAnalysis provides an interface to the Language Analysis Manager that lets you analyze morphemes^[45] in text.

  ^[45] A morpheme is a meaningful linguistic unita distinctive collection of phonemes having no smaller meaningful parts. A phoneme is an indivisible phonetic unit in a language.

• LaunchServices provides an interface for opening URLs and starting applications, including opening documents with either a specified application or the default.

• PrintCore provides an interface to the printing subsystem.

• QD provides the QuickDraw API.

• SpeechSynthesis provides an interface for generating synthesized speech.