Image Performance Issues

In the resolution independent drawing universe of Quartz 2D, images are a bit like a racing duck out of water. Getting them to run quickly requires a bit of attention and training.

It's relatively easy to recognize some of the performance issues that working with images in the Quartz 2D environment will engender. For example, in addition to the geometrical transformations needed to take an image from user space to the device's coordinate system, images incur the added overhead of resampling and interpolating the pixels of the image on the destination device.

There are other performance impacts of images, however, that may not be as immediately obvious. Working with images also can involve working with large volumes of pixel data. Shuffling around all that data can also have a performance impact on your application. For example, if your image data is very large, drawing it may require the system to page in a lot of virtual memory. Going to the disk to page in memory is a very slow operation.

This section discusses a potpourri of issues that may affect your application when working with Quartz 2D. Our goal is not to solve every performance problem that could arise when working with Quartz 2D and images, however thinking about these issues may help you track down and resolve performance problems within your own application.

Pixel Data Is Immutable

It is important to note that regardless of the data provider that offers the pixel data, the image model presented to your application by CGImage requires the pixel data to be immutable for the life of the image. Once you have created a CGImage with its associated data provider, the data provided should be the same every time the computer asks for it.

The immutability of the pixels gives the operating system the freedom to optimize the way it draws your image. For example, if you ask the computer to draw the image repeatedly to the screen, Quartz 2D might hypothetically decide to cache a representation of that image in VRAM. The GPU on the graphics card could reproduce this image from VRAM anywhere on the display very quickly.

If the pixels of the image were not immutable, however, optimizations of this kind would be much more difficult.

It is important to realize that this immutability of the image is your application's responsibility and may not be enforced by the operating system. If you do create a bitmap context and a CGImage on the same memory buffer and try to draw the CGImage after changing the pixel data, the current version of the operating system may properly draw the modified image to some devices and not draw it properly on others. Moreover, your application might break in future versions of the operating system when Apple changes the behavior of the operating system.

Reuse Your CGImageRefs

The fact that you must create a new CGImage every time your image data changes is closely related to a performance guideline which recommends that you reuse the same CGImageRef to draw some image data if that data has not changed.

The reasons for reusing a CGImageRef are very similar to the reasons the operating system requires pixel data to be immutable. If you reuse an existing CGImageRef, the operating system has more opportunities to help you optimize the drawing of your images.

Continuing with the hypothetical optimization just mentioned, if the computer caches information about your image (like the VRAM cache mentioned in the previous section), that cached image information will be part of the opaque CGImage object. When you reuse that CGImage object, you give the operating system the opportunity to use that cached information to draw your image more quickly.

Use Resampling and Image Interpolation Correctly

When drawing images, interpolation can be a time-consuming operation. You should only use image interpolation when you need high quality rendering. For example, if your application is going to allow the user to drag an image around the screen, and you wish to provide feedback showing the image's position as it is dragged, you might consider drawing the drag feedback with a low-quality image interpolation setting to maximize the performance of the drag operation. Once the user has dropped the image, you could then redraw the image with a higher quality in its final position.

By the same token down-sampling an image takes some processing time. Consider an application that handles large image but draws thumbnails in a scrolling list. This application will get better performance if it calculates and caches the scaled down thumbnail images than if it tries to repeatedly scale and draw the full-size images.

Color Management Issues

Quartz 2D carefully incorporates color management as a fundamental feature of the graphics library. Both CGImages and CGContexts have color spaces associated with them. When you draw an image in a context, the color management system in Quartz carefully maps the colors from the image into the color space of the destination context.

This additional color matching step does not come for free. If you plan to draw an image to a particular color space repeatedly, you can improve drawing performance by skipping this color matching step. To do that, you can create a new image from pixel data that has already been corrected to the destination color space.

Speeding up this color management step is another instance where honoring the immutability of pixels and reusing CGImages can give you a performance boost. The computer might be able to cache intermediate results of color matching calculations to speed up subsequent renderings of a CGImage.

For many images, you might be able to use Quartz to create a new image in the destination color space. The following code sample illustrates one strategy for creating a color corrected version of an image. This sample converts an image to the color space of the main display. You might use this code to transform an image that you will draw in a window repeatedly.

Listing 8.4. Converting an Image to the Display's Color Space

CGColorSpaceRef CreateDisplayColorSpace() {     CGColorSpaceRef retVal = NULL;     CMProfileRef systemMonitorProfile = NULL;     // As the system for the default display profile.     CMError getProfileErr = CMGetSystemProfile(&systemMonitorProfile);     if(noErr == getProfileErr) {             // Create a CGColorSpace from that profile             retVal = CGColorSpaceCreateWithPlatformColorSpace(                            systemMonitorProfile);             // Be sure to close the profile             CMCloseProfile(systemMonitorProfile);     } else {            // If we could not open a profile... we simply return the            // generic RGB color space.            retVal = CGColorSpaceCreateWithName(                           kCGColorSpaceGenericRGB);     }     // This is a "Create" routine so the caller is responsible for     // releasing the result.     return retVal; } void ReleaseTempImageCallback(void *info, const void *data, size_t size) {     free(info); } CGImageRef CopyImageWithDisplayColorSpace(CGImageRef sourceImage) {     const UInt8 bytesPerPixel = 4;     CGImageRef retVal = NULL;     size_t width = CGImageGetWidth(sourceImage);     size_t height = CGImageGetHeight(sourceImage);     CGColorSpaceRef displayColorSpace = CreateDisplayColorSpace();     // Allocate a block of memory large enough to hold the     // color transformed image.     // We assume the display is an ARGB color space.     // Calculate rowbytes to be a multiple of 16 bytes     size_t rowBytes = ((bytesPerPixel * width) + 15) & ~15;     void *imageBuffer = malloc(rowBytes * height);     // We create a bitmap graphics context that draws into our block of     // memory and matches any drawing to the context to the display     // color space.     CGContextRef offscreenContext =             CGBitmapContextCreate(                    imageBuffer, width, height, 8,                    rowBytes, displayColorSpace,                    kCGImageAlphaPremultipliedFirst);     // Now we can draw the source image into the offscreen     // context. This allows the computer to color correct the     // image to the display color space     CGContextDrawImage(offscreenContext,            CGRectMake(0, 0, width, height), sourceImage);     // We're done with the bitmap context.     CGContextRelease(offscreenContext);     offscreenContext = NULL;     // Now we create another image using our color corrected image data.     CGDataProviderRef imageDataProvider =            CGDataProviderCreateWithData( imageBuffer, imageBuffer,                    rowBytes * height, ReleaseTempImageCallback);     retVal = CGImageCreate(                    width, height, 8, 32, rowBytes,                    displayColorSpace, kCGImageAlphaPremultipliedFirst,                    imageDataProvider, NULL, false,                    kCGRenderingIntentDefault);     CGDataProviderRelease(imageDataProvider);     imageDataProvider = NULL;     return retVal; } 

Listing 8.4 calls into service a number of Quartz topics not yet covered. In particular, it creates an offscreen graphics context and a CGImage. Creating an image was discussed earlier in this chapter, and creating offscreen contexts is covered in the chapter on offscreen drawing.

The first routine CreateDisplayColorSpace uses a ColorSync routine to retrieve the color profile of the main display. It then creates a color space based on that profile and returns it to the caller. In the unlikely event that the computer cannot retrieve the system profile, then the code uses the generic RGB color space as the destination color space.

The second routine ReleaseTempImageCallback is used by the CGDataProvider used when the image was created. Data providers are discussed later in this chapter. This routine allows the code to release the image data of the new image when the application is done with it.

Much of the work in this code sample is handled by the CopyImageWithDisplayColorSpace. This routine creates a new image based on source image data but color corrected to the display color space. The routine does make the simplifying assumption that the display color space will be an RGB color space. It also assumes the image might have an alpha channel that the code should preserve.

The code begins by extracting attributes of the source image. From those attributes it allocates an image buffer large enough to hold a copy of the image but repurposed as a 32 bit-per-pixel ARGB image. The code creates the context with the display's color space. The code then draws the image into the offscreen context. This drawing operation induces Quartz to correct the source image into the color space of the context, the display color space. Finally, the code takes the now corrected image data and constructs a new CGImage from it.

The end result of all this work is a CGImage that does not have to be color corrected (again) when drawn to the display. This Quartz 2D-based technique is certainly not the only way to change the color space of the image. You could also use ColorSync to correct your image data prior to creating the original CGImage.

Compositing Takes Time

Every time the computer has to draw an alpha composited pixel, it must read the value of the pixel in the destination context, multiply its value by the color of the pixel being composited, and write the results back into the destination. This read-modify-write cycle can take some time. Contrast this with the instance where the pixel being drawn is opaque. The computer can skip the read and modify steps and simply write the opaque pixel to the destination. Needless to say, this can be a significant performance boost and means that, in some cases, you may be able to boost the performance of your application by correctly setting the alpha channel information in your images to exclude alpha if the image has none.

Images and VRAM

As Mac OS X evolves, the video memory on the graphics card is rapidly becoming a premium resource. When you draw an image to the display the computer will have to transfer some fraction of the image data across the graphics bus to the video card. This ties the performance of the graphics to the speed of the VRAM bus.

For some graphics, the computer may be able to send the drawing commands that create the graphics to the video card faster than it can pass over an image generated by those drawing commands. For example, suppose your application needs to fill a large area of a window with a blue field and place a small icon in the middle of it. If you draw this graphic from a cached CGImage, the computer might have to transfer all the pixels of the image across the bus. If, however, you can draw the blue field using a line art primitive, the computer can issue the drawing commands for the background using a few bytes, and the graphics processor on the video card can quickly fill the rectangle in the frame buffer. Drawing the icon might still require sending data across the bus, but certainly it will not require as much work as transferring the larger image.

Moreover, as the graphics system evolves to rely more heavily on the graphics card memory space in VRAM, it will quickly become crowded. If you are drawing several images, you will be driving the graphics resources of other applications out of VRAM which can have an adverse affect on the behavior of the system as a whole.

These facts should not be taken as a general injunction against drawing images in windows. However, there may be instances in your application where you have used an image in the past, but replacing that code with similar code that does not use images might improve performance.

Most often, this kind of thing comes into play in applications written to pixel-based libraries like QuickDraw. Those applications would often cache graphics in offscreen bitmaps as an optimization technique. The application would then transfer those graphics to the display because the pixel copy operation (from main memory to main memory) was a cheaper operation than reissuing the drawing commands.

If your graphics are relatively simple, say the graphic picture of the scale on a ruler, then you may actually enjoy enhanced performance by removing the "optimization" of caching the graphics in an offscreen bitmap. The computer may be able to draw the scale more quickly using line art primitives given that those primitives cause fewer bytes to be shipped across the video bus and much of the rendering can be handled asynchronously by the GPU.

Keep It Simple, Then Test It

Perhaps the most important point to make about images and Quartz 2D is that the library does not treat images the same way that traditional graphics libraries do. Moreover, advances in the graphics architecture and other systems on Mac OS X gives the system a very different profile than other graphics environments. For example, the sheer volume of pixel data with many images can lead to problems with the virtual memory system. As a result, some of the techniques you would use to optimize drawing in other graphics libraries can have an adverse effect on the performance of Quartz 2D.

When in doubt, your best bet is to keep your drawing code as simple as possible. When you keep it simple, you make it easier for the operating system to work on your behalf and optimize the graphics pipeline. If you do this and still find your graphics has disappointing performance, carefully apply profiling tools, among others, to determine where your bottlenecks are.

If for no other reason, images are an important part of the Quartz 2D imaging model because they represent the best way to exert pixel-level control over your graphics. For developers making the transition from pixel-based graphics libraries, images may be the key to duplicating functionality that Quartz 2D cannot duplicate in other ways. As you work with images, however, keep in mind that they consume a lot of memory and are inherently resolution dependent. These facts have an impact on both the performance of programs that use images and the quality of the output they generate.