Chapter 21: Direct Manipulation and Pointing Devices

Modern graphical user interfaces are founded on the concept of direct manipulation of graphical objects on the screen: buttons, sliders, and other function controls, as well as icons and other representations of data objects. The ability to choose, group, and move objects on the screen is fundamental to the interfaces we design today. But to perform these manipulations, we also require input mechanisms that give us the flexibility to do so. This chapter discusses both the basics of direct manipulation and the various devices that have been employed to make such manipulation possible.

Direct Manipulation

Ben Shneiderman, professor of Computer Science at University of Maryland, coined the term direct manipulation in 1974. Its three elements (paraphrased by the authors) consist of:

• Visual representation of the manipulated objects

• Physical actions instead of text entry

• Immediately visible impact of the operation

A less-rigorous definition would say that direct manipulation is clicking and dragging things; and although this is true, it can easily miss the point that Shneiderman subtly makes. Notice that of his three points, two of them concern the visual feedback the program offers the user and only the second point concerns the user's actions. It might be more accurate to call it "isual manipulation" because of the importance of what we see during the process. Unfortunately, I've seen many instances of direct-manipulation idioms implemented without adequate visual feedback, and these idioms fail to satisfy the definition of effective direct manipulation.

AXIOM 

Rich visual interaction is the key to successful direct manipulation.

Yet another observation about direct manipulation—one that is hidden by its obviousness—is that we can only directly manipulate information that is already displayed by the program; it must be visible for us to manipulate it, which again emphasizes the visual nature of direct manipulation. If you want to create effective direct-manipulation idioms in your software, you must take care to render data, objects, controls, and cursors with fine graphic detail and richness.

Direct manipulation is simple, straightforward, easy-to-use, and easy-to-remember. However, most users, when first exposed to a given direct-manipulation idiom, cannot intuit or discover it independently. Direct manipulation idioms must be taught, but the teaching of them is trivial—usually consisting of merely pointing them out—and, once taught, they are seldom forgotten. It is a classic example of idiomatic design. Adding metaphoric images may help, but you cannot depend on finding an appropriate one; and if you do, you cannot depend on it communicating clearly to all users. Resign yourself to the burden of teaching direct manipulation idioms. Console yourself with the ease of that teaching.

With regard to direct manipulation, Apple's Human Interface Style Guide says, "Users want to feel that they are in charge of the computer's activities." These published guidelines and the Macintosh user interface itself make clear that Apple believes in direct manipulation as a fundamental tenet of good user-interface design. However, user-centered design guru Don Norman (1989) says "Direct manipulation, first-person systems have their drawbacks. Although they are often easy to use, fun, and entertaining, it is often difficult to do a really good job with them. They require the user to do the task directly, and the user may not be very good at it." Whom should we believe?

The answer, of course, is both of them. As Apple says, direct manipulation is an extremely powerful tool; and as Norman says, the tool must be put into the hands of someone qualified to use it.

This contradiction should illustrate the differences between two distinct types of direct manipulation. Pushing a button is direct manipulation and so is drawing with the pen tool in a paint program. Although almost any user can push a button, few are capable of drawing well with the pen tool (or for that matter with a real pen). These examples illustrate the two variants of direct manipulation: program manipulation and content manipulation.

Program manipulation versus content manipulation

Program manipulation includes actions like button pushing and scrolling and is equally accessible to all (non-handicapped) users. Program manipulation should require no special skills or training and is typically focused on the management of the program and its interface.

Content manipulation, on the other hand, is involved primarily with the direct, manual creation, modification, and movement of data with the pointing device; and although it can be performed by anyone, its results will always be commensurate with the skill and talent of the manipulator.

Applications like Corel Draw!, Adobe Photoshop, and Macromedia Fireworks are focused on drawing operations, and these fit into the definition of content manipulation. The same holds for 3D modeling applications like SketchUp or Alias|Wavefront Maya. Programs like Visio or even PowerPoint strain the definition, but even their more-structured interfaces are still content-centered and require some graphic talent from the user. We'll discuss drawing idioms in more detail in Chapter 24.

In the program manipulation category, we find five varieties of direct manipulation:

• Selection

• Drag and drop

• Control manipulation

• Resizing, reshaping, and repositioning

• Connecting objects

We will discuss each of these in the remaining chapters of Part V.

Three phases of the direct manipulation process

We can divide the direct manipulation process into three distinct phases:

1. Free Phase: Before the user takes any action

2. Captive Phase: After the user has begun the drag

3. Termination Phase: After the user releases the mouse button

In the free phase, nothing has happened yet. The job of the interface is to indicate direct-manipulation pliancy at the appropriate time.

In the captive phase, the interface has two tasks. It must positively indicate that the direct-manipulation process has begun, and it must visually identify the potential participants in the action.

In the termination phase, the interface must plainly indicate to the user that the action has terminated and show exactly what the result is.

Visual feedback for direct manipulation idioms

As we've discussed, the key to successful direct manipulation is rich visual feedback. Let's take a more detailed look at some visual feedback methods.

Depending on which phase you are in, there are two variants of cursor hinting. During the free phase, any visual change the cursor makes as it passes over something on the screen is called free cursor hinting. After the captive phase has begun, changes to the cursor represent captive cursor hinting. Many programs use a hand-shaped cursor to indicate that the document itself, rather than the information in it, can be dragged.

If you are holding down a meta-key during the drag to drag away a copy of the object instead of the object itself, the cursor may change from an arrow to an arrow with a little plus sign over it to indicate that the operation is a copy rather than a move. This is a clear example of captive cursor hinting.

When an object is dragged, the cursor must drag either the full representation of the object or some simulacrum of that representation. In a drawing program, for example, when you drag a complex visual element from one position to another, it may be too difficult for the program to actually drag the image (due to the computer's performance limitations), so it often simply drags an outline of the object.

Direct manipulation is critical to effective interaction

The advent of the Web and HTML in the early 1990s in some ways represented a giant step back for the industry in terms of interface and interaction design. The industry suddenly got hold of the notion that an interaction limited to scrolling a page and single-clicking a link was not only acceptable, but that it was superior. Well, why not? What could be simpler than a single click to do something? This view was, and is, shortsighted. Although single clicks are (barely) adequate for moving from one document to another (you still need to scroll, after all), they are supremely inadequate for most other tasks, such as moving, grouping or organizing things, drawing and sculpting things, positioning and sizing things, and other human tasks. There is no reason to limit either the Web or the desktop to the single click, and we can hopefully look forward to Web applications that are as rich in their interaction as desktop applications, as technology improves. Luckily, the hardware we have is more than up to the task of rich use of direct manipulation; we need only make sure our software continues to keep pace. In the next section of this chapter, we discuss common pointing devices and the capabilities they provide.