Thesis 10

Everyware necessitates a new set of human interface modes.

One of the most obvious ways in which everyware diverges from the PC case is its requirement for input modalities beyond the standard keyboard and screen, trackball, touchpad, and mouse.

With functionality distributed throughout the environment, embedded in objects and contexts far removed from anything you could (or would want to) graft a keyboard onto, the familiar ways of interacting with computers don't make much sense. So right away we have to devise new human interfaces, new ways for people to communicate their needs and desires to the computational systems around them.

Some progress has already been made in this direction, ingenious measures that have sprouted up in response both to the diminutive form factor of current-generation devices and the go-everywhere style of use they enable. Contemporary phones, PDAs, and music players offer a profusion of new interface elements adapted to their context: scroll wheels, voice dialing, stylus-based input, and predictive text-entry systems that, at least in theory, allow users of phone keypads to approximate the speed of typing on a full keyboard.

But as anyone who has spent even a little time with them knows, none of them is entirely satisfactory. At most, they are suggestive of the full range of interventions everyware will require.

One set of possibilities is suggested by the field known as "tangible media" at the MIT Media Lab, and "physical computing" to those researching it at NyU's Interactive Telecommunications Program. The field contemplates bridging the worlds of things and information, atoms and bits: Physical interface elements are manipulated to perform operations on associated data. Such haptic interfaces invoke the senses of both touch and proprioceptionwhat you feel through the skin, that is, and the sensorimotor awareness you maintain of the position and movement of your body.

In a small way, using a mouse is physical computing, in that moving an object out in the world affects things that happen on screen. The ease and simplicity most users experience in mousing, after an exceedingly brief period of adaptation upon first use, relies on the subtle consciousness of cursor location that the user retains, perceived solely through the positioning of the wrist joint and fingers. It isn't too far a leap from noticing this to wondering whether this faculty might not be brought directly to bear on the world.

An example of a tangible interface in practice is the "media table" in the lobby of New york's Asia Society, a collaboration between Small Design Firm and media designer Andrew Davies. At first glance, the table appears to be little more than a comfortable place to sit and rest, a sumptuously smooth ovoid onto which two maps of the Asian landmass happen to be projected, each facing a seated user. But spend a few minutes playing with itas its design clearly invites you toand you realize that the table is actually a sophisticated interface to the Asia Society's online informational resources.

Off to the table's side, six pucks, smoothly rounded like river stones, nestle snugly in declivities designed specifically for them. Pick one up, feel its comfortable heft, and you see that it bears lettering around its rim: "food," or "news headlines," or "country profiles." And although it's sufficiently obvious from the context that the stones "want" you to place them over the map display, a friendly introduction gives you permission to do just that.

When you do, the map zooms in on the country you've chosen and offers a response to your selection. Holding the "news headlines" stone over Singapore, for example, calls up a live Web feed from the Straits Times, while holding "food" over Thailand takes you to recipes for Tom yum Kung and Masaman Curry. you can easily spend fifteen minutes happily swapping stones, watching the display smoothly glide in and out, and learning a little bit about Asia as you're doing so.

As the media table suggests, such tangible interfaces are ideal for places where conventional methods would be practically or aesthetically inappropriate, or where the audience might be intimidated by them or uncomfortable in using them. The production values of the lobby are decidedly high, with sumptuous fixtures that include a million-dollar staircase; the Asia Society had already decided that a drab, standard-issue Web kiosk simply wouldn't do. So part of Small Design's reasoning was aesthetic: it wanted to suggest some connection, however fleeting, to the famous garden of the Ryoan-ji temple in Kyoto. If the media table is not precisely Zenlike, it is at the very least a genuine pleasure to touch and use.

But Small also knew that the Asia Society's audience skewed elderly and wanted to provide visitors who might have been unfamiliar with pull-down menus a more intuitive way to access all of the information contained in a complex decision tree of options. Finally, with no moving parts, the media table stands up to the demands of a high-traffic setting far better than a standard keyboard and trackball would have.

Though the Asia Society media table is particularly well-executed in every aspect of its physical and interaction design, the presentation at its heart is still a fairly conventional Web site. More radical tangible interfaces present the possibility of entirely new relationships between atoms and bits.

Jun Rekimoto's innovative DataTiles project, developed at Sony Computer Science Laboratories (CSL), provides the user with a vocabulary of interactions that can be combined in a wide variety of engaging waysa hybrid language that blends physical cues with visual behaviors. Each DataTile, a transparent pane of acrylic about 10 centimeters on a side, is actually a modular interface element with an embedded RFID tag. Place it on the display and its behaviors change depending on what other tiles it has been associated with. Some are relatively straightforward applications: Weather, video, and Paint tiles are exactly what they sound like. Playing a Portal tile opens up a hole in space, a linkage to some person, place or object in the real worlda webcam image of a conference room, or the status of a remote printer. Some Portals come with an appealing twist: the Whiteboard module not only allows the user to inscribe their thoughts on a remote display board, but also captures what is written there.

Parameter tiles, meanwhile, constrain the behavior of others. The Time-Machine, for example, bears a representation of a scroll wheel; when it is placed next to a video tile, the video can be scrubbed backward and forward. Finally, inter-tile gestures, made with a stylus, allow media objects to be "sent" from one place or application to another.

From the kind of physical interaction behaviors we see in the media table and the DataTiles, it's only a short step to purely gestural ones, like the one so resonantly depicted in the opening moments of Steven Spielberg's 2002 Minority Report. Such gestural interfaces have been a continual area of interest in everyware, extending as they do the promise of interactions that are less self-conscious and more truly intuitive. Like physical interfaces, they allow the user to associate muscle memory with the execution of a given task; theoretically, anyway, actions become nearly automatic, and response times tumble toward zero.

With this payoff as incentive, a spectrum of methods have been devised to capture the retinue of expressive things we do with our hands, from the reasonably straightforward to arcane and highly computationally intensive attempts to infer the meaning of user gestures from video. Some of the more practical rely on RFID-instrumented gloves or jewelry to capture gesture; others depend on the body's own inherent capacitance. Startup Tactiva even offers PC users something called TactaPad, a two-handed touchpad that projects representations of the user's hands on the screen, affording a curious kind of immersion in the virtual space of the desktop.

With the pace of real-world development being what it is, this category of interface seems to be on the verge of widespread adoption. Nevertheless, many complications and pitfalls remain for the unwary.

For example, at a recent convention of cartographers, the technology research group Applied Minds demonstrated a gestural interface to geographic information systems. To zoom in on the map, you place your hands on the map surface at the desired spot and simply spread them apart. It's an appealing representation: immediately recognizable and memorable, intuitive, transparent. It makes sense. Once you've done it, you'll never forget it.

Or so Applied Minds would have you believe. If only gesture were this simple! The association that Applied Minds suggests between a gesture of spreading one's hands and zooming in to a map surface is culturally specific, as arbitrary as any other. Why should spreading not zoom out instead? It's just as defensibly natural, just as "intuitive" a signifier of moving outward as inward. For that matter, many a joke has turned on the fact that certain everyday gestures, utterly unremarkable in one culturethe thumbs-up, the peace sign, the "OK" signare vile obscenities in another.

What matters, of course, is not that one particular system may do something idiosyncratically: Anything simple can probably be memorized and associated with a given task with a minimum of effort. The problem emerges when the different systems one is exposed to do things different ways: when the map at home zooms in if you spread your hands, but the map in your car zooms out.

The final category of new interfaces in everyware concerns something still less tangible than gesture: interactions that use the audio channel. This includes voice-recognition input, machine-synthesized speech output, and the use of "earcons," or auditory icons.

The latter, recognizable tones associated with system events assume new importance in everyware, although they're also employed in the desktop setting. (Both Mac OS and Windows machines can play earconson emptying the Trash, for example.) They potentially serve to address one of the concerns raised by the Bellotti paper previously referenced: Used judiciously, they can function as subtle indicators that a system has received, and is properly acting on, some input.

Spoken notifications, too, are useful, in situations where the user's attention is diverted by events in their visual field or by the other tasks that he or she is engaged in: Callers and visitors can be announced by name, emergent conditions can be specified, and highly complex information can be conveyed at arbitrary length and precision.

But of all audio-channel measures, it is voice-recognition that is most obviously called upon in constructing a computing that is supposed to be invisible but everywhere. voices can, of course, be associated with specific people, and this can be highly useful in providing for differential permissioningliquor cabinets that unlock only in response to spoken commands issued by adults in the household, journals that refuse access to any but their owners. Speech, too, carries clear cues as to the speaker's emotional state; A household system might react to these alongside whatever content is actually expressedyes, the volume can be turned down in response to your command, but should the timbre of your voice indicate that stress and not loudness is the real issue, maybe the ambient lighting is softened as well.

We shouldn't lose sight of just how profound a proposition voice-recognition represents when it is coupled to effectors deployed in the wider environment. For the first time, the greater mass of humanity can be provided with a practical mechanism by which their "perlocutionary" utterancesspeech acts intended to bring about a given statecan change the shape and texture of reality.

Whatever else comes of this, though, computing equipped with tangible, gestural, and audio-channel interfaces is set free to inhabit a far larger number and variety of places in the world than can be provided for by conventional methods.