Chapter 4: Multimodal Interaction on Mobile Artefacts

Laurence Pasqualetti, Laurence Nigay, Moustapha Zouinar, Pierre Salembier, Guillaume Calvet, Julien Kahn, Ga «tan Rey, France T l com R&D, University of Grenoble and University of Toulouse,

France

1 Introduction

Recent progress achieved with the miniaturisation of microprocessors and wireless networks make it possible to consider that the "grey box" of the personal computer is condemned to disappear or at least not to be the only place of interaction between the user and the numerical world. This comes about as a result of a double movement resulting from technological work on the concept of ubiquitous computing and the disappearing computer and from the evolution of ideas in the field of models of interaction. Indeed, research is now gradually being directed towards models of interaction in which data-processing resources are distributed in a multitude of usual objects with which the user will interact in an explicit or implicit way. The idea here is to use the environment as an interface, as a system of manipulation of technical resources functionally limited but contextually relevant (concept of " tangible " interface). The device can be physically handled in a meaningful way: action with the device returning to a preset function (that requires the definition of a semantics and possibly of a syntax of the interaction with the device).

The concept of communicating objects covers very diverse technological and conceptual realities; among the specific properties generally attached to these objects one will retain in particular:

• numerical increase rather than substitution: the direct idea is to set out again usual objects in which one endeavours to preserve the intrinsic advantages inherent in their material constitutivity (in particular properties of affordability and support of awareness) at the same time associating additional functionalities (eg: paper and the numerical pen, the communicating refrigerator etc).

• transportable character of devices: it makes it possible to be given "embarked" resources of processing and communication and thus support mobility and intellectual nomadism. These devices can take several forms (PDA, portable telephone, communicating clothing etc).

• capacity to communicate in an autonomous or controlled way: in addition to their traditional function of support to the communication between users, portable devices can detect the presence of a device of the same type (or data-processing resources distributed in the environment) and exchange information according to possibly pre-established rules but independently of any command given intentionally by the user.

In the HOURIA project we study the way in which certain physical achievements of this concept of communicating object could be used easily and in an effective way by mobile individuals, and to be integrated without causing rupture to their daily activities and the physical and social environments in which they are located at every moment. We justify our choice of the multimodality by the following:

• These artefacts rely on physical devices with restricted capacities , outside the traditional framework (a large screen, a keyboard and a mouse). It is thus advisable to conceive methods of interaction relying on paradigms other than direct handling "screen-mouse" like the tangible interfaces or the "embodied user interface".

• The context of use of these artefacts is by definition very variable. Indeed, the physical (noise etc) and social characteristics (intimacy, intrusiveness etc) of the environment determine a whole set of contextual constraints which will require interactional adaptations; from this point of view, multimodality constitutes a type of answer adapted to this requirement of adaptability and adaptivity. The development of mobile computing thus represents a field highly suited to the application of multimodal interaction techniques.