Hack57.Combine Modalities to Increase Intensity

Hack 57. Combine Modalities to Increase Intensity

Events that affect more than one sense feel more intense in both of them.

The vision and audition chapters (Chapter 2 and Chapter 4, respectively) of this book look at the senses individually, just as a lot of psychologists have over the years. But interesting things begin to happen when you look at the senses as they interact with one another.¹

Multisensory information is the norm in the real world, after all. Tigers smell strong and rustle as they creep through the undergrowth toward you. Fire shines and crackles as it burns. Your child says your name as she shakes your shoulder to wake you up.

These examples all suggest that the most basic kind of interaction between two senses should be the enhanced response to an event that generates two kinds of stimulation rather than just one. Information from one sense is more likely to be coincidence; simultaneous information on two senses is a good clue that you have detected a real event.

5.6.1. In Action

We can see the interaction of information hitting two senses at once in all sorts of situations. People sound clearer when we can see their lips [Hack #59] . Movies feel more impressive when they have a sound track. If someone gets a tap on one hand as they simultaneously see two flashes of light, one on each side, the light on the same side as the hand tap will appear brighter.

Helge Gillmeister and Martin Eimer of Birkbeck College, University of London, have found that people experience sounds as louder if a small vibration is applied to their index finger at the same time.² Although the vibration didn't convey any extra information, subjects rated sounds as up to twice as loud when they occurred at the same time as a finger vibration. The effect was biggest for quieter sounds.

5.6.2. How It Works

Recent research on such situations shows that the combination of information is wired into the early stages of sensory processing in the cortex. Areas of the cortex traditionally thought to respond to only a single sense (e.g., parts of the visual cortex) do actually respond to stimulation of the other senses too. This makes sense of the fact that many of these effects occur preconsciously, without any sense of effort or decision-making. They are preconscious because they are occurring in the parts of the brain responsible for initial representation and processing of sensationanother example (as in [Hack #15] ) of our perception not being passive but being actively constructed by our brains in ways we aren't always aware of.

Macaluso et al.³ showed that the effect can work the other way round from the one discussed here: touch can enhance visual discrimination. They don't suggest that integration is happening in the visual cortex initially, but instead that parietal cortex areas responsible for multisensory integration send feedback signals down to visual areas, and it is this that allows enhanced visual sensitivity.

For enhancement to happen, it has to be labeled as belonging to the same event, and this is primarily done by the information arriving simultaneously. Individual neurons [Hack #9] are already set up to respond to timing information and frequently respond strongest to inputs from different sources arriving simultaneously. If information arrives at different times, it can suppress the activity of cells responsible for responding to inputs across senses (senses are called modalities, in the jargon).

So, what makes information from two modalities appear simultaneous? Obviously arriving at the exact same time is not possible; there must be a resolution of the senses in time below which two events appear to be simultaneous.

Although light moves a million times faster than sound, sound is processed faster once it gets to the ear [Hack #44] than light is processed once it gets to the eye. The relative speed of processing of each sense, coupled with the speed at which light and sound travel, leads to a "horizon of simultaneity"⁴ at about 10 meterswhere visual and auditory signals from the same source reach the cortex at the same time.

Most events don't occur just on this 10-meter line, of course, so there must be some extra mechanisms at work in the brain to allow sound and light events to appear simultaneous. Previously, researchers had assumed that the calculation of simultaneity was approximate enough that time difference due to arrival time could be ignored (until you get to events very far awaylike lightning that arrives before thunder, for example). But now it appears that our brains make a preconscious adjustment for how far away something is when calculating whether the sound and the light are arriving at the same time.5 Another mechanism that operates is simply to override the timing information that comes from vision with the timing information from auditory information [Hack #53] .

5.6.3. End Notes

1. To start following up the research on crossmodal interactions, you could start by reading Crossmodal Space and Crossmodal Attention by Charles Spence and Jon Driver. This is an edited book with contributions from many of the people at the forefront of the field. You can read more about the Oxford University crossmodal research group on its home page: http://www.psych.ox.ac.uk/xmodal/default.htm.

2. Gillmeister, H., & Eimer, M. (submitted). Multisensory integration in perception: tactile enhancement of perceived loudness.

3. Macaluso, E., Frith, C. D., & Driver, J. (2000). Modulation of human visual cortex by crossmodal spatial attention. Science, 289, 1206-1208.

4. Pöppel, E. (1988). Mindworks: Time and Conscious Experience. New York: Harcourt Brace Jovanovich.

5. Sugita, Y., and Suzuki, Y. (2003). Audiovisual perception: Implicit estimation of sound-arrival time. Nature, 421, 911.