Hack53.Put Timing Information into Sound and Location Information into Light

Hack 53. Put Timing Information into Sound and Location Information into Light

The timing of an event will be dominated by the sound it makes, the location by where it looks as if it is happeningthis is precisely why ventriloquism works.

Hearing is good for timing [Hack #44] but not so good for locating things in space. On the flip side, vision has two million channels for detecting location in space but isn't as fast as hearing.

What happens when you combine the two? What you'd expect from a well-designed bit of kit: vision dominates for determining location, audition dominates for determining timing. The senses have specialized for detecting different kinds of information, and when they merge, that is taken into account.

5.2.1. In Action

You can see each of the two senses take control in the location and timing domains. In the first part, what you see overrules the conflicting location information in what you hear; in the second part, it's the other way around.

5.2.1.1 Vision dominates for localization

Go to the theater, watch a film, or play a movie on your PC, listening to it on headphones. You see people talking and the sound matches their lip movement [Hack #59] . It feels as if the sound is coming from the same direction as the images you are watching. It's not, of course; instead, it's coming at you from the sides, from the cinema speakers, or through your headphones.

The effect is strongest at public lectures. You watch the lecturer on stage talking and don't notice that the sound is coming at you from a completely different direction, through speakers at the sides or even back of the hall. Only if you close your eyes can you hear that the sounds aren't coming from the stage. The visual correspondence with the sounds you are hearing causes your brain to absorb the sound information into the same event as the image, taking on the location of the image. This is yet another example (for another, see [Hack #58] ) of how our most important sense, vision, dominates the other senses.

Incidentally, this is how ventriloquism works. The ventriloquist knows that if the timings of the dummy's lip movements are close enough to the sounds you hear you will preconsciously locate the sounds as coming from the dummy. Every time we go to the cinema we are experiencing a ventriloquism effect, but it is so finessed that we don't even notice that it is part of the show.

T.S.

5.2.1.2 Audition dominates for timing

Vision doesn't always dominate. Watch Ladan Shams's "Sound-induced Illusory Flashing" movies at Caltec (http://neuro.caltech.edu/~lshams/demo.html; QuickTime).¹ They show a black dot flashing very briefly on a white background. The only difference between the movie on the left and the movie on the right is the sound played along with the flash of the dot. With one set you hear a beep as the dot appears; with another set you hear two beeps.

On Ladan Shams's page, you have the option of watching a number of different pairs of movies. These correspond to different computer speeds. Start with the ones at the top and run them all until you find the one with the strongest effect.

Notice how the sound affects what you see. Two beeps cause the dot not to flash but to appear to flicker. Our visual system isn't so sure it is seeing just one event, and the evidence from hearing is allowed to distort the visual impression that our brain delivers for conscious experience.

When the experiment was originally run, people were played up to four beeps with a single flash. For anything more than one beep, people consistently experienced more than one flash.

Aschersleben and Bertelson² demonstrated that the same principle applied when people produced timed movements by tapping. People tapping in time with visual signals were distracted by mistimed sound signals, whereas people tapping in time with sound signals weren't as distracted by mistimed visual signals.

5.2.2. How It Works

This kind of dominance is really a bias. When the visual information about timing is ambiguous enough, it can be distorted in our experience by the auditory information. And vice versawhen auditory information about location is ambiguous enough, it is biased in the direction of the information provided by visual information. Sometimes that distortion is enough to make it seem as if one sense completely dominates the other.

Information from the nondominant sense (vision for timing, audition for location) does influence what result the other sense delivers up to consciousness but not nearly so much. The exact circumstances of the visual-auditory event can affect the size of the bias too. For example, when judging location, the weighting you give to visual information is proportional to the brightness of the light and inversely proportional to the loudness of the sound.³ Nevertheless, the bias is always weighted toward using vision for location and toward audition for timing.

The weighting our brain gives to information from these two senses is a result of the design of our senses, so you can't change around the order of dominance by making sounds easier to localize or by making lights harder to locate. Even if you make the sound location-perfect, people watching are still going to prefer to experience what they see as where they see it, and they'll disregard your carefully localized sounds.

5.2.3. End Notes

1. Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408, 788.

2. Aschersleben, G., & Bertelson, P. (2003). Temporal ventriloquism: crossmodal interaction on the time dimension: 2. Evidence from synchronization. International Journal of Psychophysiology, 50(1-2), 157-63.

3. Radeau, M. (1985). Signal intensity, task context, and auditory-visual interactions. Perception, 14, 571-577.

5.2.4. See Also

• Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078-1093.

• The advice in this hack, and other good tips for design can be found in Reeves et al. (2004). Guidelines for multimodal user interface design. Communications of the ACMSpecial Issue on Multimodal Interfaces, 47(1), 57-59. It is online at http://www.niceproject.com/publications/CACM04.pdf.