|
Hack 53. Put Timing Information into Sound and Location Information into Light
The timing of an event will be dominated by the sound it makes, the location by where it looks as if it is happeningthis is precisely why ventriloquism works. Hearing is good for timing [Hack #44] but not so good for locating things in space. On the flip side, vision has two million channels for detecting location in space but isn't as fast as hearing. What happens when you combine the two? What you'd expect from a well-designed bit of kit: vision dominates for determining location, audition dominates for determining timing. The senses have specialized for detecting different kinds of information, and when they merge, that is taken into account. 5.2.1. In ActionYou can see each of the two senses take control in the location and timing domains. In the first part, what you see overrules the conflicting location information in what you hear; in the second part, it's the other way around. 5.2.1.1 Vision dominates for localizationGo to the theater, watch a film, or play a movie on your PC, listening to it on headphones. You see people talking and the sound matches their lip movement [Hack #59] . It feels as if the sound is coming from the same direction as the images you are watching. It's not, of course; instead, it's coming at you from the sides, from the cinema speakers, or through your headphones. The effect is strongest at public lectures. You watch the lecturer on stage talking and don't notice that the sound is coming at you from a completely different direction, through speakers at the sides or even back of the hall. Only if you close your eyes can you hear that the sounds aren't coming from the stage. The visual correspondence with the sounds you are hearing causes your brain to absorb the sound information into the same event as the image, taking on the location of the image. This is yet another example (for another, see [Hack #58] ) of how our most important sense, vision, dominates the other senses.
5.2.1.2 Audition dominates for timingVision doesn't always dominate. Watch Ladan Shams's "Sound-induced Illusory Flashing" movies at Caltec (http://neuro.caltech.edu/~lshams/demo.html; QuickTime).1 They show a black dot flashing very briefly on a white background. The only difference between the movie on the left and the movie on the right is the sound played along with the flash of the dot. With one set you hear a beep as the dot appears; with another set you hear two beeps.
Notice how the sound affects what you see. Two beeps cause the dot not to flash but to appear to flicker. Our visual system isn't so sure it is seeing just one event, and the evidence from hearing is allowed to distort the visual impression that our brain delivers for conscious experience. When the experiment was originally run, people were played up to four beeps with a single flash. For anything more than one beep, people consistently experienced more than one flash. Aschersleben and Bertelson2 demonstrated that the same principle applied when people produced timed movements by tapping. People tapping in time with visual signals were distracted by mistimed sound signals, whereas people tapping in time with sound signals weren't as distracted by mistimed visual signals. 5.2.2. How It WorksThis kind of dominance is really a bias. When the visual information about timing is ambiguous enough, it can be distorted in our experience by the auditory information. And vice versawhen auditory information about location is ambiguous enough, it is biased in the direction of the information provided by visual information. Sometimes that distortion is enough to make it seem as if one sense completely dominates the other. Information from the nondominant sense (vision for timing, audition for location) does influence what result the other sense delivers up to consciousness but not nearly so much. The exact circumstances of the visual-auditory event can affect the size of the bias too. For example, when judging location, the weighting you give to visual information is proportional to the brightness of the light and inversely proportional to the loudness of the sound.3 Nevertheless, the bias is always weighted toward using vision for location and toward audition for timing. The weighting our brain gives to information from these two senses is a result of the design of our senses, so you can't change around the order of dominance by making sounds easier to localize or by making lights harder to locate. Even if you make the sound location-perfect, people watching are still going to prefer to experience what they see as where they see it, and they'll disregard your carefully localized sounds. 5.2.3. End Notes
5.2.4. See Also
|
|