Hack59.Hear with Your Eyes: The McGurk Effect

Hack 59. Hear with Your Eyes: The McGurk Effect

Listen with your eyes closed and you'll hear one sound; listen and watch the speaker at the same time and you'll hear another.

If there were ever a way of showing that your senses combine to completely change your ultimate experience, it's the McGurk Effect. This classic illusion, invented by Harry McGurk (and originally published in 1976¹, makes you hear different sounds being spoken depending on whether or not you can see the speaker's lips. Knowing what's going to happen doesn't help: the effect just isn't as strong.

5.8.1. In Action

Watch Arnt Maas's McGurk Effect video (http://www.media.uio.no/personer/arntm/McGurk_english.html; QuickTime with sound). You can see a freeze frame of the video in Figure 5-3.

Figure 5-3. Arnt Maas's McGurk Effect video

When you play it with your eyes closed, the voice says "ba ba." Play the video again, and watch the mouth: the voice says "da da." Try to hear "ba ba" while you watch the lips move. It can't be done.

5.8.2. How It Works

The illusion itself can't happen in real life. McGurk made it by splicing the sound of someone saying "ba ba" over a video of him making a different sound, "ga ga." When you're not watching the video, you hear what's actually being spoken. But when you see the speaker too, the two bits of information clash. The position of the lips is key in telling what sound someone's making, especially for distinguishing between speech sounds (called phonemes) like "ba," "ga," "pa," and "da" (those which you make by popping air out).

Visual information is really important for listening to people speak. It's a cliché, but I know I can't understand people as well when I don't have my glasses on.

M.W.

We use both visual and auditory information when figuring out what sound a person is making and they usually reinforce each other, but when the two conflict, the brain has to find a resolution. In the world the brain's used to, objects don't usually look as if they're doing one thing but sound as if they're doing another.

Since visually you're seeing "ga ga" and audition is hearing "ba ba," these are averaged out and you perceive "da da" instead, a sound that sits equally well with both information cues. In other situations, visual information will dominate completely and change a heard syllable to the one seen in the lip movements.²

Remarkably, you don't notice the confusion. Sensory information is combined before language processing is reached, and language processing tunes into only certain phonemes [Hack #49] . The decision as to what you hear is outside your voluntary control. The McGurk Effect shows integration of information across the senses at a completely preconscious level. You don't get to make any decisions about this; what you hear is affected by what goes in through your eyes. It's a good thing that in most circumstances the visual information you get matches what you need to hear.

5.8.3. End Notes

1. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-747.

2. Fusion of the sound and sight information is possible only when you have experience with a suitable compromise phoneme. One of the interesting things about phonemes is that they are perceived as either one thing or the other, but not as in-between values. So although there exists a continuum of physical sounds in between "ba" and "da," all positions along this spectrum will be perceived as either "ba" or "da," not as in-between sounds (unlike, say, colors, which have continuous physical values that you can also perceive). This is called categorical perception.

5.8.4. See Also

• Hearing with Your Eyes (http://ccms.ntu.edu.tw/~karchung/Phonetics%20II%20page%20seventeen.htm; QuickTime) has a collection of McGurk Effect movies.