Hack 57. Fool Your Audience s Perception

Hack 57. Fool Your Audience's Perception

What you hear isn't always what you see.

Producing and editing a video project involves working with both audio and video. Most people know they can use audio to smooth over cuts [Hack #46] and cover other problems [Hacks #55 and #56]. But you can also use the combination of audio and video to trick your viewers into seeing one thing, but hearing another.

The McGurk effect is an excellent demonstration of this fact.

For more information on the McGurk effect, and other visual tricks, check out Mind Hacks (O'Reilly) by Tom Stafford and Matt Webb. You can also search online for "McGurk effect" to find out more of the scientific reasons behind it, as well as view some demonstration videos.

5.8.1. Creating the Effect

The McGurk effect occurs when you see something and it influences what you hear, and vice-versa. Harry McGurk and John MacDonald explained the effect in a 1976 article titled "Hearing lips and seeing voices" (Nature 246, 746 748). The effect can be seen in video where a person is mouthing "ga," but the audio is "ba," and what you think you hear is "da." To view a sample online, visit http://www.media.uio.no/personer/arntm/McGurk_english.html. To create the effect, you will need to acquire audio and video from two different sources.

5.8.1.1. Acquiring the video.

As mentioned, to create the effect requires video of someone mouthing "ga." The best way to record the footage is to have someone simply say "ga," because you'll remove out the audio portion when you edit the project together. To make the editing process easier, you will want the person on camera (you?) to speak the word at least a handful of times. Something like "ga …ga…ga…ga…ga…" will work well.

5.8.1.2. Acquiring the audio.

Your audio portion can be spoken by the same person in the video, or by someone else. You will want the audio to consist of the sound "ba," and it should be paced at the same rate as the video portion. If you need a visual cue, you can always play back your video segment, with the volume turned off, while recording your audio segment. So, for your audio segment you should have "ba…ba…ba…ba…ba…"

5.8.1.3. Editing the footage.

Once you've acquired your footage, import it to your editing system. Then, add the video (ga ga) to your timeline. After adding the video to your timeline, you need to remove the audio portion of the signal:

Avid

Use the Lift/Overwrite Tool to select the audio and lift it from the timeline.

Final Cut

Press F4 (function 4) and the number of the track you would like to lock (e.g., 1). Then, select the audio tracks and delete them from the timeline.

Movie Maker

Right-click on the audio and select Mute from the menu.

Premiere

Lock the video track, using the lock button on the timeline. Then, delete the associated audio.

iMovie

Click on the clip in the timeline, and then choose Advanced Extract Audio. Click on the new audio clip and choose Edit Clear.

Some systems have a Lock feature available within the timeline. If your system offers the feature, it is just another way for you to lock your video track and then remove the associated audio track. Figure 5-7 shows the interface for Adobe's Premiere Pro and the Lock feature available in the timeline.

After removing the audio from your "ga ga" footage, you need to add your "ba ba" audio to your timeline. To do this, follow a process similar process to the one used for the video:

Figure 5-7. Locking a track in Premiere Pro

Avid

Use the Lift/Overwrite Tool to select the video and lift it from the timeline.

Final Cut

Press F5 (function 5) and the number of the track you would like to lock (e.g., 1). Repeat the process for your second track, if you've used stereo audio. Then, select the video track and delete it from the timeline.

Movie Maker

Drag the "ba ba" footage to the Audio/Music track. Only the audio will be added to your timeline.

Premiere

Lock the audio tracks, using the Lock button on the timeline. Then, delete the associated video.

iMovie

Click on the clip in the timeline, and then choose Advanced Extract Audio. Click on the video clip and choose Edit Clear.

Once you have your independent video and audio clips, you need to line them up. Drag your audio until it lies below your video in the timeline. From here, you need to nudge your audio forward or backward in order to have it match your video. You might also need to cut your audio footage into sections, so that you can line up each "ba" with each "ga." Once you've lined up your audio and video segments, play your timeline.

The effect should be apparent (at least to 98% of the population). Even though you know that the audio is "ba," your mind will translate it to "da" to match what you are seeing. You can close your eyes and play the video to make sure nothing "funny" has happened to your audio.

5.8.2. Using the Effect in Real Projects

Now that you've experimented with the McGurk effect, you might be wondering how you can use it in a real-world scenario. Well, the effect obviously works well with speech. As a visual-only experiment, have a friend speak the words "olive juice." Then, ask her to only mouth the words. What is it that you think she's saying?

Taking the visual aspect of what your mind interprets, now have someone stand behind your friend and have him speak "I love you" while she mouths "olive juice." Beside the strange fact that you now have a woman telling you she loves you in a man's voice, what is it that you're hearing? What is it that you're seeing?

This type of editing occurs frequently when major motion pictures are shown on television. For example, some four-letter words cannot be broadcast on television. Therefore, select words are edited over the original censored words to make the movie acceptable for broadcast. You will notice this when someone says, "Forget you!" but you know they said something else in the original version.

But you don't have to be limited to trying to match words to people's mouths. Editing audio can prove very useful, especially when you need to re-craft a story after it's been shot. A good example of this is when you have a wide-shot of a scene and would like to change what's being said. You can use audio from another scene and place it underneath your wide-shot, and then try to lineup the audio with someone's lips moving. In a wide-shot it will be difficult for your audience to lip-read, and they will rely on (and trust) what they hear.

The most impressive use of this effect I have personally experienced was on The Larry Sanders Show. We had a scene where one character, Artie, had his back to the camera and another, Larry, was speaking a line. We needed to drop one of Larry's lines, but we couldn't cut away. So, the editor borrowed one of Artie's lines from a take we weren't going to use and placed it where Larry was speaking. When watching the scene, even though Larry's lips were moving, your focus turned to Artie and your mind ignored the fact that Larry was supposed to be talking.

So, don't forget your audio tracks. Not only is audio half of the medium (even though it takes up a small percentage of the signal), but it also can help you rewrite scenes, cover mistakes, and move your story forward when your video falls short.