Goat or motorcycle? Sound radically alters our visual perception

In a well-known illusion called the McGurk effect, we see a person pronouncing the “word” ba over and over again. But if the lip movements change to correspond to another word, we hear something else, even though the sound emanating from the person’s lips has not changed. This illusion shows that visual information plays a large role in how we hear speech. 

Research published in the journal Psychological Science now shows that sound has a big influence on what we see. Jamal Williams of the University of California-San Diego and his colleagues created a set of ambiguous images by combining the features of pairs of objects, such as a cat and kettle, an airplane and bird, or a goat and Vespa scooter.

They then showed these images to participants, together with natural sounds that were related or unrelated to one of the objects, and asked them to decide what they had seen by clicking a button. For example, when viewing an image created by morphing an airplane with a bird, they heard the sound of an airplane, a bird, or a hammer.

The researchers found that related sounds strongly influence how the participants perceived the ambiguous images. In the example above, hearing the sound of an airplane or a bird shifted their processing of the image, so that they saw the object that matched the sound they heard. The sounds also affected the participants’ response time. When they heard an unrelated sound, it took them longer to report what object they saw. 

Two further experiments involved exactly the same procedure, except that this time the sounds were played after presentation of the images, or just before, with varying amounts of time between the sound and the image. In both cases, the sounds did not affect the participants’ judgments.

Together, these results show that the sounds altered visual processing of the images, and not the participants’ decision-making or prior knowledge, which is also known to influence visual perception. 

Hearing is believing

Earlier studies have shown that sound can enhance early stages of visual processing, make moving images clearer, and make higher-level visual processing tasks such as object recognition faster and more accurate. 

This new study shows that sound can have more dramatic effects on visual processing, and shift the visual representations of objects from one category to a completely different one — but only when images and sound overlap in time.

The findings show that visual perception is an on-going, active process which, rather than being isolated, involves multi-sensory integration, with sounds triggering inferences that are used to interpret ambiguous images.