Say have you heard the one about the McGurk effect?

We normally think about sight and sound as being different senses. What we hear and what we see are independent of each other. So it was something of a surprise when researchers discovered that what we see can actually change what we hear.

The McGurk Effect, named after Harry McGurk, one of the researchers who first discovered this strange reality, shows that what you see is what you hear. The experiments involved taking videos of people saying various different syllables. Syllables such as Ba and Ga are sounds that are the building blocks of all spoken words. The videos were edited to replace the recorded sound with another prerecorded syllable sound. The new edited video was then played to an observer and they were asked what the person in the video was saying. What the researchers found was that observers always go for the syllable they see being spoken on the video rather than the actual sound being heard.

This effect is very general, it works with people from all language backgrounds, young children, or even when you mix male and female faces and voices. What this strange McGurk illusion shows is that our brain is using the visual information to work with even when it's in direct contradiction to the actual sound. Your eyes help you hear. Stranger still the effect has also been shown to work when observers touch rather than look at the face, so it's not just about sight.

Understanding how our brain works in going from measurements of physical things like light wavelength, which is the starting point for vision, or changes in air pressure, the starting point for sound, to create the perceptions of seeing and hearing is a fascinating fundamental problem. All this information comes into our brain and somehow it turns into our perception of the world, but how on earth does it do it? At present we have some clever ideas, and some intriguing clues, but the problem has not been solved yet.

To try and unravel this human mystery will take many more years of hard work, and it will need a range of different skills; from the psychologist doing experiments to the mathematicians and computer scientist trying to build and test computer simulations of the process, but clever clues like the illusion of the McGurk effect can help us along the way. It is also important if we are to build computers and robots that we find more naturally easy to talk with. It suggests they need to get more than just the sounds right, but the lips too!

Try it yourself with the Face Base Rap Experiment