When Words Don’t Do Enough: How Our Brains Connect the Gaps With Gestures

When Words Don’t Do Enough: How Our Brains Connect the Gaps With Gestures

People tend to think of language as the brain’s main form of communication. What if the brain doesn’t only rely on the auditory information it receives but just as much on visual stimulation, especially when understanding is hard?

In this course, we have learned that language is not processed as a single channel but as a dynamic, multimodal system by the brain. This becomes clearer when we look at how people comprehend communication in ordinary situations. The translation of words is but a small portion of the huge work of communicating. Effective communication requires attention, prediction, and integration of different brain systems. A recent research study of bilingual youngsters has demonstrated that the integration of different facets of communication is important. Researchers observed that children used more hand motions that accompany speech when they listened to stories in their weaker language. In fact, individuals were more likely to remember details of a story when the gestures (hands)  in their weaker language matched the verbal meaning, but not in their primary language.   

That implies something more profound about how the brain reacts to stress. When language becomes more difficult to understand, the brain doesn’t just try harder; it compensates. In reality, the young people in the study, after having processed their inferior language, shifted their visual attention and paid more attention to the speaker’s hands. The increased attention meant gestures had a greater effect on memory and understanding.

But not all gestures were that helpful. Sometimes, gestures can be detrimental to comprehension, in particular in the stronger language, when they provide redundant or incongruent information. This demonstrates a very important constraint: extra cues can benefit the brain, but only if they are in addition to the message. Otherwise, they interfere with processing and increase cognitive load.

This finding is directly relevant to more general notions in neuroscience about resource allocation and attention. The brain's processing power is finite and thus is constantly making choices about what to focus on. Less capable systems (e.g., language understanding) recruit other systems (e.g., visual processing) to help them. In this case, gesture acts as a kind of “back-up system” for understanding.

This has implications for the real world, outside of the lab. Take into account classes in which the learners are learning a language other than their mother tongue. Teachers are natural users of gestures or movement, and research shows the importance of gestures for understanding. The barriers of language often pose problems in international communication. Even in ordinary conversations, gestures can be more significant than we realize in transmitting complex ideas.

This also raises interesting considerations about the evolution of communication. We live in a text-heavy world of emails, messages, and AI-generated replies, and in doing so, we are stripping away one of the brain’s most powerful processing tools: visual, embodied clues. If gestures help understanding when words become hard, what happens when these clues are entirely removed?

This study ultimately re-conceptualizes our idea of communication. It is more than knowing the right words to say or speaking in a clear voice. It’s about how the brain makes meaning out of various streams of information, particularly when it is challenged.

The crucial lesson is not just that gestures are helpful, but rather that the brain is continually changing and trying to make sense of the world with whatever information it has. So the real question is: are we communicating as well as we think we are? Or, since understanding depends on more than just words, are we shutting part of the brain out of the conversation?