The Interplay Between Gestures & AI Language Learning in Language Acquisition

    In today’s global day and age, learning a second language and multilingualism is highly valued, both in the workforce and as a societal tool. AI-powered language learning tools like Duolingo and generative AI tutors are taking advantage of this increased interest in multilinguality and rapidly rising in popularity. AI-chatbots provide accessible and interactive ways for students to practice conversation skills and have shown to improve vocabulary and grammatical proficiency. However, they often lack nonverbal cues like gestures, which play a crucial role in communication. An important question arises: is the lack of nonverbal gesture in AI-powered language learning models detrimental to long term language learning?

    In the research paper  “Language Proficiency Impacts the Benefits of Co-Speech Gesture for Narrative Understanding Through a Visual Attention Mechanism”, Dr. Wakefield investigated how gestures in a spoken narrative affect children’s comprehension by tracking their visual attention and recall. Children listened to stories in both their primary language (English) and second language (Polish), with gestures that either matched or slightly differed from speech. Results showed that matching gestures significantly improved recall in Polish (their second language) but had little effect in English (their first language), suggesting that gestures play a greater role in comprehension when used in a listener’s less proficient language. Additionally, results demonstrated that mismatching gestures in both languages actually hindered language learning and led to a greater misunderstanding of the story’s narrative. 

    These findings could have direct implications for AI-driven models of language learning. Evidently, physical gestures are a highly valuable aspect of language-learning, especially for multilingual children. In a pilot study done by Dr. Petra Polakova and Dr. Blanka Klimova, a 10-15% improvement in students’ language learning was demonstrated in using AI-language learning models and chatbots, proving that “conversational chatbots can significantly improve foreign language acquisition among university students”. However, the research may be limited in its applications to Dr. Wakefield’s study, as this pilot study was done on university students rather than children. Additionally, the study identified limits in reviewing the efficiency of AI-language learning, including the lack of long-term examination of students’ language proficiency. Because these chatbots did not contain interactive gestures, their efficiency in enhancing comprehension may be limited compared to human interaction, particularly if assessed in children. 

    Combining the ideas of these studies, future research should focus on integrating synchronized, research-backed gestures and complex, refined conversations into their programs in order to enhance comprehension, particularly for second-language learners. However, gestures and animations in AI-language learning must be carefully reviewed and pre-programmed, as Dr. Wakefield demonstrated the harmful effects of mismatching gestures and potential misunderstanding. By refining both verbal and nonverbal aspects of AI-driven language learning, future advancements can create more immersive and effective educational tools. 

References

1. Natalia Zielinski, Elizabeth Wakefield: Language Proficiency Impacts the Benefits of Co-Speech Gesture for Narrative Understanding Through a Visual Attention Mechanism. CogSci 2021

2. Polakova, P., & Klimova, B. (2024). Implementation of AI-driven technology into education – a pilot study on the use of chatbots in foreign language learning. Cogent Education, 11(1). https://doi.org/10.1080/2331186X.2024.2355385