Modeling the multimodality of language production
I’ve modeled the multimodality of language production for three main reasons. First, one of my main interests is explanation and argumentation. In 2003, I defended a PhD in Cognitive Science on explanatory activity in interaction where I analyzed how physics teachers in training explained videos of students’ lab work: The importance of gaze and gesture in interactive multimodal explanation(1). But what are the general competencies involved in collaborative explanation? Colleagues and I studied how such competencies evolved throughout childhood (e.g. gradual mastery of the explanation’s cognitive content, but also gradual mastery of managing the explanatory interaction) and also how context makes a difference in how explanations are given: Explanatory content and visibility effects on the young child’s verbal and gestural behavior in free dialogues(2) and Alors, là, je vais vous expliquer comment on joue ! » Explication procédurale finalisée dans une perspective développementale [Ok, I’m going to teach you how to play! Finalized procedural explanation within a developmental perspective(3).
Second, learning physics proved to be fertile ground for further study of the multimodal elements of language that make up problem solving where we show how students’ understanding of conceptual content is expressed not only through words, but also through coverbal gestures, and the manipulation of lab work objects such as batteries, wires, and lightbulbs: Learning physics as coherently packaging multiple sets of signs(4); Les fonctions cognitives de la reformulation multimodale et polysémiotique dans deux situations didactiques en physique [Cognitive functions of multimodal and polysemiotic reformulation in two physics teaching-learning situations](5), and Conceptual Change and Sustainable Coherency of Concepts Across Modes of Interaction(6).
Third, research in the Jen.lab project on epistemic games led to a collaboration with David Williamson Shaffer on the analytical interest of connecting levels of analysis (e.g. verbal, co-verbal, para-verbal aspects of human communication with more macro level descriptions of specific sequences of human interaction Gaining Insight by Transforming Between Temporal Representations of Human Interaction(7). Our collaboration on using a new version of Epistemic Network Analysis called Transmodal Analysis to analyze multimodal data continues with GeDéco (pdf in French). This project analyzes the mumtimodal interactions of children explaining and playing a collaborative game called Hanabi. In a focus on how intersubjectivity is constructed across species, these analyses will also be compared to the spontaneous play of bonobos and chimpanzees in the Let’s Play project (pdf in French), one of the 8 large projects, financed by ASLAN, a laboratory of excellence I have led since 2018.
(1) Lund, K (2007). The importance of gaze and gesture in interactive mulitmodal explanation. International Journal of Language Resources and Evaluation, 2007, 41 (3-4), pp.289-303. ⟨1007/s10579-007-9058-0⟩. ⟨halshs-00373412⟩
(2) Mazur-Palandre, A. & Lund, K (2016). Explanatory content and visibility effects on the young child’s verbal and gestural behavior in free dialogues. Language, Interaction and Acquisition. 7(2), 180-211.
(3) Mazur-Palandre, A., Colletta, J.M. & Lund, K. (2021) « Alors, là, je vais vous expliquer comment on joue ! » Explication procédurale finalisée dans une perspective développementale [Ok, I’m going to teach you how to play! Finalized procedural explanation within a developmental perspective. Langages, Armand Colin (Larousse jusqu’en 2003).
(4) Lund, K., Bécu-Robinault, K. (2010). Learning physics as coherently packaging multiple sets of signs. In Gomez, K., Lyons, L., & Radinsky, J. (Eds.) Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010), (Vol I, pp. 404-411). International Society of the Learning Sciences: Chicago IL.
(5) Lund, K. & Becu-Robinault, K. (2010). Les fonctions cognitives de la reformulation multimodale et polysémiotique dans deux situations didactiques en physique. Dans A. Rabatel (Ed.), Les reformulations pluri-sémiotiques en contexte de formation (pp. 211-244). Presses universitaires de Franche-Comté. halshs-00376594.
(6) Lund, K. & Bécu-Robinault, K. (2013). Conceptual Change and Sustainable Coherency of Concepts Across Modes of Interaction. In D. D. Suthers, K. Lund, C. P. Rosé, C. Teplovs & N. Law (Eds.), Productive Multivocality in the Analysis of Group Interactions. In C. Hoadley & N. Miyake (Series Eds.), Computer Supported Collaborative Learning Series: Vol. 15 (pp. 311-330). New York: Springer.
(7) Lund, K., Quignard, M., & Shaffer, D.W. (2017). Gaining Insight by Transforming Between Temporal Representations of Human Interaction. Journal of Learning Analytics. 4(3) 102-122. 10.18608/jla.2017.43.6