Modeling the multimodality of language production

Modeling the multimodality of language production

In this theme, I am interested in exploring the co-construction of gesture and talk both from the point of view of a single interlocutor’s production as well as from the point of view of interacting participants. The work in a developmental psycholinguistics context is on-going and has been accomplished thanks to Audrey Mazur-Palandre and Jean-Marc Colletta. Here, I seek to describe such co-elaborations both within and across participants with the goal of modeling multimodal language use in the context of interactive explanation. I am doing other work in this theme with Karine Robinault that is inspired by the notion of semiotic bundles (Arzarello) where modeling language production includes the manipulation of resources as well as gesture and talk; we are focusing on the nature of multimodal and multimedial reformulation in teachers’ expert talk and in learners’ novice talk in the physics classroom and on how practices of reformulation can either favor or hinder learning. Finally, in the context of the ANR JEN.lab project, and in particular with David Shaffer and Matthieu Quignard, Vicky Markaki leads multimodal analyses on digital epistemic games.