Studying and designing visual authoring tools powered by generative AI to support teachers in creating, and learners in learning from, multimodal and interactive agent-based learning activities

Publié le
Unité de recherche
IRISA - UMR 6074
Description du sujet de la thèse

Context: Artificial Intelligence (AI) and Generative AI (GAI) are important priorities in the Digital Europe program [1], with potential to reshape education and workplaces [2, 3, 13]. This proposal examines the design and use of visual authoring tools [4, 5, 10] powered by GAI to support teachers in creating, and learners in learning from, multimodal, interactive agent-based lessons in programming, writing, and concept explanation [e.g., 6, 7, 8], for middle, vocational, and higher education (Figure 1). It's crucial and timely to leverage GAI capabilities to meet the needs of modern education. 

schéma de la figure 1
Figure 1: Overview of our model to support teachers in creating, and learners in learning from, multimodal and interactive agent-based learning activities. A scenario to illustrate this approach: Alex, a teacher, aims to create an interactive lab (activity) to help her students grasp the concepts of loops in JavaScript. Her approach involves several key steps: (1) Multimodal content creation: Initially, Alex plans to employ various forms of content, including texts, images, and animations, to illustrate the context, intricacies, and significance of loops. Multimodal contents will foster students’ engagement and comprehension. (2) Diverse example creation: Alex intends to create various loop examples of different types, such as: for, while, do…while, for of, for in, forEach, for await…of, break, continue, map, and reduce. Various examples will offer students diverse perspectives and applications of loop concepts. (3) Iterative validation of examples: Then, Alex plans to iterate through the examples to ensure their relevance and select the most effective ones for the students. (4) Adaptation to various learning strategies: Alex seeks to adapt the examples into various learning strategies, including fill-in-the-blanks, skeleton, spot and fix errors, code from scratch, etc. Various strategies reinforce learning. (5) Automated feedback integration: Alex aims to implement an automated feedback system within the lab, enabling students to receive suggestions, motivations, hints, and explanations tailored to their individual needs. Personalized feedback enhances motivation and comprehension. (6) Automated assessment integration: Alex plans to incorporate an automated assessment to validate students' completion of each part of the lab. The assessment provides students with immediate evaluation on their outcomes. (7) Insight gathering on student learning: Finally, Alex intends to gather insights into student interaction with the lab (analytics), including metrics, such as time spent, performance levels, and areas of difficulty. These insights enable her to identify student needs and tailor future teaching approaches accordingly.

Challenge: First, creating interactive activities is challenging for teachers, because the authoring process demands proficient knowledge in programming and specialized toolkits [e.g., 9]. Second, existing technologies offer off-the-shelf content (one-size-fits-all), which lack customization, limiting teachers' ability to tailor experiences and content to their needs [e.g., 12]. Third, it is still challenging for teachers to create original content for various types, especially for multimodal content, such as text, audio, image, video, 3D objects, and interactive elements [e.g., 10]. Finally, creating self-paced activities that engage learners is challenging, learners often get stuck, bored, and off-task [e.g., 11].

Lock 1: Why GAI? GAI has wide-ranging applications in multiple fields like generating multimodal content, writing, programming, and question-answering [e.g., 6, 7, 8]. Harnessing GAI capabilities in creating effective educational technologies is crucial and timely for three reasons: (1) enabling personalized learning, fostering exploration and self-regulation; (2) broadening access to quality learning activities; and (3) saving teacher time for more targeted interventions, such as individual coaching, providing feedback, and debriefs [13, 14].

Lock 2: Why visual authoring tools? Visual authoring tools [4] aim to streamline content creation and learning processes via intuitive visual user interfaces [5], catering to non-technical users [e.g., 9, 10]. Creating, and learning from, various interactive activities is cognitive demanding, because users often need to view, filter, analyze, and make sense of various sources of information. Visual external representations [16] have shown to support cognitive offloading and scaffolding [17]. They facilitate learning, writing, collaboration, and problem-solving [e.g., 15, 18].

Lock 3: Gaps in GAI-powered authoring tools? While there is an increasing rise of GAI tools there are still five main gaps [e.g., 2, 3, 6, 7, 8]. Gap 1: Existing GAI (ChatGPT, Stable Diffusion, etc.) lack authoring tools for educators to build interactive learning materials. Similarly, learners can find it challenging to learn from the outputs of current GAI authoring tools. Gap 2: Lack of features tailored to educational needs hinders full exploitation of GAI in teaching and learning. Gap 3: There is a lack of design-based research involving teachers and learners in the design process of GAI tools. Gap 4: There is limited research into impacts/effects of integrating GAI authoring tools in classrooms. Gap 5: There is limited research into socio-technical and ethical implications of GAI tools for education, regarding issues like privacy, bias, harm, and hallucination.

Objectives and Methodology: We aim to collaborate with Canopé and Rennes University to conduct research across middle, vocational, and higher educational levels; and thus examine the transfer of our approach to different contexts. We set to address the following research questions:
1) What do teachers and learners need for GAI authoring tools for programming, writing, and concept explanations?
2) What capabilities of GAI tools for authoring original activities of various types for programming, writing, and concept explanations?
3) How do we design and use visual authoring tools powered by GAI to support teachers in creating, and learners in learning from, multimodal and interactive agent-based learning activities?
4) How to integrate GAI authoring tools into classrooms effectively? What benefits do such tools bring to classrooms, and what challenges might entail?

To this end, we aim to: 1) Deploy participatory approaches to systematically collect the needs and requirements of experts, teachers, and learners for GAI for education and training. 2) Examine through design novel visual authoring tools powered by GAI to support the needs of teachers and learners for multimodal agent-based lessons. 3) Develop design recommendations highlighting opportunities and challenges of  GAI-powered authoring tools, for a better integration in everyday learning environments. 4) Derive guidelines and recommendations to support the provision of digital skills in a human-centric manner regarding GAI technologies.

This research project will be part of the Pinfig authoring tool [22]. Pinfig develops a set of visual authoring tools to support learning, writing, brainstorming, and collaboration.

We will follow an iterative participatory human-centered design [19, 20, 21]. By actively involving experts, teachers, and learners in the design process, the proposal ensures that the tools being developed are aligned with the real-world needs and preferences of those who will be using them. 

Our work will include: (1) expert workshops involving researchers, experts, teachers, and pedagogical engineers, to collect user scenarios and use requirements for GAI authoring tools for education; (2) participatory design workshops involving teachers and learners to collect needs, tasks, design ideas, and early usability testing; (3) design and development involving prototyping visual authoring tools powered by GAI; and finally, (4) in-situ pilots involving experiments with our pilot teachers and their learners.



[1] European commission, “The Digital Europe Programme | Shaping Europe’s Digital Future,” January 16, 2024.

[2] Sætra, Henrik Skaug. “Generative AI: Here to Stay, but for Good?” Technology in Society 75 (November 1, 2023): 102372.

[3] Lim, Weng Marc, Asanka Gunasekara, Jessica Leigh Pallant, Jason Ian Pallant, and Ekaterina Pechenkina. “Generative AI and the Future of Education: Ragnarök or Reformation? A Paradoxical Perspective from Management Educators.” The International Journal of Management Education 21, no. 2 (July 1, 2023): 100790.

[4] Cypher, Allen, et al. No code required: giving users tools to transform the web. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2010.

[5] Shneiderman, Ben, Plaisant, Catherine, Cohen, Maxine, Jacobs, Steven and Elmqvist, Niklas. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 6 Boston: Pearson, 2016.

[6] Denny, Paul, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. “Computing Education in the Era of Generative AI.” Communications of the ACM 67, no. 2 (February 2024): 56–67. 

[7] Gero, Katy Ilonka, Tao Long, and Lydia B Chilton. “Social Dynamics of AI Support in Creative Writing.” In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–15. Hamburg Germany: ACM, 2023.

[8] Kazemitabaar, Majeed, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, and Tovi Grossman. “CodeAid: Evaluating a Classroom Deployment of an LLM-Based Programming Assistant That Balances Student and Educator Needs.” arXiv, January 20, 2024.

[9] Ez-zaouia, Mohamed, Iza Marfisi-Schottman, Maysa Oueslati, Cendrine Mercier, Aous Karoui, and Sébastien George. “A Design Space of Educational Authoring Tools for Augmented Reality.” In Games and Learning Alliance, 13647:258–68. Cham: Springer International Publishing, 2022.

[10] Ez-Zaouia, Mohamed, Iza Marfisi-Schottman, and Cendrine Mercier. “Authoring Tools: The Road To Democratizing Augmented Reality For Education.” In Proceedings of the 15th International Conference on Computer Supported Education, 1:115–27, 2023.

[11] Baker, Ryan Shaun, Albert T. Corbett, Kenneth R. Koedinger, and Angela Z. Wagner. “Off-Task Behavior in the Cognitive Tutor Classroom: When Students ‘Game the System.’” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 383–90. CHI ’04. New York, NY, USA: Association for Computing Machinery, 2004.

[12] Vermette, Laton, Joanna McGrenere, Colin Birge, Adam Kelly, and Parmit K. Chilana. “Freedom to Personalize My Digital Classroom: Understanding Teachers’ Practices and Motivations.” In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 318:1-318:14. CHI ’19. New York, NY, USA: ACM, 2019.

[13] Chiu, Thomas K. F. “The Impact of Generative AI (GenAI) on Practices, Policies and Research Direction in Education: A Case of ChatGPT and Midjourney.” Interactive Learning Environments 0, no. 0 (2023): 1–17.

[14] Smolansky, Adele, Andrew Cram, Corina Raduescu, Sandris Zeivots, Elaine Huber, and Rene F. Kizilcec. “Educator and Student Perspectives on the Impact of Generative AI on Assessments in Higher Education.” In Proceedings of the Tenth ACM Conference on Learning @ Scale, 378–82. L@S ’23. New York, NY, USA: Association for Computing Machinery, 2023.

[15] Ez-Zaouia, Mohamed, and Rubiela Carrillo. "The Group Folding Effect: The Role of Collaborative Process Structuring and Social Interaction in Group Work."  ACM Trans. Comput.-Hum. Interact. 31, 2, Article 15 (April 2024), 44 pages.

[16] Walny, J., S. Carpendale, Nathalie Henry Riche, G. Venolia, and P. Fawcett. “Visual Thinking in Action: Visualizations as Used on Whiteboards.” IEEE Transactions on Visualization and Computer Graphics 17, no. 12 (December 2011): 2508–17.

[17] Risko, Evan F., and Sam J. Gilbert. “Cognitive Offloading.” Trends in Cognitive Sciences 20, no. 9 (September 2016): 676–88.

[18] Binks, A., Toniolo, A., & Nacenta, M. A. (2022). Representational transformations: Using maps to write essays. International Journal of Human-Computer Studies, 165, 102851.

[19] Muller, Michael J., and Sarah Kuhn. “Participatory Design.” Commun. ACM 36, no. 6 (June 1993): 24–28.

[20] Zimmerman, John, Jodi Forlizzi, and Shelley Evenson. “Research through Design as a Method for Interaction Design Research in HCI.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 493–502. San Jose California USA: ACM, 2007.

[21] Barab, S., & Squire, K. (2016). Design-based research: Putting a stake in the ground. In Design-based Research (pp. 1-14). Psychology Press.

[22] Ez-Zaouia Mohamed, “Pinfig – Brainstorm. Learn. Write. Collaborate. Visually”, ( n.d.)

Liste des encadrants et encadrantes de thèse

Nom, Prénom
Miklos, Zoltan
Type d'encadrement
Directeur.trice de thèse
Unité de recherche

Nom, Prénom
Ez-Zaouia Mohamed
Type d'encadrement
Unité de recherche
Ez-Zaouia Mohamed
06 98 67 68 46
AI, Generative AI, Authoring Tools, Visualization, Multimodality, Participatory Design, Education