Studying and designing visual authoring tools powered by generative AI to support teachers in creating, and learners in learning from, multimodal and interactive agent-based learning activities

Context: Artificial Intelligence (AI) and Generative AI (GAI) are important priorities in the Digital Europe program [1], with potential to reshape education and workplaces [2, 3, 13]. This proposal examines the design and use of visual authoring tools [4, 5, 10] powered by GAI to support teachers in creating, and learners in learning from, multimodal, interactive agent-based lessons in programming, writing, and concept explanation [e.g., 6, 7, 8], for middle, vocational, and higher education (Figure 1). It's crucial and timely to leverage GAI capabilities to meet the needs of modern education. 

Image

Figure 1: Overview of our model to support teachers in creating, and learners in learning from, multimodal and interactive agent-based learning activities. A scenario to illustrate this approach: Alex, a teacher, aims to create an interactive lab (activity) to help her students grasp the concepts of loops in JavaScript. Her approach involves several key steps: (1) Multimodal content creation: Initially, Alex plans to employ various forms of content, including texts, images, and animations, to illustrate the context, intricacies, and significance of loops. Multimodal contents will foster students’ engagement and comprehension. (2) Diverse example creation: Alex intends to create various loop examples of different types, such as: for, while, do…while, for of, for in, forEach, for await…of, break, continue, map, and reduce. Various examples will offer students diverse perspectives and applications of loop concepts. (3) Iterative validation of examples: Then, Alex plans to iterate through the examples to ensure their relevance and select the most effective ones for the students. (4) Adaptation to various learning strategies: Alex seeks to adapt the examples into various learning strategies, including fill-in-the-blanks, skeleton, spot and fix errors, code from scratch, etc. Various strategies reinforce learning. (5) Automated feedback integration: Alex aims to implement an automated feedback system within the lab, enabling students to receive suggestions, motivations, hints, and explanations tailored to their individual needs. Personalized feedback enhances motivation and comprehension. (6) Automated assessment integration: Alex plans to incorporate an automated assessment to validate students' completion of each part of the lab. The assessment provides students with immediate evaluation on their outcomes. (7) Insight gathering on student learning: Finally, Alex intends to gather insights into student interaction with the lab (analytics), including metrics, such as time spent, performance levels, and areas of difficulty. These insights enable her to identify student needs and tailor future teaching approaches accordingly.

Challenge: First, creating interactive activities is challenging for teachers, because the authoring process demands proficient knowledge in programming and specialized toolkits [e.g., 9]. Second, existing technologies offer off-the-shelf content (one-size-fits-all), which lack customization, limiting teachers' ability to tailor experiences and content to their needs [e.g., 12]. Third, it is still challenging for teachers to create original content for various types, especially for multimodal content, such as text, audio, image, video, 3D objects, and interactive elements [e.g., 10]. Finally, creating self-paced activities that engage learners is challenging, learners often get stuck, bored, and off-task [e.g., 11].

Lock 1: Why GAI? GAI has wide-ranging applications in multiple fields like generating multimodal content, writing, programming, and question-answering [e.g., 6, 7, 8]. Harnessing GAI capabilities in creating effective educational technologies is crucial and timely for three reasons: (1) enabling personalized learning, fostering exploration and self-regulation; (2) broadening access to quality learning activities; and (3) saving teacher time for more targeted interventions, such as individual coaching, providing feedback, and debriefs [13, 14].

Lock 2: Why visual authoring tools? Visual authoring tools [4] aim to streamline content creation and learning processes via intuitive visual user interfaces [5], catering to non-technical users [e.g., 9, 10]. Creating, and learning from, various interactive activities is cognitive demanding, because users often need to view, filter, analyze, and make sense of various sources of information. Visual external representations [16] have shown to support cognitive offloading and scaffolding [17]. They facilitate learning, writing, collaboration, and problem-solving [e.g., 15, 18].

Lock 3: Gaps in GAI-powered authoring tools? While there is an increasing rise of GAI tools there are still five main gaps [e.g., 2, 3, 6, 7, 8]. Gap 1: Existing GAI (ChatGPT, Stable Diffusion, etc.) lack authoring tools for educators to build interactive learning materials. Similarly, learners can find it challenging to learn from the outputs of current GAI authoring tools. Gap 2: Lack of features tailored to educational needs hinders full exploitation of GAI in teaching and learning. Gap 3: There is a lack of design-based research involving teachers and learners in the design process of GAI tools. Gap 4: There is limited research into impacts/effects of integrating GAI authoring tools in classrooms. Gap 5: There is limited research into socio-technical and ethical implications of GAI tools for education, regarding issues like privacy, bias, harm, and hallucination.

Objectives and Methodology: We aim to collaborate with Canopé and Rennes University to conduct research across middle, vocational, and higher educational levels; and thus examine the transfer of our approach to different contexts. We set to address the following research questions:
1) What do teachers and learners need for GAI authoring tools for programming, writing, and concept explanations?
2) What capabilities of GAI tools for authoring original activities of various types for programming, writing, and concept explanations?
3) How do we design and use visual authoring tools powered by GAI to support teachers in creating, and learners in learning from, multimodal and interactive agent-based learning activities?
4) How to integrate GAI authoring tools into classrooms effectively? What benefits do such tools bring to classrooms, and what challenges might entail?

To this end, we aim to: 1) Deploy participatory approaches to systematically collect the needs and requirements of experts, teachers, and learners for GAI for education and training. 2) Examine through design novel visual authoring tools powered by GAI to support the needs of teachers and learners for multimodal agent-based lessons. 3) Develop design recommendations highlighting opportunities and challenges of  GAI-powered authoring tools, for a better integration in everyday learning environments. 4) Derive guidelines and recommendations to support the provision of digital skills in a human-centric manner regarding GAI technologies.

This research project will be part of the Pinfig authoring tool [22]. Pinfig develops a set of visual authoring tools to support learning, writing, brainstorming, and collaboration.

We will follow an iterative participatory human-centered design [19, 20, 21]. By actively involving experts, teachers, and learners in the design process, the proposal ensures that the tools being developed are aligned with the real-world needs and preferences of those who will be using them. 

Our work will include: (1) expert workshops involving researchers, experts, teachers, and pedagogical engineers, to collect user scenarios and use requirements for GAI authoring tools for education; (2) participatory design workshops involving teachers and learners to collect needs, tasks, design ideas, and early usability testing; (3) design and development involving prototyping visual authoring tools powered by GAI; and finally, (4) in-situ pilots involving experiments with our pilot teachers and their learners.