I. Tomažič, A. Pšeničnik, J. Mravlje
Large language models (LLMs) are revolutionising education by offering personalised learning experiences, automating tasks, and enhancing accessibility. MagicSchool AI (https://www.magicschool.ai/) uses LLMs to help educators save time and enhance teaching by generating educational content and streamlining tasks, thus allowing them to focus on more complex roles. While LLMs offer significant benefits, it is crucial to be aware of potential issues like over-reliance and the need for human oversight in evaluating LLM-generated content.
The present research was a part of the Slovenian project Modernisation of pedagogical study programmes - Modernisation of the NOO PSP following digitalisation and sustainability (2023-2026), which is funded by the Ministry of Education and the European Union - NextGenerationEU. Two generations of students, pre-service biology teachers, were included in the study (n=16). In the academic year 2023/2024, nine out of ten students participated; in the study year 2024/2025, seven out of ten students participated. The study was conducted within a regular special didactics subject, named Biological practicum for teachers. First, students were delivered the questionnaire (pre-test) using a Likert-type scale (15 items) about:
(1) affective viewpoints regarding AI,
(2) their perceived competences for using AI,
(3) their perceived competences for using AI in instruction (teaching),
(4) their acceptance of using AI tools in teaching and
(5) their self-efficacy beliefs regarding using AI tools.
An open-ended question about their opinions on using AI tools was placed in the last part of the questionnaire.
Following the questionnaire, a short teacher-centred lecture about the cell cycle was introduced to present the teacher-centred approach to the students. This brief lecture was prepared using MagicSchool AI with graphical and video materials from an existing upper secondary biology textbook; however, students were not told about the use of AI. Next, they were instructed to individually design their teacher-centred lesson plan for teaching about cellular respiration in upper secondary school. This phase was evaluated according to students' prior teaching experiences. Afterwards, they were informed that the previous lecture was prepared using MagicSchool AI, and the tool was introduced briefly. Students then explored the tool, compared their lesson plans with the AI-generated ones, and followed with a discussion about their insights. After all the mentioned activities, students were again delivered the same questionnaire (post-test).
Results show that most first-generation students did not use AI (LLM) tools, while all second-generation students were AI users and were more familiar with different AI tools. The aforementioned results led to more positive belief ratings of second-year students on all belief dimensions (pre-test). Because of a small sample size, the effect sizes were calculated using the formula r = Z/SQRT(n), in addition to using the Mann-Whitney nonparametric test. The same applied to the assessment of differences resulting from our teaching. Regardless of generation, pre- versus post-test belief ratings were more positive after completing activities (Wilcoxson test and effect sizes). However, the differences in the second-generation students' ratings were not as high as those in the first-generation students. The implications for pre-service teachers' education will be discussed.
Keywords: Generative AI, large language models, MagicSchool AI, views, pre-service teachers, biology.