ABSTRACT VIEW
USING AN LLM-BASED FRAMEWORK TO ANALYZE THE STUDENT PERFORMANCE
A. Fernández-Isabel, C. Lancho, I. Martín de Diego, A. Udías, A. Alonso-Ayuso, C. Alfaro, E. López, F. Ortega, J. Gómez, J. M. Moguerza, M.J. Algar
Rey Juan Carlos University (SPAIN)
Teaching has already adopted Technologies of Communications and Information (TCI) for learning tasks in multiple areas of education. In this context, the most recent advances in Artificial Intelligence (AI) related to chatbots and large language models (LLMs) as assistants have been a revolution. However, these systems provide more global solutions that may fail when addressing more specific educational needs.

On the other hand, in the current digital era, the communication paradigm has shifted, and younger generations may feel more comfortable communicating through digital platforms rather than face-to-face interactions. Besides, for some students, face-to-face teacher-student academic tutoring may be challenging for different reasons, like schedule, shyness, insecurity, etc. All these situations harm the development of subjects. Thus, developing new systems to provide particular assistance or tutoring to students is key.

Following this line, this work presents ViLT, an intelligent virtual tutor designed to encapsulate a subject's specific and necessary knowledge. ViLT acts as an always-available online mentor, helping students solve their questions about the subject efficiently and truthfully while studying. The system combines the traditional dynamic of teacher-student academic tutoring with the most cutting-edge AI technology to tackle the evolving needs of the students.

ViLT is adaptable for every subject and the needs of teachers. Teachers are responsible for feeding the virtual tutor by providing the subject materials. This ensures that the tutor is tailored to the subject matter and that it offers appropriate responses. Moreover, ViLT gathers information about the students related to the questions and interactions made, and the feedback about the tool provided by the students regarding its utility and capability to solve problems. In this sense, keywords, tutoring time, length, and number of conversations are evaluated.

The virtual tutor can help in two distinct manners:
(1) by encouraging students to ask doubts and
(2) by providing teachers with valuable information regarding which are the doubts of the students, where they are having more trouble, which part of the subject requires more attention and practice, etc. and, consequently, teachers can adapt and improve the subject according to all this information.

In particular, ViLT has been used in three subjects at Rey Juan Carlos University: Statistical Inference from the Data Science Degree, Data Mining from the Mathematics Degree, and Advanced Computer Architectures from the Computer Science Degree. The data collected by the tool has been analyzed to extract insights into the most frequent topics, additional exercises demand, number of questions, number of students using ViLT, and degree of satisfaction. These findings have been compared to classic face-to-face tutoring.

The results are confirmatory. Thus, ViLT has improved the quality of the learning process in educational settings, facilitating the resolution of doubts, and providing other academic assistance to students on specific topics. It could lead to new educational trends and techniques where teachers can use these systems to provide continuous and complementary support to their students.

Keywords: Virtual tutor, Student support, Student performance analysis, Large language models, Artificial Intelligence.

Event: INTED2025
Track: Innovative Educational Technologies
Session: Generative AI in Education
Session type: VIRTUAL