The integration of Large Language Models (LLMs) with educational tools offers significant potential to enhance student learning by providing tailored, contextually accurate responses through chatbots. This thesis investigates the implementation of Retrieval Augmented Generation (RAG) systems to augment LLMs for academic use, particularly in assisting students with high school Physics and undergraduate Mathematics courses. The research involves an experimental approach where various RAG configurations were systematically constructed and tested. Each configuration combined different parameters and tools, including small and large language models, various text splitting techniques, and diverse vector stores for semantic search.

Synthetic question-answer pairs were generated using course materials from MIT’s OpenCourseWare to evaluate the performance of these configurations. The experiments aimed to identify which RAG setups could most effectively retrieve and generate relevant answers from the provided educational content. Results revealed that the highest performing RAG configurations achieved over 64% accuracy in Physics and 66% in Mathematics. These configurations varied in chunk sizes, overlaps, and the specific models used, highlighting the nuanced impact of each parameter on performance.

The study concludes that while LLM size and complexity play a role, the choice of text splitters and embedding models are critical in optimizing RAG systems for academic applications. The findings suggest that carefully tuned RAG-powered LLMs can serve as effective virtual teaching assistants, offering significant benefits in terms of accessibility and accuracy of academic support. This research provides a foundational framework for future enhancements in educational chatbots, emphasizing the importance of open-source tools and ethical considerations in their development and deployment.

Keywords: Educational Chatbots, Physics and Mathematics Education, Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), Synthetic Data Generation.