Modern artificial intelligence tools, particularly large language models (LLM), also find applications in evaluating students' knowledge in IT fields. A specific subset in this area is the teaching of programming. Traditionally, classical programming tasks are evaluated using so-called autograders, but their code assessment capabilities are mainly based on verifying the functional correctness of the code. However, artificial intelligence, especially LLMs, provides opportunities to automate the assessment process and refine and extend the assessment of programming knowledge and skills. One way is, for example, to generate tests for auto-grading. More interesting, however, is the possibility of assessing code quality and style and adherence to required procedures or algorithm efficiency.

This paper discusses a particular application of using LLM to automate the assessment in the teaching of programming. A pilot solution that enables automatic evaluation of programming tasks without the teacher's need for manual intervention will be presented. The system supports the assignment of tasks, the interactive definition of evaluation criteria and their weights, and subsequent analysis of the submitted code and generation of feedback, while running the code is not a necessary part of the process. Thus, using language models increases the assessment's efficiency and objectivity while reducing the educator's involvement. The presented application has been tested in a real classroom with positive results. The paper will also discuss the experience gained and possible ways for future development and expansion of the application.

Keywords: AI, LLM, teaching programming, knowledge evaluation.