In this article, we present an experiment and its results regarding the use of ChatGPT in Mathematics with Commerce and Marketing freshmen students of the Faculty of Economics of the University of Debrecen.

The purpose of the experiment was to make the subject of Mathematics more interesting for students and to familiarise them with the limitations of using ChatGPT 3.5.

We asked ChatGPT to generate exercises and corresponding solutions for each student within the topics of combinatorics, classical probability, and Bayes’ Theorem. Next we asked the students to solve the exercises assigned to them and crosscheck the solution generated by ChatGPT. With this, we wanted to ensure that students learn about the limitations of this tool and later use ChatGPT with sufficient criticism.

As part of this exercise, we have checked the types of exercises generated by ChatGPT, their variety, degree of difficulty, the correctness of the solutions of our students and that of the generated solutions by ChatGPT.

This was done to make an educated decision on whether ChatGPT can be used for task generation and how reliable the solution it provides is. We must remember that ChatGPT is a language model and places words next to each other on a probabilistic basis. In Mathematics exercises, a single word can often play an important role, just think of the difference of drawing balls from a box with or without repetition.

Based on the definition of a language model, we assumed before the experiments that ChatGPT is not really suitable for solving mathematical exercises, but is capable of generating many different and varied exercises. This would suggest that it can be a useful helper for a Mathematics teacher in creating content for practicals and revision exercises.

In the paper we show the difficulty and variety of the exercises we receive and review how far AI can be trusted to solve these problems, as well as how difficult was solving these exercises for our students.

Keywords: ChatGPT, teaching experience, mathematics, probability, generating exercices.