With the launchof ChatGPT, a conversational Artificial Intelligence (AI) model, in November 2022, Open AI has established a new benchmark for human-computer interaction by simulating human-like conversations. Generative AI, particularly tools like ChatGPT, hold immense potential to revolutionize numerous fields and industries. Despite its transformative potential in education and research, its application requires careful and responsible consideration . On the one hand, it can be used to enhance self-determined learning, improve learning performance, and provide personalized support. On the other hand, the use of these tools raises concerns about academic integrity, plagiarism and the potential impact on critical thinking skills. Therefore, this paper aims to investigate the impact of using ChatGPT on the quality of student papers, to understand how AI tools affect academic performance, both positively and negatively. This leads to the following research question: What impact does the use of Artificial Intelligence (AI) tools in the scientific writing process have on students’ performance?

In the first-semester Bachelor’s degree course for the introduction to academic writing, 51 students aged between 18 and 21 were tasked to write a paper about a topic relevant to the degree program "Management, Communication and IT". Students were encouraged to use all available supporting AI tools. Based on this task, two data sets have emerged that are available for further analysis. Firstly, these are the papers written by the students. Secondly, students were asked to complete a comprehensive questionnaire to assess the intensity of use and usefulness of AI tools at different stages of the academic writing process. These stages encompass starting with an initial topic overview, narrowing down the topic, developing the research question, conducting a literature review, summarizing and paraphrasing, structuring the paper, and enhancing grammar and writing style. The intensity of use and usefulness was assessed using a 5-point Likert scale.

Although AI tools are frequently used by students, the results of this study indicate that their overall impact on academic performance is minimal, with the exception of improving grammar and writing style. The regression analysis demonstrated that neither the intensity nor the perceived usefulness of AI tools significantly predicted academic performance, with models explaining only a small proportion of the variance in grades.

Keywords: ChatGPT, Large Language Models, higher education, empirical study.