AI chatbots based on LLMs, such as ChatGPT, have entered our lives and our schools. Both students and teachers are exploring the potentials and draw-backs of these kind of AI tools. This study aims to explore to what extend explicit instructions contribute to the use and perceived usefulness of AI chatbots by students, and whether there is a difference in use of AI chatbots for different students, regarding characteristics like their motivation and prior skills. The use of AI chatbots was evaluated in different scenario’s in different courses within the bachelor and master programme Food Technology at Wageningen University in the academic year 2023-2024. The scenario’s differed in type of assignment, amount of instructions on the use of AI chatbots and in whether the use of AI chatbots was allowed. Student experiences were gathered by means of questionnaires with between 93 and 261 students per scenario. In assignments with explicit instructions on how to write prompts more students used AI chatbots (75% and 82%) than in assignments with no prompt instructions (53% and 54%). When the use of AI chatbots was not allowed still 30% of the students indicated to have used an AI chatbot. Students regarded the outcome of the AI chatbots comparable useful to or slightly less useful than information found on the internet. Information sources provided in a course, by other students and by teachers were all in general regarded as more useful than the outcomes of AI chatbots. No significant differences were found between the % of students that used AI chatbots, the perceived usefulness, and the intended grade of the student for that assignment. Regarding prior skills, students who indicate to have good writing skills and/or adequate English skills in general used AI chatbots less in the writing assignment than students who indicated they had poor writing and/or inadequate English language skills. The three main reasons given by students why they did not use AI chatbots are that they feel like it is not needed, that the AI chatbot is unreliable, and that they want to learn themselves. In conclusion, it is advisable to warn students about the drawbacks of using AI chatbots, give them clear prompt instructions if using these tools is allowed and especially when it comes to writing assignments, explain students how to deal with the outcomes.

Keywords: AI chatbots, LLMs, higher education, student experience.