In the category of written exams, a commonly used format is multiple-choice questions or true/false questions, also known as mechanized assessments. This format is convenient for teachers because it allows the evaluation of a large number of students and a substantial amount of content consistently and rapidly. One possibility is that the use of multiple-choice and true/false questions evolved based on the teacher’s level of experience, resulting in a significant bank of valid questions capable of precisely assessing student achievements and learning levels. However, this is not often the case. Mechanized assessment is not without difficulties and can have a negative impact on student learning. The most significant issue is that most multiple-choice questions typically do not provide sufficient information about students’ fundamental understanding, unlike open-ended questions. Consequently, this evaluation format may ultimately reward surface learning, which involves simple factual recall.

These reasons prompt us to reflect deeply on the widespread use of multiple-choice questions in Chemistry exams and consider whether they are adequately designed to achieve the goal of promoting meaningful and lasting student learning. This project aims to study and statistically analyze multiple-choice questions in core Chemistry courses taught by professors from various areas. Our goal is to make this evaluation tool more reliable, useful, precise, and standardized in assessing our students.

The statistical study will be conducted using the software called “Jamovi”. Jamovi is an advanced spreadsheet tool that enables complex statistical calculations simply and efficiently. It utilizes R as its underlying infrastructure, providing access to a wide range of classical and advanced statistical tests commonly used in social sciences. Thus, it is presented as an effective tool for the statistical analysis of the results obtained in the answers (correct or not) to multiple choice questions and for the establishment of trends, interrelationships, and detection of errors in a large amount of data.

Among the parameters analyzed for student response analysis, we include the difficulty index, discrimination index, distractor frequency, reliability assessed through Cronbach’s and McDonald’s coefficients to understand item relationships, and Pearson correlation index (Heatmap). Contrary to Jamovi’s original development focus on social sciences, our group has successfully employed this tool to draw intriguing conclusions in the educational context, specifically in Chemistry instruction at the higher education level.

Acknowledgement:
Funded by the Universidad Rey Juan Carlos under the 2023/24 Educational Innovation Projects call and ESCET Innovation Projects 22/23.

Keywords: Higher education, multiple-choice questions, statistical analysis, Jamovi.