ABSTRACT VIEW
Abstract NUM 395

AN EVALUATION OF MULTIPLE-CHOICE QUESTIONNAIRES THROUGH THE LENS OF ERROR
J.A. Domínguez Vázquez1, B.M. García Otero2
1 Universidad de Valladolid (SPAIN)
2 Universidad Europea Miguel de Cervantes (SPAIN)
The study focuses on scoring methods for short multiple-choice tests, which are typically made up of up to ten questions and are often incorporated into mixed exams. A significant problem identified is that traditional scoring methods, which heavily penalise incorrect answers, discourage students from answering doubtful questions. This reluctance stems from a fear of severe score reductions, even among well-prepared students.

The fundamental premise of the proposed new methods is that human error is inevitable and that initial mistakes should not be punished so severely, or even 'forgiven'. Conversely, as errors accumulate, penalties should become more severe to effectively filter out students who rely on chance for success. The overarching goal is to incentivise students to answer more questions, thereby enabling them to demonstrate their knowledge more effectively, while still being able to differentiate between those who are prepared and those who are not.

The sources analyse and propose several scoring methods:
- Classical Linear Exact Method: This standard approach penalises incorrect answers, meaning a student who guesses at random receives a zero score.
- Linear Approximate Method without Penalisation for 'p' Initial Errors: This method reduces the number of errors counted for penalisation, while still using the classical method to calculate the penalty weight.
- Linear Exact Method without Penalisation for 'p' Initial Errors: Unlike the approximate method, this approach recalculates the correct penalty value w to ensure that a score of zero is obtained through random guessing, even with p non-penalised errors.
- Progressive Methods: These methods allow non-constant, progressive penalties for each additional error, starting with zero penalty for the first error and increasing the penalty for subsequent errors to ensure that a score of zero is obtained through random guessing, as classical linear method.
- Linear approximations to progressive methods: These methods are simplified linear models from before Progressive Methods, to make the scoring system practical for students, to understand and apply it during exams. These linear approximations simplify the penalties to only three values: zero penalty for "p" first errors, a light penalty "w1" for next errors, and a strong penalty "w2" for the rest of errors. We provide two different ways of calculating the penalties w1 and w2 to ensure a score of zero under the same conditions as the classical linear method.

These last methods are designed to encourage responses and differentiate between students who have studied and those who have not. Methods that do not penalise the first mistake allow a 'brilliant' student to achieve an 'A' (equivalent to 9.0 or 10.0) even if they make one mistake out of ten questions. We will show a comparative table among all methods. Importantly, all methods ensure that students who guess randomly still receive a zero score.

This last method has been used for last 3 course years. Survey feedback from students yielded resoundingly positive results:
- 100% of respondents preferred the new scoring method.
- 91.7% felt that the new method encouraged them to answer questions they were unsure about.
- No students felt disadvantaged or benefited unfairly compared to their peers.
- 50% thought that their obtained score was a better reflection of their actual knowledge.
- 75% of students found it easy to calculate their potential score during the exam.

Keywords: Multiple-choice questionnaires, penalties, based error learning, new marking approaches.

Event: ICERI2025
Session: Assessment and Evaluation
Session time: Monday, 10th of November from 12:30 to 13:45
Session type: ORAL