We present the design, experimentation and discussion of a tool aimed at assessing and fostering the engagement of the students in face-to-face and distance learning classes and relating the engagement to the development of the lecture. The tool, called the Student Feedback App (SFA), was developed inside the Erasmus+ project We-Collab, addressing challenges that often characterize distant learning and poorly designed lessons, but it could be exploited for other kinds of synchronous events.

The basic idea was to gather voluntary subjective feedback messages from the students during the event based on a well-rounded vocabulary of ‘reactions’ capable of expressing cognitive and emotional states. The idea is relatively simple but a preliminary investigation revealed that no satisfactory realization of it was available, i.e. one that provides not too trivial feedback, such as ‘I'm confused’ only, nor requires too much effort on the part of the teacher in lesson planning, such as preparing ad-hoc questionnaires.

We wanted the lecturer to exploit the student feedback in real-time to assess the level of engagement and identify the main difficulties encountered; for this purpose, the SFA always keeps a concise ‘teacher dashboard’ updated. But we also wanted to allow the lecturer to analyze the student engagement later, and relate it to the lesson material. In the same project, we also experimented with the use of neurophysiological sensors to collect signals more directly related to the cognitive and emotional state of the students; being able to correlate and possibly integrate subjective and objective feedback would have allowed us to validate to some extent both kinds of experiments.

To answer the requirements above, the SFA records every reaction message, together with a precise timestamp, in the database of our learning platform; it can also send the data to a Learning Record Store (LRS) as a stream of xAPI statements. The SFA allows users to switch from the ‘send reaction’ screen (a simple virtual keyboard) to a ‘chat’ screen; the chat stream is shown in the teacher dashboard, too; the chat adds some flexibility for the case that the reaction vocabulary appears too limited or the teacher wants to send some instruction in broadcast mode.

The development of the SFA taught us some technical lessons: we were able to appreciate the advantages of modern web-based technologies, such as ‘web sockets’ for two-way system communication over HTTP and ‘reactive’ javascript frameworks for UI management, freeing us from the peculiarities of native mobile development environments. However, most of the discussion was about ethical and usability issues. To answer the first ones, we accommodate a wide range of privacy levels, even different ones for different users. The advance arrangements between the teacher and the students were simplified by modeling each teaching session as an ‘event’ fully described in a shared ‘calendar’.

It is not easy to report on the experimentation since it was conducted in different modes at different sites (e.g., face-to-face and at distance, with and without the parallel use of neurophysiological devices). Also, some teachers evaluated the usefulness of the feedback in real-time, while others drew interesting observations from the deferred analysis of the data collected.

Keywords: Student Engagement Assessment, Real-Time Feedback, Distance Learning Tools, Learning Record Store (LRS).