In this study, we leveraged traditional psychometric approaches and large language models to understand the coding behaviors of entry-level college students. Detailed programming process data were collected using keystroke logs from approximately 180 students working on 21 Python coding tasks designed by content experts. These tasks measured foundational coding skills, including string, integer, and list operations, as well as looping, branching, and nested branching. We extracted around 20 behavioral and coding process features from the keystroke logs, such as time spent debugging, time spent planning code, total number of keystrokes, number of attempts to check code accuracy against test cases, and time spent on each attempt. The quality of the code at any point during the coding process, including the final submission, was recorded based on the number of passed test cases. Given that different test cases target different subskills, detailed analysis of test case pass/fail results provided diagnostic information on students' coding proficiency. The final code submissions were further processed using a BERT large language model, from which embeddings were used to create 10 indicators of code quality. In this paper, we present results on predicting code quality from behavioral and process variables, as well as features extracted from the large language model. We also present results on clustering coding behaviors (e.g., persistent coder, proficient and efficient coder, inefficient coder) and their association with task difficulty, which have practical implications for both learners and educators.

Keywords: Coding, behavioral process, LLM, psychometrics.