Simulated workplace environments provide the context in which individuals may demonstrate competency mastery in real-world scenarios. Simulation based assessments may incorporate roleplay, multi-step project or problem sets, collaborative or creative problem-solving exercises, and other strategies that have notable advantages over traditional assessments, particularly for assessing mastery of higher-order competencies and far transfer. Virtual simulation assessment environments offer benefits such as cost savings, lower barriers to access, and increased safety as compared to assessments conducted in physical environments. The study described here employed a virtual simulation environment to assess the mastery of core cybersecurity competencies.
A virtual simulation environment was developed in collaboration with subject matter experts, simulation developers, and pedagogical specialists. We used a collaborative design approach to ensure alignment among features of the simulation, guidance provided to test takers, and core competencies. Network topography was designed to faithfully represent a real corporate environment. Specifically, the simulated network environment hosted multiple technologies that could be involved in cybersecurity events; could be replicated with minimal hardware requirements; was secure enough to enable appropriate responses by test takers; and was vulnerable enough for bad actors to cause malicious acts. This configuration enabled us to assess cybersecurity specialists' technical and non-technical skills, and offered the opportunity to probe mastery of multiple skill types simultaneously and in diverse contexts of application.
Competency measurement requires (1) isolating specific component skills and (2) valid, reliable assessment of mastery for each component. Requiring test takers to exercise diverse skills simultaneously, in concert and in tension with one another, adds meaningful authenticity to assessment tasks in simulated environments. This integration, however, actively inhibits the isolated assessment of specific skills. We found that effective assessment required carefully distinguishing among component skills, and that multiple lines of assessment evidence were required to independently assess specific higher-order skills. This paper details our process for identifying an distinguishing component skills, ensuring alignment between assessment activities in a virtual simulation and demonstrations of skills mastery, and designing multiple lines of assessment evidence. This approach is content-agnostic and will be particularly useful for competency-based assessment in fields which require both application of complex procedural skills and professional certification, such as cybersecurity or nursing.

Keywords: Measurement, simulation, competency-based education.