Cyber threats have become ubiquitous as criminals extend their reach and cyber becomes a front in conflicts between different peoples and a major source of revenue for criminal organizations. Personnel responsible for cyber defense are becoming increasing critical. However, there is a shortfall between the number of individuals training to enter cyber security and the projected demand for these skills. Consequently, methods and technologies are needed to enhance and accelerate the training of cyber security personnel. Previous research has demonstrated the benefits of automated performance assessments as a means to target training to the specific needs of individual students. The current paper describes an extension of these capabilities to cyber security training exercises. In these exercises, students are placed in teams and must work together, using appropriate software tools and online resources, to conduct forensic analysis for cyber crimes. Individual and team performance is assessed on the basis of successfully solving individual challenges and applying information from individual challenges to correctly ascertain an overall picture of the who, what and why of the crimes. The current paper describes a framework for conducting cyber security training exercises with an emphasis on instrumentation to enable automated performance assessment. Instrumentation captures students’ computer-based transactions in a log that is time-synched with the game-server used to deliver challenges and register student responses. Analyses were conducted to better understand the factors that distinguish more or less effective student performance and techniques developed to automatically parse logs of student activities into meaningful blocks of task-oriented activity. These capabilities are a prerequisite for the development of real-time automated assessment of student performance within the context of cyber security exercises.