Evaluation of non-technical competencies is typically done on a more subjective level, relying heavily on instructors to make acute observations during a training scenario. A behavior detection prototype was created by the authors to enable more consistent evaluation of non-technical competencies, addressing the challenge of standardizing training and evaluation in this area. This paper describes how the prototype analyzes training data to detect instances where a student demonstrates behaviors that provide evidence of defined competencies. The prototype was built to focus on discrete events that can occur in a training simulation and assess a student’s response to each of the events that transpire. By doing this, the system can detect and assess both planned and unplanned events during a training exercise. A further benefit of the event-based design is that context can be considered when evaluating student responses as evidence of competency, or lack thereof, providing the ability to evaluate soft skills in a more objective way. This allows the system to collect evidence of competencies such as decision making, situation awareness, and workload management, creating objective data that can be used for instructional purposes.

To evaluate the accuracy and functionality of the data generated by the prototype, the authors integrated their system with a desktop trainer simulating a Control Display Unit (CDU) during a non-normal flight scenario. A study was run where instructor participants were shown recordings of student participants to assess their competency performance in the scenario, independently of the prototype. The instructors then analyzed their own data side by side with the prototype output to evaluate and quantify their level of agreement. The results demonstrate the prototype effectively identifies evidence of competency, aligning with traditional instructor assessments, thus improving the data collected during training to enhance the training experience.

Keywords
ASSESSMENT;AUTOMATION;COMPETENCY BASED TRAINING;HUMAN PERFORMANCE;PROFICIENCY;PROTOTYPING

Additional Keywords
Behavior Detection