Effective training relies on skill practice, and research has shown that effectiveness is greatly increased with the integration of training strategies such as performance-based feedback. Typical behavioral performance assessment methods are not sufficient for tasks comprised of perceptual and cognitive components as they cannot measure subtle implicit behavior at the perceptual level (e.g. visual scan) or cognitive states (e.g., disengagement). To enable feedback that addresses these, it is necessary to assess performance at a more granular level than most systems can achieve. Without this, instructors are limited to providing feedback to address observable errors, and cannot diagnose and remediate underlying sources of poor performance.

This paper describes a prototype built to study the effectiveness of integrating advanced behavioral and neurophysiological sensors for diagnosing root causes of performance deficiencies in an aircraft instrument landing trainer. The Advanced Training Evaluation System (ATES) incorporated (1) a performance measurement suite that integrates eye tracking and electroencephalography (EEG) with behavioral metrics to capture both learner cognitive state and perceptual performance; (2) a diagnostic engine that identifies root cause(s) based on behavioral and neurophysiological measures; and (3) After Action Review (AAR) displays that present instructors/learners with performance and state summaries, deficiencies and training intervention recommendations. ATES was integrated with an existing testbed consisting of a 737 flight simulator and a real-time evaluation system based on behavioral measures. It was hypothesized that ATES would allow instructors to more effectively identify the root cause of performance deficiencies, resulting in a more effective AAR debrief. An initial pilot study was conducted to provide a limited evaluation of the system. Results included demonstration that learner visual attention, cognitive interest, and cognitive state can be effectively recorded and assessed to provide a more granular understanding of trainee state, indicating a potential benefit of neurophysiological measures in training. Further studies are planned to examine training benefits of the system.