Measuring mission-critical, higher-order cognitive constructs automatically is a priority within the DoD to achieve third-offset goals, accelerate training of complex skills, and support multi-domain warfare. Situation Awareness (SA) is one such construct, but its measurement is burdensome, relies on post-hoc analyses, and provides little immediate value in training or operational environments. Wilson and colleagues (2020) demonstrated that targeted application of Machine Learning (ML) on biometric data can yield real-time, accurate performance classifiers evaluating pilot eye scan techniques and mental workload. The current study established the relationship between the three levels of SA (i.e., perception, comprehension, and projection) and corresponding proxy ML classifiers, such as eye scan accuracy and mental workload, to lay the foundation for a real-time SA index that is diagnostic of performance. Forty participants, including pilots with varying levels of expertise and an ab initio control group, participated in a meticulously controlled, within-subjects experiment that involved flying an intercept mission using an F/A-18 mixed-reality trainer. Situation awareness was measured using the Situation Awareness Global Assessment Tool (SAGAT), and we used the NASA Task Load Index (NASA-TLX) to gauge mental workload – both subjective metrics are industry gold standards. Eye scan accuracy and mission performance were graded by experienced instructor pilots. Statistical analyses describe the relationship between level of expertise, eye scan accuracy, mental workload, and performance across SA levels. This experiment is unique as it presents results that provide a foundation for a real time, objective, and accurate SA index. Study findings highlight expected benefits for both training complex skills and high-stakes dynamic operational environments.
Human Performance Analysis and Engineering
Towards the Development of an Automated, Real-Time, Objective Measure of Situation Awareness for Pilots
33 views