Measuring mission-critical trust between human operators and collaborative synthetic teammates is a priority within the DoD to achieve third-offset goals, accelerate automation design and training for hybrid human-machine teams, and support next generation multi-domain warfare. Achieving proper trust calibration has long been a primary mechanism by which Human-Machine Team (HMT) performance can be maximized by avoiding system distrust and over-trust. However, proper trust calibration hinges on the implementation of effective trust calibration techniques based on real-time trust assessment. The current study establishes the relationship between HMT trust, workload, and performance in a Search and Rescue (SAR) paradigm where human operators supervise intelligent Unmanned Air Vehicle (UAV) assets to achieve mission success in a constructive synthetic environment. A novel trust measure was developed and piloted in this experiment to precisely measure subjective trust variations across time and in conjunction with target task elements. Thirty participants, including UAV operators and novices, participated in a rigorously controlled, within-subjects experiment that involved supervisory control of intelligent UAVs promoting collaborative decision-making via system recommendations across four SAR missions. Workload was manipulated by alternating the number of UAVs to supervise across each trial. Trust was assessed via our novel measure in addition to established metrics. HMT mission outcomes were measured via Measures of Performance (MOPs) and Measures of Effectiveness (MOEs). Objective biometric-based metrics were also used to measure operator workload using the Cognitive Workload Classifier (CWC), and Index of Cognitive Activity (ICA). Statistical analyses describe the relationship between trust, workload, and performance, and the impact of automated recommendation accuracy on HMT trust and mission outcomes. This experiment is unique as it provides a foundation for a real-time self-report measure of trust that can be directly compared to real-time physiological measures. Study findings further discuss intervention techniques to maintain proper trust calibration in operational environments.
Keywords
AGENT-BASED SIMULATION, AI, COGNITIVE, HUMAN FACTORS, HUMAN PERFORMANCE, M&S, SYNTHETIC ENVIRONMENT, UAV
Additional Keywords
Human Machine Teaming, Trust