Abstract
Military training in the U.S. increasingly utilizes immersive simulation technologies to enhance Distributed Mission Operations (DMO) and the Joint Simulation Environment (JSE). As extended reality technologies (XR)—including augmented reality (AR), mixed reality (MR), and virtual reality (VR)—evolve, their hardware and software capabilities are continuously explored. These technologies offer a variety of input modalities for interacting with and selecting objects in virtual environments, such as eye, head, and hand tracking, as well as controllers and other handheld devices. Despite the range of interaction methods available, research on the optimal input modality for training tasks in dynamic environments is insufficient.
The Air Force Research Lab’s Gaming Research Integration for Learning Laboratory in Dayton, OH, has developed a testbed to evaluate the efficiency and effectiveness of seven input modalities to address this gap. The virtual environment simulates flight deck panels to serve as a procedural task trainer and to collect data on timing, error rate and type, usability, and user preference. A previous study with five input modalities found that dwell selection methods were the least preferred, with users showing a strong preference for the ability to click a button on a controller to confirm their selection (Fussell et al., 2024). Building on this, the next phase of research incorporates two eye-tracking modalities and additional tasks using a second virtual flight panel. Seven input modalities will be compared using a mixed-methods, within-subjects experimental design. The research aims to understand how different input modalities affect a user's ability to complete procedural tasks in XR and how they influence user experience. Data collection is projected to continue through May 2025. Preliminary evidence suggests that eye tracking presents challenges that may render it ineffective in complex training environments. This research will offer guidance on enhancing the design of XR devices for dynamic training scenarios.
Cleared Case Number: AFRL-2025-0864