Abstract:

Military personnel face increasingly complex tasks in their line of work, often requiring significant, costly training and time for knowledge transfer from experts to trainees. However, by utilizing modern technology such as mixed reality and artificial intelligence, the overhead for these training operations can be significantly reduced while maintaining the efficacy of the training program. In this paper, we detail a task guidance system that trained inexperienced pilots in routine and emergency operations within a UH-60 helicopter. We translated expert knowledge and procedural documents into artificial intelligence, which used mixed reality to guide novices via look and operational cues. We reduced the costs of a full UH-60 cockpit simulation by mixing a virtual reality setting with the real-world haptic feedback of touchscreens that represented the buttons the pilot would be pressing. In our experiment, users would take on the role of a co-pilot walking through a pre-flight checklist before taking off. They would then encounter inclement weather, and a sudden altitude drop, causing them to transition into an emergency procedure scenario. The artificial intelligence monitored their actions with its expert knowledge database and guided users to the next step via operational and look cues. The operational cues tested were Pick Arrow, Ghost Hand, and Target Outline. The look cues tested were Look Arrow, Attention Funnel, and EyeSee360. We ran multiple users with varied flight experiences through the system, and they showed expert capabilities in walking through both procedures. Our experiment shows the capability for mixed reality and artificial intelligence to provide effective training while reducing cost and time requirements.

Keywords: AI;AUGMENTED AND VIRTUAL REALITY (AR/VR);EMERGING TECHNOLOGIES;HUMAN FACTORS;MIXED REALITY

Ryan McMahon

Affiliation

Refine Results

Using Mixed Reality and Artificial Intelligence for Complex Task Guidance in a UH-60 Environment