Main Subcommittee Category: Emerging Concepts and Innovative Technologies

How would you label your submission?: Industry

Abstract:

Studies have shown that digital training and interactive multimedia instruction (IMI) products improve learner outcomes through increased interactivity and immersion. Instructor-led IMI is a specific type of training where instructors guide students through a digital course, providing invaluable feedback and guidance. However, an instructor’s time is finite, and teachable moments only scale linearly. Qualified instructors are ‘aging out’ of training programs, which could limit the number of students allowed to take specific training. What if we could use AI to augment an instructor? Or even create a virtual surrogate instructor to allow real instructors to teach and reach more students?

This paper describes a study funded by US Army PEO Aviation that leverages large language models (LLMs), text-to-speech (TTS), and other AI tools to create a high-fidelity intelligent agent (IA). The agent can interact with the user and the virtual environment to emulate a subject matter expert for instructor-led training. Users can interact with the agent freely through voice or text during self-paced learning to fill in any gaps in the learning content. The paper outlines a prototype application's capabilities and technical limitations, what tools and technologies were considered, and the tests and results to select those technologies integrated into the final prototype. The prototype architecture, which includes the interactive agent's 3D avatar, voice, perception and context of the virtual environment, and course material is also discussed as well as a custom-trained LLM and knowledge base that serves as the "brain" of the agent. Finally, the paper covers the agent's current hardware requirements and usage limits, lessons learned during development, and future steps for continued development of the prototype.

Keywords: AI;AVATAR;DIGITAL-GAME-BASED-LEARNING;EMERGING TECHNOLOGIES;IMMERSIVE;INTERACTIVE;TRAINING

Additional Keywords not in the list above: LLM, TTS, IMI

Menu

Mr. Nelson Hurst