Abstract
The Department of Defense (DoD) and the United States Air Force rely on computer-generated forces (CGFs) to reduce costs and resources in fighter pilot training. Software such as the Next Generation Threat System (NGTS) enables complex tactical engagement simulations using a red vs. blue wargaming paradigm (e.g., 2-v-2 dogfights or coastal attack and defense engagements). However, the number of large-scale scenarios necessary for advanced pilot training is limited. Key obstacles in preparing scenarios for wargaming in NGTS include the time and expertise required to devise and script complex scenarios, as well as the challenge of transcribing this technical knowledge to the machine via graphical user interfaces (GUIs) and software engineer support.
In this work, we introduce a novel multimodal generation framework that leverages generative AI (GenAI) and large language models (LLMs) augmented with structured data representations to revolutionize scenario creation within the NGTS ecosystem. Our approach integrates natural language processing with image-based descriptive schemas and automated XML code generation, allowing for the dynamic creation of complex red teaming scenarios from simple text prompts. By fusing text, visual schematics, and code, our system not only streamlines the development of expansive training scenarios but also empowers non-expert users to rapidly generate and iterate on simulation content.
This innovative method enhances pilot training efficiency and effectiveness by significantly broadening the scenario landscape available for CGF-based exercises. Moreover, our multimodal generation strategy paves the way for future advancements in training and simulation, enabling more adaptive, scalable, and interactive wargaming environments that align with the evolving demands of modern defense operations.