Currently, military training needs assessments and curriculum reviews rely on time-intensive manual analysis of surveys and interviews, introducing lag, inconsistency, and lack of scalability. The proposed integrated AI analytics aims to fundamentally transform this process. In this paper we discuss novel integrated AI-powered analytics that automatically analyze survey responses to: (1) identify training gaps and effectiveness issues; (2) understand differences across career field needs; and (3) assist in reworking surveys and evaluation interviews for more actionable insights.
Specific AI techniques include topic modeling to automatically identify themes related to gaps and effectiveness; syntactic parsing and semantic embeddings clustering to extract detailed insights; connotation analysis to assess student opinions and experiences at scale; and large language models to generate redesign recommendations. Together these methods allowed to surface pain points using the example use case like outdated equipment limiting realism and insufficient hands-on time. Connotation analysis revealed negative perspectives around trainer aids and aircraft matching scenarios, while additional comments and training preparation skew positive.
The integrated AI analytics fundamentally advance the state of the art, moving from periodic manual surveys toward dynamic AI-assisted assessment and improvements. This will provide agile, optimized expertise development to sustain readiness. Longer-term, the proposed AI analytics will boost consistency, efficiency, and personalization of training via continuous automated analysis of human feedback. Specifically, AI-driven visual platform will fuse data to reason about training effectiveness; (2) speech recognition will enable naturalistic data collection, and (3) LLM driven agent-based simulations will complement surveys. The paper and presentation will cover the methods used, the benefits and challenges of these approaches and feedback for government and industry on implementing similar processes within their own training evaluation programs.