The increased adoption of mixed-reality simulation-based training environments along with the use of multimodal sensing devices has led to a proliferation of participant interaction and behavior data that can be collected while scenarios are executed. Analysis of this rich data using advanced AI and machine learning algorithms makes it possible to create robust multi-dimensional models of individual and team performance that can include psychomotor, cognitive, metacognitive, and affective factors. Moreover, through a multi-dimensional approach one can compute performance metrics within single training instances, as well as across a full course of training scenarios. This provides valuable feedback to trainees and their instructors on their skill levels, proficiency, and progression over time.  However, developing objective data-driven performance metrics come with a set of challenges that include data collection and aggregation, pre-processing and alignment, data fusion, and the use of advanced multimodal learning analytics (MMLA) algorithms to compute individual and team performance.

In this work, we present a case study of teams of three to four soldiers training on dismounted battle drills in a mixed reality-training environment. We develop our multimodal computational architecture and demonstrate the use of advanced machine learning based MMLA algorithms to analyze the collected training data that spans video, speech, and simulation logs. We model the progression of the teams of soldiers through the course of multiple training scenarios and show their progression over time on both operationalized domain-specific performance metrics, as well as higher-level cognitive and metacognitive processes., To show how these automated analysis methods can be presented to trainees and instructors to supplement traditional expert feedback, we present the MMLA analysis through the lens of distributed cognition. In addition, we show how results from our analysis methods can be used to provide suggestions for both future training needs and potential improvements to the training environment.

Keywords
AI,BEHAVIOR MODELING,COGNITIVE,HUMAN FACTORS,HUMAN PERFORMANCE,LEARNER ANALYTICS,MACHINE LEARNING,PERSONALIZED TRAINING,TEAM TRAINING

Additional Keywords
Multimodal, Distributed Cognition