Infantry training, ranging from basic training to joint exercises can become more effective through automated performance evaluations and performance based real-time training adaptation and control. In this paper, we will introduce an automated behavior analysis and performance evaluation computational framework that we developed based on United States Marine Corps (USMC) Tactics, Techniques and Procedures (TTP) for a wide range of training objectives. We will also discuss the experimental results of instrumented training systems using this framework for data collection, performance evaluation and multimedia after-action-review (AAR).

We model warfighters' behavior (individually and in teams) as states, and the causes of state transition as trigger-events. Each state has a set of performance metrics. Both states and trigger events have detectors. TTP are represented as hierarchical Finite State Machines (FSM) with associated performance metrics. Training exercises are constructed as a set of preplanned trigger events to exercise all states defined in the training objectives. Behavior analyses use sensor data observations to estimate states, and performance evaluations compute performance metrics given the estimated states of the trainees. We also develop a novel Histograms of Oriented Occurrence (HO2) algorithm for individual and team action recognition.

We instrumented training systems for both outdoor urban operations and indoor close-quarter battle. Video cameras capture the training exercises and automatically analyze behavior and evaluate performance. Each warfighter's location, weapon pose, and head orientation is tracked using a combination of video-based people tracking, GPS, RFID (Radio Frequency ID), video analysis and inertia navigation sensors. Gunshots are captured through trigger sensors. Our system estimates behaviors and corresponding performance metrics in real-time, and ingests those data into a database. Events and videos are overlaid on a 3D-model of the training site to enhance AAR and situational awareness, and furthermore, AAR allows searching and browsing of training events and the computation of statistics.