In virtual military training, tracking and evaluating trainee behavior throughout a simulation exercise help address the specific training needs, improve the realism of simulations, and customize the training experience. While it is straightforward to parse the event log of a simulation to identify atomic behaviors such as unit movements or attacks, it remains difficult to fuse these events into higher-level actions that better characterize trainees’ intentions and tactics. For example, if each unit is controlled by an individual trainee, how should the movement information from all units be aggregated to determine what formation the group is moving in? Similarly, how can all of the information from nearby terrain environments be combined with kinetic actions to determine whether the trainees are executing an ambush attack, or is simply engaging the enemy group? While an experienced human observer-controller can quickly assess the battle map to provide an appropriate interpretation for such events, it remains a challenging task for computers to automatically detect such high-level behaviors when performed by human trainees. 

In this work, we proposed a machine-learning (ML) framework for recognizing tactical events in virtual training environments. In our approach, unit movements, surrounding environments, and other atomic events are represented as a 2D image, allowing us to solve the action detection problem as image classification and video temporal segmentation tasks. In order to bootstrap ML models for these tasks, we utilize synthetic training data to procedurally generate a large amount of annotated data. We demonstrate the effectiveness of this framework in the context of a virtual military training prototype, detecting troop formations and other tactical events such as ambush and patrolling.