As the U.S. military begins to explore using autonomous and artificially intelligent agents, serious consideration must be given to how these agents will interact with humans. Adding AI teammates will require understanding current warfighter behavior and how AI agents can augment their capabilities. The goal is to allow AI agents to perform low level and dangerous tasks so humans can focus on high-level battle management. However, in order to conduct this battle management role successfully operators will require high quality data driven insights that help them make sense of an increasingly complex battlespace.

                This work begins to look at quantifying and measuring behavior in an air combat simulation using machine learning. The goal is to build an understanding of key performance metrics that help drive mission success or failure. From these insights machine learning agents can be created and tuned to properly weight or select the correct behaviors to maximize the warfighters chances of winning. The initial data analysis strategy is based around answering two simple questions: 1) Did blue win? 2) If not, why? The analysis specifically looks at determining the importance of different metrics to the outcome of the scenario. Run level metrics like loss exchange ratio are fed into a Random Forest classifier. This classifier makes a win or loss prediction based on scenario metrics. Then, based on the importance of each feature for making the win/loss decision the relative importance of each metric can be gauged. Initial analysis of the data suggests only a handful of traditional performance metrics play a role determining win/loss for the scenario. The final paper will describe model development and the analysis results. Ultimately, the paper will provide insight for the broader community on how to use ML driven methods to develop battle analysis insights for the warfighter.