Over the past few decades, simulator use has increased greatly, due in part to its cost-efficiency and ability to provide training experiences that would be impractical or unsafe to conduct otherwise (e.g., emergency procedures). This increase in simulator use has coincided with an explosion in “big data,” more specifically, human performance data that are collected from a large number of learners (n), measured variables (v), and measurements per unit time (t) (Adjerid & Kelley, 2018). However, as the resulting corpus of human performance data expands, it becomes increasingly more difficult to mine for trends. Resulting in a large pool of recorded data that is not immediately useable without extensive workarounds, manpower, or software algorithms. For example, consider the use case of simulated Air Force engagements. At any single Air Force training facility, there could be simulator records from hundreds of training scenarios per year with a variety of different characteristics (e.g., offensive counter-air maneuvers, defensive counter-air maneuvers, two-ships, four-ships, etc.). However, certain limitations of the data, such as unstandardized start and stop times of the engagements, hinder the ability to easily mine the data for historical norms, proficiency, or other human performance outcomes. As a result, the ability to interpret or draw conclusions from the data is much more limited, despite the robust pool of data. In this paper, we present the findings from a multi-year research and development effort that focuses on extracting meaningful human performance metrics from a “data lake” of roughly 3,500 data recordings that represent 10,000 training scenarios over the course of more than 15 years. We present best practices and lessons learned for parsing the data lake contents so that readers can better understand the implications of data limitations and how to address them in their own work.

Menu

Dr. Luke Waggenspack