Data driven approaches to measurement can provide unique insights into team states and processes critical to effective training that are often difficult to achieve through traditional observational or survey-based methodologies. The Rational Approach to Developing Systems Based Measures (RADSM; Orvis, Duchon, & DeCostanza, 2013) is one process that has been successfully applied to identify and extract data features from sources such as communications (e.g., email, chat, face-to-face), systems interactions, and physiological sensors. The extracted features are often diverse in how they are extracted and represented (e.g., network-based metrics, content analysis features, temporal dynamics) and can number in the hundreds to thousands of items. This presents a daunting challenge in integrating these heterogeneous items and making sense of their relationships to the overall team state or process measurement and the impact on training. In this paper, we describe an approach utilizing machine learning (ML) models to integrate and better understand heterogeneous measurement items. ML models can integrate features derived from a variety of data through multiple algorithmic means into a single score or set of scores. When objective performance outcomes are available (e.g., from observations or systems data), supervised ML models (e.g., Classification Trees) are utilized to learn the relationships between the data features and the outcomes. In addition, we demonstrate how ML methods can highlight the relative impact of individual features on the overall team states or processes. This data fusion research provides insight into advanced approaches for constructing training metrics from multiple sources of systems-based data. These analytic methodologies enable trainers to gain a better understanding of team states and processes through utilization of data sources and measurement items that have proven difficult to make sense of previously.
Machine Learning Approach to Integrating Heterogeneous Team State and Process Measurement Items
3 Views