Multinational training exercises are an important part of developing joint preparedness. Increasingly, participants in multinational training exercises are provided with e-Learning training prior to and during the exercise. Understanding whether materials are well-aligned to the needs of participants during the training exercise helps us to understand whether the materials will ultimately be beneficial in the settings the training exercises are preparing forces for: planning and conduct of a combined and joint Crisis Response Operation (CRO), using Standing Operating Procedures (SOP), within a NATO-led operation.
However, methods for establishing the alignment of e-Learning to a training exercise have thus far largely been based on the judgment of domain experts. Domain experts do not always perfectly understand where their training materials are having their greatest impacts on performance, particularly for highly complex domains. We propose a new approach, based directly on trainee data. Specifically, we analyze the alignment between e-Learning and training by analyzing data on how specific trainees use the e-Learning, and the performance of the trainees’ units on specific objectives.
We frame this measure of alignment as a Q-Matrix, a representation of the links between two sets of constructs. Q-Matrices are commonly used in cognitive diagnostic testing and intelligent tutoring systems to represent the links between latent student skills and specific performance items. In this case, we use Q-Matrices to represent the connections between the use of specific e-Learning modules and training unit performance on specific objectives. We propose a concrete heuristic for this mapping procedure based on time-on-task and performance ratings. We apply this heuristic to study the applicability of a set of three e-Learning training modules to trainee performance at conducting current operations and both mid-term and long-range planning, analyzing this question within the context of data from a multinational training exercise CJSE19 with participation from 12 countries.