Intelligent Tutoring Systems (ITS) hold the potential to unlock a new era of adaptive, learner-centric training, but much of the current research focuses on learner-tutor interactions. Alternatively, this paper describes ongoing research to demonstrate an automated data analysis capability that supports training effectiveness evaluation from the tutor authoring perspective. The research team investigated: how to extend the Army Research Laboratory's Generalized Intelligent Framework for Tutoring (GIFT) to provide this automated analysis capability; what data collection mechanisms could be utilized to support the analysis; and how to present the outputs to support decision-making about instructional strategy in an ITS. The resulting proof-of-concept operationalized this framework to support the rapid, high-level, visually intuitive analysis of effectiveness at a user-selected level of granularity and to then offer a mechanism to delve deeper and explore individual factors to ultimately identify areas for improvement.
The research team simulated experimental learner demographic and performance data to verify and evaluate the proposed methods since there were no examples of completed GIFT courses at the time of this research. A statistical engine was used to identify factors that contribute significantly to training effectiveness and to support investigation of the research questions. An Army marksmanship course was chosen as a use case since it relies on multiple training delivery techniques and includes factors external to the GIFT environment, such as experience with first-person shooters or prior experience. The research utilized standard GIFT data sources alongside Experience Application Programming Interface (xAPI) formatted data to combine disparate sources and support integration of multiple perspectives. While the extent to which the experimental data represents real-world use cases will have to wait for the ongoing GIFT courses to be completed, the experimental data provided a foundation for initial technical evaluation of this proof-of-concept system based on existing data sources.