“Big data” sources are enabling training system designers to leverage analytics to create individualized learning approaches better aligned to student needs.  As our ability to collect and understand performance data improves, the need to reexamine legacy training evaluation strategies and systems has grown as well.  Analytics has improved our understanding of how students achieve competency and provided insight into system-wide changes that are required to sustain or improve individual and program achievement.  The challenge for instructional designers is how to “design-in” feedback mechanisms for decision-making processes.  Feedback data can determine not only the effectiveness of student accomplishment, but also the efficiency of all enterprise processes aligned with evolving expectations of the gaining unit. 

Instructional designers need to know how to access performance data to validate current training system design and shape future training system constructs.  We know technology generates data, but what data should feed advanced AI algorithms to support training and resource decisions at the local and enterprise levels?  How can emerging training and learning technologies integrate with analytic strategies to identify Training Return on Investment (T-ROI) and future value propositions?  How can training managers know they are meeting both student and end user requirements? 

This paper explores how to develop and deploy evaluation strategies that provide meaningful input to training system design in complex programs. It provides approaches to help instructional designers, training managers, and instructors leverage feedback to shape long term learning behaviors, refine syllabus requirements, and improve overall system performance aligned with end user expectations. The paper recommends effective ways to examine data on how students achieve their learning objectives, and describes a framework to help training managers know what should be measured and how much, from whom and how often - questions fundamental for training system optimization today and into tomorrow.