Scenario-based training (SBT) exercises are being increasingly used in a variety of military and non-military environments. While frameworks exist to facilitate the exercise design process (e.g., Instructional Systems Development, Systems Approach to Training, Joint Training System, Event-Based Approach for Training), the extent that a resulting exercise design actually incorporates a given framework is rarely assessed. Without this assessment, there is no way to determine whether the exercise facilitates learning. One potential reason for the lack of assessment is that most frameworks describe the phases of an SBT exercise (i.e., planning, preparation, implementation, and analysis) as separate components. By looking at each component separately, a comprehensive assessment of the overall exercise is difficult. Effective assessment requires a framework which views the exercise phases as a system of inter-dependent components.

A systems-oriented approach to training design structures training opportunities by tightly linking critical tasks, learning objectives, exercise design, performance measurement, and feedback. While existing guidelines identify what characteristics should be present within a given component of a training system, few guidelines document the linkages that must be present across the components. Based on a review of team training literature and observations of training exercises, fourteen systems-oriented guidelines for exercise preparation, conduct, and debrief were generated. The guidelines, which are based on instructionally-sound principles for learning, were used to assess the utility of five SBT exercises designed by three organizations.

This paper will describe the use of the guidelines for assessing exercise designs. Specifically, the paper will: (a) provide an overview of SBT, (b) present the guidelines, (c) describe the application of the guidelines to assess five SBT exercises, (d) briefly discuss the findings and lessons learned from the assessments, and (e) outline implications for future research, development, and implementation.