While a certain level of surprise is required for nearly any type of learning, it is a challenge to provide for surprises in an effective way. Simulation enables the training designer with powerful options to provide for surprising experiences, either to engage students, to stimulate thinking, or to learn to deal with them. Dealing with emergencies or replanning for example are explicit training objectives in many simulator sessions – although the students often already expect the surprising events. Alternatively, surprises in simulation sessions can be instrumental to achieve a context in which other training objectives can be achieved, such as leadership, decision making, and coordination. This study explores the nature of surprises and provides suggestions for designing surprises in training and subsequently for assessing its effectiveness. The framework for designing and evaluating surprises relates to the capabilities that cause the surprise (this may be cue based, narrative based or personal-based) as well as a human (surprise) information processing model. Assessing the effects of surprises is relevant during the design of the training scenario to tailor the effects to the target audience, and may also have the potential to guide the instructor during the training to inject weaker or stronger events. The use of electro encephalogram (EEG) is a promising technique for assessing mental state levels of relaxation, attention, or agitation/confusion. In this study EEG is applied to analyze brainwave patterns and investigate the potential for assessing the effects of a variety of surprise types in a VBS training scenario. Preliminary results indicate that EEG is sufficiently sensitive to measure mental state effects of surprising events. More study is required to determine the validity of the measurements and whether it can be used as the single technique or that a toolkit using a variety of techniques are needed.