Assessment of complex task performance is crucial to evaluating personnel in critical job functions such as Navy damage control operations aboard ships. Games and simulations can be instrumental in this process, as they can present a broad range of complex scenarios without involving harm to people or property. However, automatic performance assessment of complex tasks is challenging, because it involves the modeling and understanding of how experts think when presented with a series of observed in-game actions.

Our previous research was focused on developing a conceptual framework for assessing complex behaviors in non-linear, 3-D computer-based simulation environments. Building on this research, the focus of this paper is on automatic complex task scoring of decision making ability critical to Navy damage control operations. We are using our existing 3-D simulation of the interior of a naval ship (Koenig et al., 2009) which includes both fire-fighting and flooding damage control scenarios. When assessing performance, human expert scoring can be limiting, as it depends on subjective observations of in-game player's performance which in turn is used to interpret their mastery of key associated cognitive constructs.

We introduce a computational framework that incorporates the automatic performance assessment of complex tasks or action sequences as well as the modeling of real-world, simulated, or cognitive processes by modeling player actions, simulation states and events, conditional simulation state transitions, and cognitive construct dependencies using a dynamic Bayesian network. This novel approach combines a state-space model along with a probabilistic framework of Bayesian statistics, which allows us to draw probabilistic inferences about a player's decision making abilities. Through this process, a comparison of human expert scoring and dynamic Bayesian network scoring is presented.