Evaluation of decisions made in complex, multi-entity scenarios, such as those typical of LVC training, is extremely difficult. In simple, low entity-count scenarios, it can be possible to rely on a library of goals and metrics defined for the evaluation of the performance of and decisions by a single operator, where there is one best option to select or correct decision to be made. As mission complexity and entity count increase, however, these assumptions fail. A critical need exists for informed evaluation of decisions made in complex situations in which it is not possible to unambiguously prioritize or achieve a set of entity-level goals and expectations.

An alternative available for construction of complex metrics is the specification of behavior envelopes (Jones et al., 2015), a tool that can be used to define the window of variation around which a set of goals or behaviors can be interpreted to have been met or to have occurred (Wray et al, 2021; Jones et al., 2015).  A behavior envelope consists of three primary components: A situational context defining observable features of a situation, as well as unobservable features describing the internal state of the target being evaluated; expectation constraints that the behavior should meet in situations where the behavior context applies; and finally, a non-binary scoring/fit function that evaluates how well the behavior being observed conforms to the expectation constraints.

This paper explores the application of behavior envelopes to the use case of multi-entity performance and decision evaluation in complex scenarios, through analysis of a corpus of non-deterministic, high iteration count synthetic, labeled SAF defensive counter-air (DCA) mission iterations. Specification of the context, expectations, and scoring/fit of the observed complex mission envelopes are used as the basis for specification and quantification of the achievement of multi-entity mission criteria.

Keywords
BIG DATA;LVC;M&S;METRICS

Additional Keywords