The task of consolidating and quantifying complex decision-making across a multi-attribute space is a challenge (Achkoski et al., 2017; Klein, 2008). This work describes a technique for quantifying human expert multi-faceted decision-making in the context of medical triage into a limited number of dimensions. The analytic and visualization technique described here also serves as a basis for both machine-readable and human-interpretable comparisons between human and algorithmic decision-making outcomes.

Tactical Combat Casualty Care (TC3) and medical triage scenarios involve complex situations comprised of factors including time pressure, high stakes, and uncertainty (Joint Trauma System, 2020; Klein, Orasanu, Calderwood, & Zsambok, 1993). Medical triage scenarios are an exercise in satisficing, since by its very definition, “triage” refers to the prioritization of limited tasks and resources. In complex triage scenarios, experts will disagree on what set of decisions is truly optimal (Achkoski et al., 2017). This effort used a dataset of 28 scenario responses from practitioners to capture complex triage decisions and translate the factors underlying them to quantifiable, comparable metrics. This modeling is a critical prerequisite for a measurement and visualization tool that could lend itself to comparisons of decisions made by human experts with decisions made by AI systems.

The goal of this effort was design of a simple system for representing and capturing treatment and resource allocation decisions by triage managers and translating those decision data to a limited number of dimensional attribute scores describing the decision-makers. This reduction made it possible to consolidate this representation of the data representing a group of decision-makers in a single multi-dimensional index. Machine learning techniques were then used to evaluate the predictive relationships among scenario characteristics and decision-maker attributes, also discussed (Zheng, Aragam, Ravikumar, & Xing, 2018).