Over their careers, Soldiers must exercise a variety of attributes and competencies to effectively adapt to the varying contextual demands of emerging operational conditions. It is challenging to determine when Soldiers have been adequately prepared to meet such demands because, as described in Brou et al. (2022), Soldier development is often non-linear and context-dependent. Thus, rather than focusing on whether a Soldier “has” an attribute or competency, the assessment challenge becomes better understanding the likelihood of attribute-related performance under varying conditions (i.e., if in condition X, then …). The purpose of this paper is to present a method to diagnose the degree to which individuals’ attribute-related performance changes across varying contextual demands. We examine how situational judgment test (SJT) items and associated rubrics can be used to reliably assess the adaptive exercising of attributes (e.g., empathetic, patient) in an instructional setting. SJT items were constructed to systematically explore responses across conditions that varied in both problem domain (e.g., calling for fire amid uncertainty, effectively working with a struggling student) and the intensity of contextual stressors (e.g., time pressure, threat level, relationship status). Using the SJT items and rubrics, we reliably measured attribute-related performance across contextual manipulations. Findings from a sample of 345 responses across SJT variants demonstrated that agreement between raters was 97% within one point on a five-point scale. Intraclass coefficients ranged from .74 to .81 across attributes reflecting excellent levels of interrater reliability (Cicchetti, 1994). Collectively, these findings suggest that the SJT items and associated rubrics provide a reliable approach to assessing the likelihood of attribute-related performance under varying conditions. Future work will use this method to explore development across time and context by enabling the assessment of individuals’ patterns of strengths and weaknesses. These assessments will in turn provide insights into the creation of targeted instructional interventions.
Keywords
ASSESSMENT;COMPETENCY BASED TRAINING
Additional Keywords
formative assessment, situational judgment tests, context, competency