This paper documents the design, construction, verification, and validation of an automated and semi-automated hands-on summative performance assessment system intended to evaluate battlefield first responder skills (e.g., applying a tourniquet). The goal of this effort is to provide valid and reliable Army-standardized and unit-flexible Combat Lifesaver and Medic skills assessments designed to maximize automation while simultaneously minimizing administration demands and costs. The assessment system will be administered at U.S. Army Medical Stimulation Training Centers (MSTCs). MSTCs are standardized medical training platforms which enable classroom and hands-on simulation-based training of medical first responder skills. MSTC technologies provide the ability to control stimuli impacting the trainee as well as capture trainee performance data. The performance assessment system under development is part of the Medical Training Evaluation and Review System (MeTER) MSTC subsystem and will use testing rooms, programmable patient simulators and scenarios, and evaluator data collection systems (e.g., on desktop or tablet PCs) to provide standardized scenario-based (semi-)automated performance assessments. XML-based Simulations for Integrated Learning Environments (SIMILEâ„¢) software will integrate performance data automatically collected from simulation technologies, evaluator task performance scoring, standard and flexible scoring rubrics, and reporting requirements. Our test construction approach includes review of training documents, observation of current methods for combat medic hands-on skills assessments, and obtaining subject matter expert (SME) input to determine the assessment content and dimensions; scoring rubric development, and; verification and validation of both test and scoring rubric through piloting and SME review. Benefits of summative standardized assessments include Army-wide competence measurement, inter-unit comparisons, inter-Soldier comparisons, and providing feedback for training processes. Beyond MSTCs and medical tasks, however, the lessons learned here may help strengthen any performance assessment effort. Most importantly, MSTC-based assessments of first responder skills will inform training and operational decisions that could minimize preventable deaths on the battlefield.