The Air Force Research Laboratory, Human Effectiveness Directorate is using its high fidelity distributed mission training (DMT) simulation testbed to explore the impact of principled training on individual and team performance. One area of interest is the development of methods for assessing the impact of distributed mission training on pilots knowledge and understanding. In previous studies we have used traditional knowledge assessment methods, which have included paper-based fill-in-the-blank tests and computer-based concept rating tasks, pre- and post-training. With the development and definition of Mission Essential Competencies (MECs) as a novel way to define complex air combat mission proficiency, these more traditional approaches to knowledge assessment and learning are not at a level of specificity for measurement and proficiency diagnosis. This paper highlights the development and lessons learned from a vignette-based approach to knowledge assessment. Our initial development which is based on Situational Judgment Inventory (SJI) and Job Knowledge Inventory (JKI) research, used an open-ended paper-based assessment instrument, referred to as Situation Assessment and Action Selection (SAAS), to examine pilots assessment of air-to-air situations as well as their opinions on appropriate courses of action. Scoring of pilot responses was challenging. One limiting factor in using open-ended responses is the time and effort required to score them. We are exploring the use of automated scoring of the responses, beginning with Latent-Semantic Analysis (LSA). Successful LSA scoring would greatly enhance the utility of the method and support the next phase of development. The next phase of development is intended to be a more automated version of the instrument, referred to as the Air Superiority Knowledge Assessment System (ASKAS). Results from our evaluation of SAAS are presented and discussed. Lessons learned and a rationale for developing a multimedia-based assessment system is discussed. Finally, key features of ASKAS are described with respect to their potential for helping researchers and practitioners assess the impact of DMT on pilots knowledge and understanding.