Demonstrating the mission relevance of advanced training and rehearsal systems and their focus on training and evaluating warfighter needs is best achieved with objective metrics that can highlight mission performance changes. However, for a variety of reasons, it has historically been much easier to evaluate training in the traditional ways, that is, focusing on student evaluations and end-of-course tests as opposed to examining on-the-job behaviors and organizational or mission success. Student evaluations and learning tests are easy to implement but may or may not be explicitly tied to the overall training objectives. Further, these assessments do not provide any indication of the impact of training on job performance or mission effectiveness. Critical reviews found lack of an integrated system for measuring and assessing training performance, over-reliance on subjective measures of performance, and a shortage of valid, reliable, quantitative performance measures of training effectiveness. This paper highlights initial research and data collected to develop an Aircrew Measure of Effectiveness (MOE)/Performance (MOP) Hierarchical Taxonomy capable of assisting training and mission evaluators. The paper details our approach and provides data on sample mission task MOE/MOP decompositions to illustrate how a taxonomic approach can help diagnose actual aircrew mission performance of both individuals and teams. While this approach shows much promise, many technical obstacles need to be overcome before it can be completed and used routinely in an automated form. We highlight and discuss these technical challenges, propose solutions, and provide an agenda for needed research. Implications and potential future applications of the approach are discussed.