The importance of fidelity in simulation training has been established as a key factor in training effectiveness. Design and development of simulation systems is dealt with commonly as an iterative procedure in which the goal is to improve on the training effectiveness of the system. Improvements are made based on methods of identifying and resolving deficiencies. Improvements must then be considered relative to available resources to determine how to prioritize the multiple options. The current paper presents a flexible methodology for evaluating the relative effects of multiple deficiencies on a simulation system, utilizing information from warfighters about the effectiveness of a Deployable Tactical Trainer (DTT). First, subject matter experts were asked to evaluate the fidelity of the current DTT system across a set of 199 experiences and emergency procedures (EP). Next, the same subject matter experts identified deficiencies of the current system. Then the warfighters evaluated the 199 experiences and EPs for which deficiency had the most adverse affect on training. A composite score was computed for each deficiency, using weighted variables accounting for 1) the amount of improvement possible and 2) the importance of training for each experience. The deficiency scores provide a means of comparing the relative effect of each deficiency on warfighter training. The impact of utilizing multiple sources of information is presented through a comparison of the different decisions that would result from using partial information or alternative weighting methods. The proximal implications of using the proposed methodology to have the greatest impact on improvement are discussed. The distal implications of the impact the improvements have on pilot perceptions, and ultimately, on objective pilot performance measures, are also discussed. Also discussed is the versatility of the methodology for incorporating information from various sources and weighting systems to include alternative decision-making factors.