Does “validityâ€� refer to the quality of an assessment, reliability of simulator outputs, or accuracy of internal simulation models? This question emerges in medical simulation and training, as educational, clinical, and engineering communities intersect. Each has developed a validation approach to meet their needs, without clear understanding of the other perspectives. Historically, validity has been assessed using a classical framework of content, criterion, and construct validity, concluding that a simulator is or is not valid. Validity has evolved into a unitary concept of construct, consisting of five distinct sources: content, response process, internal structure, relation to other variables, and consequences. Evidence for each source supports a score interpretation for a specific population, under a specific use case. This does not indicate that the assessment itself is generally valid, much less whether the simulator can be relied upon to deliver accurate results. This unitary framework was adopted by the American Psychological Association as the standard for validating assessments and was recently endorsed as the “gold standardâ€� for validating training tools. While this framework is effective for evaluating the appropriateness of an assessment, it may not be as robust for evaluating a simulation device used for assessment. This framework does not account for the physical and functional requirements of a physical system and the implications that discrepancies in those aspects may have on training and assessment. This paper compares the classical and unitary validity methodologies with a perspective on the application to training simulators, as well as examines the inherent limitations of both. Recommendations and industry standards from other fields are also examined for applicability to surgical simulation. Finally, a recommendation for the validity classification of surgical simulators is proposed. The future of surgical certification and licensing could be reliant on simulation, however validity standards must be established to support this goal.