This paper discusses problems associated with the determination of tolerances for optical systems which are directly coupled to the human eye. Adequate total performance of the combined physical and physiological optical systems is shown to be dependent upon modifications of the optical image by the eye and processing of the retinal image by the brain. Physical, physiological, and psychological interactions are considered for application to simulator optical design.