Interacting with a Virtual Environment (VE) generally requires the user to correctly perceive the relative position and orientation of virtual objects. For some VE applications, such as near-field mixed reality, the user may also need to accurately judge the position of the virtual object relative to that of a real object, for example, the user's hand. This is difficult, since VEs generally only provide a subset of the cues experienced in the real world. Complicating matters further, VEs presented by currently available displays are often inaccurate or may be distorted due to technological limitations.

One recent development is the Head Mounted Projection Display (HMPD). It has the potential to address this interaction problem. The HMPD uses a projection lens and retro-reflective material instead of the eyepiece found in a typical head mounted display. This solves several problems in mixed reality simultaneously. Specifically, it allows for proper occlusion, for correction of optical distortion, and for images to be retro-reflected off curved surfaces. HMPD technology has been researched for use in medical visualization and collaborative virtual environment applications (Rolland, Meyer, Davis, Hamza-Lup, & Norfleet, 2002; Hua, Gao, & Rolland, 2002).

The unique combination of visual cues provided by the HMPD technology can potentially allow one to effectively reach, select, manipulate, and release virtual cockpit controls, specifically, buttons, dials, and switches in a direct and natural manner. We present a test bed design and experimental designs to explore this feasibility. We provide basic task measures and expected results for interaction with these virtual objects. Finally we provide preliminary data for accuracy and precision of one key measure for object manipulation, the user's depth perception of virtual and real objects in relation to each other. The technology has the potential to replace the physical mockup of a cockpit with its virtual counterpart, making training systems more portable, deployable, and reconfigurable.