Measuring effectiveness and the operational utility of new techniques, technologies and training through feedback of the Warfighter is critical. Paper questionnaires and interviews have merits. However, offering a relevant, compact questionnaire after each mission to test and exercise participants with different roles and backgrounds can be made more efficient with an automated survey. This paper discusses the challenges faced in survey data collection in an operationally realistic environment, and lessons learned during years of survey data collection. During the Bold Quest cycle, a survey effort was set up using such an automated tool (Questionnaire for Utility Evaluation and Survey Tool, QUEST). The concept of employment for automated surveys described here addresses many of the concerns associated with web-based surveying techniques (Sills & Song, 2002). Use of clear and concise questions and other measures lowering the burden for the respondents yield the best results from a survey effort. An automated tool ideally should work locally on any laptop or handheld device, as well as in networked conditions. It should be tailored to military environments, offering questionnaires for all phases (pre/post exercise, daily post mission). By tooling the questions to reduce burden, analyzing the language used, and taking steps to assure the relevance of the questions to each participants, more efficient data collection can occur – the kind of data collection which provides 100,000s of survey responses helping to determine the effectiveness of new military developments.