Among the most useful information in field manuals are descriptions of the procedures for certain tasks. While the assessment of procedural knowledge is typically performed using hands-on exercises, testing the recall of such knowledge could be beneficial for several reasons. First, it could serve as preparation for hands-on assessment. Second, it could also aid knowledge retention through mental rehearsals. Third, it can aid personnel in self-assessments. Finally, assessing knowledge of procedures is beneficial when the cost of failure is high. Successful implementation of procedure extraction from manuals will pave the way for automatically developing assessment criteria for simulation-based or hands-on performance assessments as well. For example, the skill throwing a hand grenade is best assessed through hands-on performance, but assessing recall of the procedure provides opportunities for testing recall before a hands-on assessment. With these benefits in mind, we investigated the potential for natural language processing (NLP) technology to accurately identify and extract descriptions of procedures from documents and construct assessment items from them. While neural network-based approaches are increasingly popular for NLP, we hypothesized that a linguistic patterns-based method would be sufficiently powerful for this task. We have implemented one such technique for procedure description extraction and subsequent generation of a test item based on the extracted procedures. We evaluated the performance of this approach in creating high quality test items that would be useful in practice. This paper will describe the technique used, provide examples, and describe a validation study to evaluate the results. It will also discuss the challenges that need to be addressed to improve accuracy and utility. The purpose is to demonstrate what level of accuracy is achieved with a rule-based approach and pave the way for future improvements.
Keywords
AI, ASSESSMENT, CONTENT GENERATION, HUMAN PERFORMANCE, NATURAL LANGUAGE PROCESSING, READINESS, TESTING
Additional Keywords
Procedural knowledge and skills