Advances in speech recognition technology have enabled the automation of grading for speech-based training courses that have historically required a trained instructor. This paper describes an effort conducted to apply this technology to Navy shipboard radio operator training. This called for simulation of two-way radio communications between the student and various Non-Player Characters (NPCs).

The Navy training use case calls for instances where a student was required to provide a verbatim radio response to a prompt, response consisting of a set of words delivered in a specific order, as well as non-verbatim radio responses, whereby students are allowed to include extra words as long as they start with a salutation and include a phrase from each required keyword slot.

In place of an instructor, these verbatim and non-verbatim instance courses use speech recognition to grade students during prompted interactions with NPCs in classroom training environments. The team generated validation corpora of human and text-to-speech (TTS) utterances representing true positive and true negative (incorrect) elements of the verbatim and non-verbatim coursework utterances.

For coursework grading, we designed and built a system based on an all-in-one speech recognition and analysis (parsing) tool. We created custom language models to recognize the required verbatim and non-verbatim utterances and refined these models based on our validation corpus. The team then conducted hyperparameter tuning through integration of an open-architecture testing system that leveraged formal methods, knowledge-based testing, and optimized search. This paper outlines the details of our approach for corpus generation, language model construction, and model validation. Results are presented achieving 97% recognition accuracy on our validation corpora of over 24,000 accurate and inaccurate utterances. Results suggest that a finely tuned and trained speech recognition and parsing model can work extremely well for the use case of supporting radio operator training.

Morgan Ulinski

Affiliation

Refine Results

Automated Radio Operator Utterance Recognition for US Navy Training