Main Subcommittee Category: Education

How would you label your submission?: Industry

Abstract:

Advances in speech recognition technology have enabled the automation of grading for speech-based training courses that have historically required a trained instructor. This paper describes an effort conducted to apply this technology to Navy shipboard radio operator training. This called for simulation of two-way radio communications between the student and various Non-Player Characters (NPCs).

The Navy training use case calls for instances where a student was required to provide a verbatim radio response to a prompt, response consisting of a set of words delivered in a specific order, as well as non-verbatim radio responses, whereby students are allowed to include extra words as long as they start with a salutation and include a phrase from each required keyword slot.

In place of an instructor, these verbatim and non-verbatim instance courses use speech recognition to grade students during prompted interactions with NPCs in classroom training environments. The team generated validation corpora of human and text-to-speech (TTS) utterances representing true positive and true negative (incorrect) elements of the verbatim and non-verbatim coursework utterances.

For coursework grading, we designed and built a system based on an all-in-one speech recognition and analysis (parsing) tool. We created custom language models to recognize the required verbatim and non-verbatim utterances and refined these models based on our validation corpus. The team then conducted hyperparameter tuning through integration of an open-architecture testing system that leveraged formal methods, knowledge-based testing, and optimized search. This paper outlines the details of our approach for corpus generation, language model construction, and model validation. Results are presented achieving 97% recognition accuracy on our validation corpora of over 24,000 accurate and inaccurate utterances. Results suggest that a finely tuned and trained speech recognition and parsing model can work extremely well for the use case of supporting radio operator training.

Keywords: AI;AUTHORING TOOLS;MACHINE LEARNING;NATURAL LANGUAGE PROCESSING

Additional Keywords not in the list above: automated speech recognition