Abstract
The US Systems Approach to Training (SAT), and its equivalent in the UK (DSAT)
places a high demand on Training Design analysts. The process is highly manual,
resulting in thousands of hours of analyst work.
This paper describes our approach to deliver an automated process for Training Design
(TD) that can deliver consistent, high quality results as well as a mechanism for
validation and verification.
Working with a Training Management System (TMS) in service with the Royal Air Force,
we have developed a process using securely hosted Large Language Models (LLMs)
within a Retrieval Augmented Generation (RAG) architecture to automate the TD
process.
Each step of the process is controlled, including text extraction, chunking strategy,
retrieval, prompt generation, style and tone, and the output format. Using one model’s
output as the input for the next model (a hierarchical structure of LLMs), we can
generate results that are not possible to achieve with a single step process. This
approach allows us to handle higher-level output and focus on more specialised tasks -
a methodology known as Hierarchical Reasoning.
The human in the loop is still key to this process. The LLM generated results are fed
directly to the TMS application via an API, along with source references for validation.
This allows the analyst to accept the outputs, modify, or reject it altogether - all within
the TMS application. This feedback is then fed back to the Secure Data platform via an
API, leading to continuous model adaptation and improvement over time.
Using a real world source document of 2,000 pages, our process generated 4,000
duties, tasks and standards - compliant with JSP822 standards. This work would take a
full-time human analyst around 3 months to complete.
Our process required 15 minutes of compute time for a total cost of $325.