The discipline of software testing is changing to align with the automated processes of Agile DevSecOps. Automated testing is executed by running test scripts or scenarios against the System Under Test (SUT) without human intervention. With many types of software (such as C4ISR applications), a challenging aspect of this testing is generating synthetic data and activity to stimulate the SUT.

Modeling and simulation (M&S) can help by providing representative test data from synthetic actors in a synthetic environment. Building on current M&S tools, a Simulation-Based Testing (SBT) approach can inject synthetic test data into a Continuous Integration / Continuous Deployment (CI/CD) pipeline. SBT requires two interconnected elements: M&S-as-a-Service (MSaaS) services and an Automation & Orchestration ecosystem.

This paper describes our Simulation And DevOps Integration Environment (SADIE) project, which built a prototype SBT toolset using existing COTS/GOTS software. SADIE is a cloud-native (deployable to any Kubernetes cluster) modular toolset that works with any SUT and CI/CD toolchain. It provides continuous automated testing based on a flexible architecture (users can author scenarios, jobs, and pipelines).

MSaaS is implemented with cloud-hosted microservices aligned with the NATO MSaaS Reference Architecture. A Scenario Editor is used to author relevant scenarios and a Scenario Repository stores and catalogs versioned scenarios. SADIE wraps existing Simulation Engines (STK, AFSIM, NGTS, OneSAF) into Simulation Service microservices.

The Automation & Orchestration ecosystem centers on the SBT Job Manager which coordinates execution of SBT jobs. The Job Manager connects Simulation Service Adapters (for configuration and control of Simulation Services) to SUT Adapters (to translate data into the required formats/schema and protocols for injection). It can be controlled from any build orchestrator (e.g., Jenkins) using a REST API.

We conclude with an end-to-end use case showing how SBT can use M&S to provide CI/CD quality gates based on regression testing results.

Menu

Alex Matthews