Over the years, the use of Artificial Intelligence (AI) solutions for high-consequence decision-making tasks has become increasingly compelling. However, the application of AI in these settings present legal, moral, and ethical challenges. Due to their highly complex and opaque decision-making, AI agents tend to be viewed as “black boxes.” In addition, testing these models is often difficult as their performance on one set of inputs does not necessarily infer their performance on others, making it difficult to predict unexpected behavior. For these reasons, placing full trust in high-consequence AI decision-making has not been feasible.

This paper is a continuation of our research and development on the topic of trust in AI. At I/ITSEC 2023, we presented a methodology to evaluate AI agent trustworthiness through behavioral modeling. Our research has continued to progress toward a new methodology used to quantify the degree to which AI-enabled systems can be trusted to operate across various scenarios in real-time. In this study we utilize a variational autoencoder and gradient mapping techniques to obtain insight into model reasoning and provide objective trustworthiness metrics with intuitive visualizations for human observers. Our proposed method provides distinction between areas of model strength and weakness across varying inputs, as well as detection of out-of-distribution data.

While performing this study, the authors review and compare our approach with current state-of-the-art methodologies in this domain. This paper includes an updated background and literature survey based on new research. Our findings are presented in the context of an experiment to test the methodology used in our approach. It then builds upon this approach to explain how we have applied this methodology to the test and evaluation of piloted and autonomous aircraft operations.

Keywords
AI;VERIFICATION, VALIDATION AND ACCREDITATION (VV&A)

Additional Keywords
Gradient Mapping, Use Case Analysis, Behavior Visualization