Recent developments in Artificial Intelligence (AI) and Machine Learning (ML) are opening novel opportunities for human-autonomy teaming (HAT) to accomplish a task, a mission, or a continuous coordinated activity. The challenge is to provide awareness and control to the humans over autonomous assets and their actions, while having trusted interactions with them as teammates and supporting HAT shared contextual understanding to accomplish the task.

Addressing this challenge is crucial for the success of hybrid human-autonomous teams pursuing a common goal. To address this challenge, we present a real-time Human Digital Twin (HDT) software architecture that integrates Large Language Models (LLM) focused on knowledge reporting, answering, and recommendations, into a visual interface that provides a life-like physical embodiment of the autonomous system. We use a metacognition approach to empower the LLM to have a deeper and personalized understanding of the human (or humans) that it needs to interact with, so it can provide context-aware responses aligned with the human’s expectations and needs. Our HDT(s) then becomes a visually and behaviorally recognized team member(s) that can be integrated into the complete life cycle of a mission, from training to deployment to after-action-review.

We present an open architecture that provides a protocol to integrate customized LLMs to enhance dialogue quality and increase context sensitivity. This architecture encompasses sophisticated speech recognition, context-aware processing for adaptive learning responses, AI-driven dialogue generation, AI emotion engine, lip-syncing, and lifelike visual and auditory feedback. With this architecture, the HDT is capable of real-time interactions without requiring explicit dialogue in advance, capturing multimodal data to create a realistic in-context conversation.

This paper describes the HDT system architecture and its performance metrics, highlighting critical functionalities developed and opportunities for further development. Our HDT is targeted to support HAT through personalized interactions, heightened realism, and adaptability to context of operation.

Keywords
AUTONOMY;OPEN ARCHITECTURE

Additional Keywords
Human Digital Twin; Large Language Models; Human-Autonomous Teams