Abstract
The development of intelligent agents for combat simulations in wargaming has traditionally relied on rule-based, scripted methodologies, with deep reinforcement learning (RL) approaches only recently introduced. While scripted agents offer predictability and consistency, they struggle in dynamic scenarios due to their rigidity. Conversely, RL agents provide adaptability and learning capabilities but face challenges such as ‘black-box’ decision-making and computational inefficiency in large-scale simulations.
This paper introduces a hierarchical hybrid artificial intelligence (AI) approach that integrates the strengths of both scripted and RL agents. The system employs scripted agents for routine, tactical-level decisions and RL agents for higher-level strategic decision-making, balancing reliability with adaptability. This division of labor mitigates the limitations of each method while enhancing overall simulation performance.
To evaluate this approach, we implemented it in the Atlatl Simulation Environment, a combat simulation designed for AI experimentation. Our method introduces an RL Manager Agent, which assigns area objectives to scripted subordinate agents, leveraging RL’s decision-making power at a strategic level while allowing scripted agents to execute predefined behaviors at the tactical level. Through structured RL training and scenario-based evaluations, we demonstrate that this hybrid approach significantly outperforms both standalone scripted and standalone RL agents within our combat simulation.
Results indicate that this hierarchical integration leads to better strategic adaptability and improved agent performance which may enable enhanced scalability, making it a viable framework for future AI-driven wargaming applications. This study highlights the potential for combining rule-based logic with adaptive learning to create more robust and effective AI agents in combat simulations. Future research will explore scalability improvements and the integration of hierarchical reinforcement learning and large language model techniques to further enhance adaptability, scalability, and performance.