The creation of behavior models for computer generated forces (CGF) is a challenging and time-consuming task, which often requires expertise in programming of complex artificial intelligence algorithms. This makes it difficult for a subject matter expert with knowledge about the application domain and the training goals to build relevant scenarios, and keep the training system in pace with training needs. In recent years machine learning has shown promise as a method for building advanced decision making models for synthetic agents. Such agents have been able to beat human champions in complex games such as poker, Go and StarCraft. There is reason to believe that similar achievements are possible in the domain of military simulation. However, in order to efficiently apply these techniques, it is important to have access to the right tools. This paper presents how open-source software can be used to efficiently establish an infrastructure for deep reinforcement learning, a machine learning technique which allows synthetic agents to learn how to achieve their goals by interacting with their environment. We begin by giving an overview of available frameworks for deep reinforcement learning, as well as libraries with reference implementations of state-of-the art algorithms. We then present a case study describing how these resources were used to build a reinforcement learning environment for a CGF software intended to support training of fighter pilots. Finally, based on a number of exploratory experiments in the presented environment, we discuss opportunities and challenges related to the application of reinforcement learning techniques in the domain of air combat training systems, with the aim to efficiently construct high quality behavior models for computer generated forces.