High-fidelity simulations of real world systems such as traffic, physical infrastructure and the civilian population are the next frontier in improving the depth and sophistication of military simulations. Such complex systems are difficult to model with complete accuracy, and often involve unobserved free parameters. To make these simulations as realistic as possible, these parameters need to be tuned by matching the simulation output to real world data collected from these systems in a process known as “calibration”. Typically, calibration is approached via brute force exploration of the parameter space in an attempt to identify the parameter setting that best reproduces patterns seen in real-world data or ascertained by subject matter experts. However with large numbers of parameters and computationally expensive simulators this quickly becomes intractable.
In structural engineering and elsewhere, this problem is often overcome using “surrogate-based” optimisation, wherein a computationally efficient surrogate model is trained on sample input-output pairs from the simulation, and is thereafter used for rapid parameter exploration. Candidates for surrogates have included machine learning models like random forests and deep nets, as well as interpolation-based techniques like kriging - the latter gives rise to a suite of tools known as Bayesian optimisation. Surrogates can be trained sequentially, in a process known as “active learning”: the next configuration is picked so as to maximise its information content, exploring the space sufficiently while still converging to the area that has so far produced the best answers.
We illustrate these techniques for the first time on a challenging calibration problem in traffic micro-simulation, using the industry-standard open-source simulator SUMO. Using real-world observed traffic data as our target pattern, we sequentially fine-tune SUMO’s initial state and configuration parameters to maximise the fit to the target pattern, resulting in a far more realistic simulator than that obtained with manual or brute-force fine-tuning.