We live in a virtual explosion of data. The Internet generates an estimated 2.5 quintillion bytes of data every day. Though the data from instrumentation on aircraft, vehicles, ships, autonomous systems, simulators, and, increasingly, humans themselves does not reach this scale, its volume is significant and increasing. It is natural to want to use this wealth of data to build realistic training scenarios. The chief difficulty is that, whatever events were recorded, they represent only one path through the world. This makes the recording suitable for replay, but a recording cannot give students the chance to make choices in the simulated world that would take them down different paths. Recordings cannot be directly used for training scenarios unless additional steps are taken. This means that accommodations must be made, through subject-matter expertise, machine learning, or both, to synthesize the data into realistic entity behaviors in a scenario. In this paper, we discuss our experiences building several systems that take these additional steps, which generally involve machine learning and intelligent agents, and we discuss in detail an effort that focuses on creating realistic constructive maritime patterns of life from real-world data. We conclude by discussing the training value of learning patterns of life from real world data, and lessons learned that will be useful to help other training professionals create realistic data-driven training scenarios.