Evaluating the effectiveness of learning technology typically involves determining the extent to which a learner has gained and can apply knowledge, with an emphasis on long-term retention. Research designs in these studies typically involve comparing a group’s performance before and after an instance of training, comparing a group who receives training to a group who does not, or comparing groups who have received different formats or levels of the same training. While these designs may be relevant for evaluating a single instance of training, they do not effectively address how people learn in the real world; learning is a process that occurs over time with repeated exposure to material across a variety of experiences and platforms. That learning can occur anytime and anyplace is the foundational premise behind the Experience API (xAPI), a specification designed to enable tracking and managing human performance data across training platforms and instances. One example of how these models fail involves evaluating the effectiveness of mobile learning. The benefits of mobile platforms are the ability to provide “just-in-time” training and performance support. The goal is not to support long-term retention of knowledge, but to provide just enough information at the point of need for the user to execute a task or solve a problem. This is a fundamentally different research question, and requires evaluations designed to determine the extent to which the user can apply information in real time. In this paper, we will discuss how xAPI data from a mobile learning application can be used to determine its effectiveness. To illustrate the methodology, we will present data from a recently conducted effectiveness evaluation of a mobile application designed to support Warfighters’ financial literacy.