The most common failing of existing lessons learned libraries is that important experiences are stored, but then are rarely retrieved when they might usefully inform decision making. Searching for lessons in existing systems is often seen as an inconvenient, time-consuming disruption when pressing decisions need to be made We describe an approach to proactively providing decision advice by extending task support tools to automatically formulate queries against a repository of past experiences. Retrieved texts are offered in context to minimize disruptions and maximize connection with ongoing work. Rather than depend on traditional text-based indexing and retrieval, we experimented with matching structured representations of a current problem/solution, pulled from decision-support tools, against narrative structures extracted from experiential texts.
We built a prototype system to support operational planners working on counterinsurgency and Stability, Security, Transition and Reconstruction (SSTR) operations. As an exploration of what an ideal information extraction system might usefully produce, we had subject matter experts select and annotate two-hundred experiential texts containing potential planning lessons. We then had a panel of forty-nine active duty and retired officers attempt mission planning exercises using a simple web-based planning tool enhanced with our automated lesson retrieval scheme.
On the specific question of whether the system could improve plan quality, a paired t-test on the users’ self-ratings of plan quality across two plans—one developed with advice retrieval support, the other without—showed a small but significant increase in self-ratings of plan quality. In opinion surveys: 69% of evaluators agreed that the system helped with planning; 92% agreed that the concept of experiential advice is valid and potentially useful for military planners; 88% agreed that the system should be further developed; and 82% agreed that work on the integrated planning tools should be continued.