Assessment is a key component of education across society. Regardless of whether the setting is academia, industry, military, or non-profit organizations, assessment is essential for gauging educational effectiveness, providing remediation to students, and informing policy and decision-making. However, the use of thorough assessments can be resource intensive. For example, instructors must devote time and effort into scoring/grading assessments. This can be especially costly when teaching complex skills that are not easily measured by convenient means such as multiple choice examinations (e.g., leadership, problem solving, critical thinking, communication). However, one can argue that these kinds of complex tasks are the ultimate goal of any educational system. Computing holds great potential for reducing the burdens associated with assessment tools designed to measure complex skills. As a case in point, consider the Consequences test (Christensen, Merrifield & Guilford, 1953). It has been used to predict meaningful outcomes for military Officers, but the scoring of the test is extremely time-consuming as it requires test administrators to read and categorize test-taker-generated statements involving the outcomes of hypothetical scenarios. If the scoring of such statements could be automated, the test would become much easier to administer widely as the costs of the assessment would be drastically reduced. The challenge to implementing such a solution has been that computationally processing natural language, especially the kind of free form, conversational responses common in everyday life, is complicated. Nonetheless, tools already exist that show potential for utilization in assessment systems that necessarily use highly unstructured, free text input. In this paper, we discuss the use of open source Python libraries for assessing short answer, free form responses in the Consequences test. Using Latent Semantic Analysis, a well-established technique that has been around since 1988, we were able achieve human-computer response categorization interrater reliabilities comparable to human-human interrater reliabilities.