In this paper we describe lessons learned applying synthetic character vignettes to assess the success of an intervention teaching at-risk adolescents social skills. We designed vignette storylines to target those social skills - engaging in negotiation, maintaining emotional control, and seeking information. We also designed language models so that when participants verbally interacted with the synthetic character, the natural language system would interpret their input as reflecting one or more social skill. Similarly, we designed behavior models so that the character would attempt different strategies to require participants to demonstrate skills. Finally, we devised methods to measure participants' engagement compared to what their engagement would be during role-play. Results from our study are highly encouraging, with adequate measures of construct and criterion validity showing adolescents' behavior while engaged with the synthetic character indeed reflect social skills. Results also show the vignettes are sufficiently realistic to highlight differences in behavior between groups of participants, such as adolescents who are or are not exposed to the intervention. We provide rationale for using synthetic characters in a variety of assessment situations relevant to the military.