Launching a lifeboat in an emergency requires safety-critical proficiency which can only be achieved and maintained with hands-on practice. Simulators have been specifically created for offshore oil and gas personnel to practice lifeboat launching and maneuvering using representative equipment and virtual environments. As an alternative to live boat training, lifeboat simulators allow for practice in plausible, high-risk events in a safe, realistic environment. An automated simulator is an alternative offering the benefit of on-demand practice while expanding training capabilities. Providing training for these types of scenarios presents challenges for evaluating trainee performance in conditions traditionally not used in training because performance metrics may not exist. The study uses simulation to assess performance in lifeboat training from two perspectives: a live instructor and automated simulator. An experiment was performed to evaluate performance of lifeboat operators in an emergency scenario which included adverse weather and hazards. A simulator was used to provide a safe and controlled means to assess trainee performance. A rubric was created to define scoring for launching and maneuvering tasks in weather, including moderate sea sates. The rubric identified quantitative measures which could be used by the simulator and live instructor to assess performance. The study compared performance measurements taken by a live instructor and simulator with automated tracking as each assessed participants in a simulated emergency exercise. The results show the simulator provided an advantage of being able to consistently track performance on tasks where multiple performance criteria were measured simultaneously. The study also identified limitations in the simulator which were not present in instructor led evaluations, including subjective measures made through visual observation. The paper discusses how simulation can be used to automate scoring and reduce instructor workload, and how simulators can be used to measure trainee preparedness for an emergency event with waves and hazards.