Abstract:

Gaining a better understanding of cyber attacker behavior can help to drive the development of more effective cyber defenses. Studying true hacker behavior, however, requires the development of highly realistic and well-instrumented cyber testbeds. In this paper, we describe key simulation challenges that we experienced while developing testbeds to execute four human subjects studies analyzing the ability to exploit cognitive biases of cyber adversaries and solutions we developed to address these problems. Specific challenges addressed include the unique experimental challenges of testbed configuration and management across numerous human subjects, generating content and variation for within-subject studies, effectively instrumenting the testbed to understand hacker behaviors, and providing guidance to ensure studies can be completed within the constraints of the study. For example, one area of challenge that we confronted was guiding attackers down specific pathways pertinent to experiments, to ensure we could collect pertinent performance data. We will describe our approaches to addressing these challenges, and how they enabled incrementally more efficient development of testbeds for studying the cognitive constraints of cyber attackers. Specifically, we will describe: (1) how we configured sets of testbeds to ensure experimental consistency across subjects as they executed each trial twice; (2) how we used LLMs to generate files and other content within each testbed, populating the testbeds with unique and realistic artifacts rather than using existing, recognizable data sets; (3) how we used a variety of methods to manage the time and progress of attackers across a range of skill levels; and (4) how we incorporated realistic tool suits common to current SIEM pipelines to ensure the tasks executed by hackers were directly relevant to their experiences on live networks.

Keywords: CYBER;ENVIRONMENTS;HUMAN PERFORMANCE;SIMULATIONS;SYNTHETIC ENVIRONMENT

Michael Sieffert

Affiliation

Biography

Refine Results

Challenges and Solutions in Using Virtual Testbeds to Study Hackers