April 19, 2012

Modeling the Effects of Base-rates on Cyber Threat Detection Performance

Modeling the Effects of Base-rates on Cyber Threat Detection Performance
Varun Dutt (varundutt@cmu.edu)
Young-Suk Ahn (ysahn@altenia.com)
Noam Ben-Asher (noamba@andrew.cmu.edu)
Cleotilde Gonzalez (coty@cmu.edu)
Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA
Cyber attacks cause major disruptions of online operations,
and might lead to data and revenue loss. Thus, appropriately
training security analysts, human decision makers who are in
charge of protecting the infrastructure of a corporate network
from cyber attacks, on different frequencies of cyber threats
(base-rates) is indispensable to improving their on-job
performance. However, little is currently known about how
training analysts on different cyber attacks, that differ in the
base-rate of cyber-threats, affects their on-job performance in
a highly dynamic environment, while confronting novel
transfer conditions.
We report a laboratory experiment where
human participants are trained on two different cyber-threat
base-rates, high and low, and are transferred to an
intermediate base-rate level of threats. The experiment helps
us to develop an understanding of the situational attributes
that participants attend to during their detection of cyberthreats.
A linear model that is based upon participants’
attended attributes and calibrated to the two base-rates during
training does well to capture the performance during transfer.
We use the calibrated model to generate predictions in novel
real-world transfer conditions that contain a low cyber-threat
base-rate and a shorter training period.
Enhanced by Zemanta