Cyber Risk Quantification is Stupid and Bad (Part 3)
Stop. Just stop.
Welcome to Part 3! If you missed the first two installments, here’s Part 1 and Part 2.)
Managing cyber risk is like boxing blindfolded against Mike Tyson.
You know where you are—a boxing ring. You know who’s out there—Mike Tyson. But you have no idea when the punch is going to happen, or where he's going to hit you, or how quickly you can roll with the punch when it comes.
But you dance and weave, and know eventually it's going to hurt.
This extreme level of uncertainty and ambiguity causes significant psychological stress (on which more another time). But beyond that, it's incredibly frustrating for non-security executives who want more certainty to inform their decision making.
Give me a security forecast as reliable as a sales forecast. Why can’t you security people tell me the future? Where is your crystal ball, Mister Fancy CISO Title, huh?
Wouldn’t it be great if we could collapse the infinite unknown into a nice little box of known quantities?
So in order to give executives what they want, and to calm the mental anxiety, we’ve invented a comfort blanket that makes the whole problem magically go away.
For a couple of decades that was security compliance. We can’t measure real security risk, but boy can we ever measure legal compliance!
Are we fully compliant with HIPAA? GDPR? CCPA? IDGAF, WTF, WTH, and the rest of the long absurd list of acronyms.
Yep!
Fully compliant! Fully secure.
Job done. Forecast: ready. Future: known. Aye aye, cap’n!
…right?
Except of course this didn’t work. All it did was quantify what minimum security due diligence looked like, put lawyers in charge of cybersecurity, and encouraged them to calculate the likelihood of regulatory enforcement.
Twenty years of this and we’re tired.
So now, instead of facing the infinite unknown, the dread uncertainty and ambiguity of the KNOWN UNKNOWN and the UNKNOWN UNKNOWN, we’ve now devised yet another straw man to comply with: cyber risk quantification.
If only we could collapse the infinite into the finite, if only we could quantify the unquantifiable, if only, if only, if only…
But the problem is that we can’t. And we won’t. Because it’s not possible.
Cybersecurity strategy remains an art the same way that military strategy remains an art. At the big picture level, the struggle between active human adversaries for dominance is not a quantifiable science, and can never be a quantifiable science.
And while playing civilian cybersecurity defense is not a military job, we nevertheless face off against active military adversaries on a daily basis.
In fact it's harder, because as a civilian security leader I am compelled by law to fight with one hand tied behind my back—I can play defense all I like, but I may not ever, under any circumstance, engage in offense, even against foreign adversaries that mean my employer harm.
If your problem space is so small, and your real security risk so negligible, then maybe you can get away with small-mindedly focusing only on regulatory compliance, or focusing only on risk quantification of the KNOWN KNOWN and ignoring the KNOWN UNKNOWN and the UNKNOWN UNKNOWN.
But the cyber defense of any company of any significant value requires you to go beyond that.
Accepting cyber risk as part of doing business is always a legitimate business decision for a CEO and Board of Directors. But they must do so explicitly, and as CISO your job is to measure risk as best you can—using whatever imperfect mixture of qualitative and quantitative metrics you have at your disposal—and to inform the business owners so they can make a decision.
But ignoring risk amounts to professional malpractice.
All models are wrong but some are useful. Cyber risk quantification is yet another metaphor to help us look at cyber risk measurement. The point of this long three-part blog post series is not to argue that cyber risk quantification is useless—far from it—but rather the danger lies in dogmatically hewing too closely to the model. Which is clearly a huge temptation for those psychologically alarmed by the extreme levels of uncertainty and ambiguity in the security profession.
Take all metaphors with a big grain of salt, including cyber risk quantification.
The danger you face is Goodhart’s Law. And this is a bigger danger in cybersecurity than in many other domains.
Goodhart’s Law says: “When a measure becomes a target, it ceases to be a good measure.”
If you:
only manage what you can measure, and
you can only measure a small fraction of cyber risk, then
you will only manage a small fraction of cyber risk.
That leaves a big chunk of risk completely ignored because you cannot force that unmeasurable risk into your quantitative mental model.
When you can't make something a science, it remains an art. And at least for present and foreseeable future, cybersecurity risk management remains more art than science.
Sit in the ambiguity and live there. That's your job as a CISO.
