Cyber Risk Quantification Makes No Sense

How Do You Quantify Uncertainty?

Only manage what you can measure, amirite?

I got a lot of feedback (and some thinly-disguised hate) for my three-part series shredding the badness and stupidity of Cybersecurity Risk Quantification (which practitioners of said mumbo-jumbo pseudo-science refer to by the confusing acronmyn “CRQ”).

CRiQey, I guess?

That’s not CRQ. Now this…

But one conversation stuck in my mind:

Random Internet Stranger: “But we can quantify uncertainty.”

Me: “Umm… OK, how?”

RIS: “Using probability.”

SRSLY?

So let's go back over the facts of life here, shall we boys and girls?

To calculate probabilities, you need data. Lots of it. Preferably all the data.

I worry that what you just heard is that I want some of the data.

You need enough data to quantify the likelihood—the probability—of different outcomes happening. Including the bad outcomes you don’t want.

How do you know if you have enough data? If you can run a profitable insurance business betting on the odds of bad things happening. Enough data to calculate the likelihood and impact of bad things happening and make a profit over the long term.

For instance, it is possible to insure against hurricane risk in South Florida because we have 150+ years of meteorological data. Even modulo the small delta due to climate change, it’s still possible to quantify risk to a very high degree.

But in the cybersecurity world, insurers are actively refusing to insure against cybersecurity risk. Why? Because they can’t quantify the risk well enough to turn a profit.

And this is before we consider the delta of change in cyberdata. (If you smoosh two words together in English, you get a new word. Stay with me here.)

Your cyberdata from the 1990s has close to zero utility in the 2020s. Consider that smartphones only became ubiquitous about a decade ago. That the enterprise move to the cloud—still ongoing—is also only about a decade old. The rate of change is so high that your data gets stale really quickly.

Even if you somehow had access to all the data—maybe you’re NSA, you’re Five Eyes, who arguably hold the position of global passive adversary on the internet (minus most Russian and Chinese network traffic)—you would still have to contend with that delta of change.

And that’s an extreme example. Private sector does not have even a small fraction of the data that Five Eyes mass surveillance agencies possess. A KNOWN KNOWN risk to NSA is a KNOWN UNKNOWN or even UNKNOWN UNKNOWN risk to a private sector insurance company, or to my employer.

So the first, and most basic point that we must contend with here, is that you cannot calculate probabilities without data, and we don’t have that data, much less fresh data. We just flat out don’t have it, and are not likely to ever have it.

Second, even if you’re Five Eyes, and have all the data in the world, you still don’t have a crystal ball, you still don’t know the KNOWN UNKNOWNS or the UNKNOWN UNKNOWNS.

How can I quantify the probability that I don’t know something (KNOWN UNKNOWNS)? How can I quantify the probability that I don’t know that I don’t know something (UNKNOWNS UNKNOWNS)?

I is confused.

Please explain to me Mister Green Book how you are going to quantify that.

Third, and perhaps the most important point of all, is that the internet is called, with good reason, the Fifth Domain. A domain of warfare. Of struggle between raw geopolitical forces that operate outside of law or morality for power.

The Treaty of Westphalia is obsolete, and like submarines playing hide-and-go-seek in the Mariana Trench, militaries armed with exploit tools play hide-and-go-seek on civilian networks. This is the digital equivalent of armies marching through wheat fields to fight each other on private property (only less obvious).

What is the risk of war? The probability of war?

No insurance company in the world will insure you against war. The nature of insurance is that it assumes that risk is not systemic in nature. If you insure against earthquakes and hurricanes at the same time (a good idea), it’s unlikely that a hurricane will wipe out Miami the same year an earthquake flattens San Francisco.

But cybersecurity risk is systemic and catastrophic in nature, because the militaries of sovereign nations will, without hesitation, project power on the Fifth Domain in pursuit of their global geopolitical power objectives.

And if the whole internet gets trashed in the process? Well, war is just politics pursued by other means, as von Clausewitz said. Violent means.

“Burn everything to the ground” has been a successful, if deplorable, military tactic in wars since the dawn of time.

Do we really think warfare on the Fifth Domain will be any different?