Cyber Risk Quantification is Stupid and Bad (Part 1)
Cyber Risk Quantification is Stupid and Bad (Part 1)
Stop. Just stop.
Where’s the data?
Cyber risk quantification is a hot topic right now. Current qualitative tools are ridiculously handwavy and bad, and everyone wants quantitative risk measurements—the holy grail.
The problem is that measuring cybersecurity risk is an unsolved problem in computer science, and the current industry approach to quantitative cyber risk is harmful and counterproductive.
If you're not already familiar with the stupidity of qualitative risk measurement, let’s just say security people look at a bug and wave a magic wand over it and bless it with a label like LOW or MEDIUM or HIGH or—if you really want to scare people—a CRITICAL.
We then proceed to create heat maps of how many CRITICALs we have in our environment and proceed to freak out about it. Clearly this is stupid, and for all sorts of reasons—a CRITICAL in your environment could be a MEDIUM in mine, or even completely unexploitable if I don't have that particular tool.
The world is crying out for more quantified risk metrics, trumpet the risk quantification prophets. Except—wait for it—current approaches to risk quantification ARE FOUNDED ON QUALITATIVE MEASUREMENTS. Like so:
“What is the financial impact of this particular attack path in our threat model?”
“Between 1 and 1.5 million dollars per incident.”
OK. That might be fine. You could be wrong but not likely an order of magnitude wrong. But risk is the combination of likelihood and impact (often expressed using the mathematically ridiculous formula “risk = impact X likelihood”).
“Well, I, as a cybersecurity professional, estimate the risk as a 1 in 20 year occurrence.”
So wait. Whoa there. Backup backup backup. Your measuring stick is qualitative. You've just built an entire quantitative mental model BASED ON QUALITATIVE MEASUREMENTS.
Quantitative measurements involve hard scientific facts, like millimeters of rainfall, or velocity of a car at time of an accident, or any number of other unambiguous, indisputable, reproducible hard measurements. There is nothing at all scientific about me guessing how likely a cyber attack is. (It's useful, it's actionable, but it's still a qualitative judgment on my part.)
And that's before we even leave the realm of KNOWN KNOWN risks. What about the KNOWN UNKNOWN and UNKNOWN UNKNOWN risks?
How do you propose to measure the likelihood of the KNOWN UNKNOWNS? How do you propose to quantitatively measure either the impact or likelihood of UNKNOWN UKNOWNS?
Please show me your risk quantification metrics for a Black Swan.
And to be clear, cybersecurity is a Black Swan-rich environment, with systemic and catastrophic risk built into most of the systems we rely on.
I once chatted with a team attempting to build models for cybersecurity insurance. They came from a background in hurricane insurance. They have weather data for the last 150 years for South Florida, and said they could tell me with a high degree of precision the likelihood that a hurricane in any given year destroys any particular house in Miami.
With data like that, you can engage in actuarial science, and on that foundation build a profitable insurance business.
And in case it's not clear, what security professionals do for a living is a form of insurance. Of risk management. The reason we want to quantify risk is so that we can spend the right amount of money, and no more than that, to manage security risk to an acceptable level.
So what would REAL cyber risk quantification look like? We will know that we have achieved meaningful cyber risk quantification when insurance companies can consistently make a profit selling cybersecurity insurance.
Not until then.
At present, it is almost impossible to acquire cybersecurity insurance for more than $100 million. I recently saw a crypto custody platform advertising $250 million of specie insurance—on total assets of upwards of $5 billion under management!
When the impact of a security incident can easily run into the multiple billions of dollars, and lead to lengthy lawsuits just to get the insurance payout, then you know we are nowhere near that level of rigor today.
But what about the things we can quantify? Shouldn't we at least quantify those, and take action based on what we CAN measure?
Of course we should. So let's talk about piracy.
