What is measurement?

You can watch this post on video instead of reading if you want to see my The Sea Hates a Coward t-shirt

Hi! I had promised to talk about how to do a good job of risk analysis. So I’m going to write a few emails that condense and present in plain language each of the big ideas that I think are important for this.

The way I like to think about doing a good job of risk analysis is: what outcome do you get if you are doing a good job. The outcome you get is better decisions. So if your risk analysis is helping you make better decisions, you are doing a good job of it.

The goal of risk analysis is to enable better decisions

One note: something you have to be careful of is outcome bias. If you make a decision about a risk, and the bad thing happens, or doesn’t happen, that by itself, the outcome, doesn’t mean you did a good job analyzing the risk.

Consider someone doing something foolish, a dangerous stunt, and not getting hurt. That doesn’t mean they correctly analyzed the risk, it means they got lucky. Watch out for outcome bias.

The first big idea is what is measurement.

Measurement is reduction of uncertainty.

When we ask someone to provide an expert estimate of the likelihood of a bad thing happening, how much risk is there around a certain thing, we now have to figure out, how do we make that measurement? When we ask someone to provide an estimate about something where there is uncertainty is, the common reaction is: “that’s impossible”. “There is no way of knowing, it might happen, it might not happen”.

A book with some useful insights on this is How To Measure Anything in Cybersecurity Risk by Douglas Hubbard.

One of the big ideas in that book is that measurement is reduction in uncertainty. That is a very useful definition!

Say for example you want to estimate how many laptops are likely to be stolen from your company next year.

“Thats impossible to know!”

“How can you possibly know?”

But if you take the approach of reducing uncertainty, with that subtle mental shift you can come up with an interesting estimate.

“Well my company has 10 people, and we have 10 laptops - oh we have 1 spare - so we have 11 laptops, and last year 1 laptop did get lost…”

Now with that information, as an expert on your business you can discard some impossible values. You know 50 laptops aren’t going to be stolen because you don’t even have 50 laptops. And you can probably guess if 1 laptop got stolen or lost last year, maybe 0 is not a reasonable number. Or maybe you still think 0 is a reasonable number.

But now you can narrow that range dramatically from “it’s unknowable”, and you have an interesting estimate.

So thats the big idea for today: measurement is reduction in uncertainty, and a key technique in figuring out a measurement is to start with absurdly wide values, and based on what you know, or some key assumptions, you can reduce the amount of uncertainty and BOOM! You’ve got a measurement.