Notes from the field: Are you comfortable? Or a bad way to measure risk

I often hear this in meetings where we discuss risk.  “Are you comfortable?”  I always feel like answering:  yes, I’m working from home, I’m in my own office, I’ve got better computers and screens than at work, natural light, no suit, dog at my feet, and good coffee, so yes, I’m very comfortable.

This really isn’t the question though.  The real intent of the question is am I ok with/will sign off on the risk of the environment proposed.

My real answer should be, it doesn’t matter my level of comfort, I’ve measured the risk, checked all my controls and they are met, so the risk is quantitatively at an acceptable level relative to our other risk.

I get presented with a lot of models that go something like this.  Based on a couple of questions and someone’s interpretation of the answers they make a traffic light model, except for colors, they are substituting a scale of 1, 2, 3.  If it’s a color or a number, its still based on their opinion of a qualitative standard.  So, it’s pseudo-math, or a lipstick of math on a qualitative pig.  This is just the problem:  lack of a quantitative measure that forces you to “go with your gut”, i.e. “your comfort level”, and rate it.

For instance, their first category in one of these models was labeled "Capability Documentation". They were gonna judge 1 = none, 2 = partial, 3 = complete.   I asked, how do you measure complete, and the only answer was “how complete and good we think the documentation is”.  This is just, then, gonna be there opinion.  If they assign numbers to that and add, subtract, multiply and divide them, it makes no difference.  Its not objective measure, it’s just opinion.

Psychologists tell us that people are notoriously bad judges of relative differences.  This is pretty much borne out by these models as everything tends to wind up 2/medium/yellow.  Also, if everything is a judgment call, nothing ever gets off yellow.  At the end of the day we live in a risky world (even breathing is a risk:  you are burning oxygen in your system), so none of us can ever eliminate risk and be “comfortable” or “green”.

What would really get you somewhere would be to actually measure some quantities: 

1 - how long is the doucmentation (pges), 

2- how many people have contributed to it (number), 

3 - how many people have reviewed it (number), 

4 - how many places is it posted (number).

5 – I could think of some others…

In this way,  if you measured objectively some quantities for every process, eventually the processes will start to cluster around some numbers:  e.g. some processes will have longer documentation (one can suppose that that is more complete, at least to a first order), a larger number of people will have reviewed it, and a larger number of people will have contributed it.  So, we can now make a stab at defining “good documentation”, based on experimental data, as those quantities above that cluster at the high numerical end, and “bad” as that which clusters at the low numberical end.

This is similar to how we first started to measure electrical phenomena, e.g. what is an electron.  People tried to determine the state by measuring its mass, volume, speed, charge, because no one could actually see an electron.  Pretty soon the measurements clustered around what we now accept for those quantities, and those are the accepted measures.  The accepted measures then told us something about what an electron was and still describe the quantum mechanical properties of the electron.  To date though, no physicist can tell you quantitatively what an electron “is”/looks like.  But, it really doesn’t matter for quantum mechanics.

So, let’s get out of our comfort zone and actually start measuring risk.  In that way, we can actually start managing it.