You Can’t Do Quantitative Risk Without Calibrated Humans

In cybersecurity, it has become almost a reflexive mantra: we need quantitative risk management. I agree—but often for very different reasons than those usually given. Before we can talk about models, Monte Carlo simulations, FAIR analyses, or loss exceedance curves, we need to confront a more basic problem.

We have not calibrated our detectors.

I came to cybersecurity through experimental physics. In that world, measurement is everything—but measurement without calibration is worse than ignorance. At particle accelerators, no one would accept a detector reading simply because it produced a number. The first question is always: how accurate is it, and under what conditions?

Every detector has:

• A sensitivity range

• A known error margin (± uncertainty)

• Biases and failure modes

• Drift over time

  
  

Until those are understood, quantified, and corrected for, the output is not “data”—it’s noise dressed up as precision.

Cybersecurity risk measurement is no different, except our detectors are human beings.

Risk Is Measured by People, Not Instruments

Unlike physics, cybersecurity does not have direct observables for “risk.” We do not measure breach probability with a photomultiplier tube or financial loss with a calorimeter. Instead, we rely on human judgment to estimate:

• Likelihood

• Impact

• Control effectiveness

• Threat actor capability

• Exposure duration

  
  

Even when these estimates are fed into quantitative frameworks, the inputs are still subjective human judgments. That means the true detector in risk management is the analyst, architect, CISO, or risk committee member providing the numbers.

Yet we treat those inputs as if they were objective measurements.

In physics, this would be unthinkable.

Every Measurement Has Error Bars

One of the most important lessons in experimental science is that a number without an uncertainty is meaningless. A result of “5” tells you nothing unless you know whether it is:

• 5 ± 0.01

• 5 ± 1

• 5 ± 10

  
  

In cybersecurity risk assessments, we routinely assign values—often with alarming confidence—without expressing uncertainty at all. Worse, we implicitly assume that different experts are interchangeable detectors producing equivalent readings.

They are not.

I recently observed this firsthand while listening to a podcast featuring three well-regarded cybersecurity experts. Each was asked to assess the risk of the same scenario. Their initial evaluations were not merely different; they were wildly at odds. Orders of magnitude apart as we’d say in physics, and so unreliable.

This wasn’t incompetence. These were smart, experienced professionals. What I was hearing was uncalibrated measurement.

Human Bias Is Detector Bias

In physics, detector bias is rigorously studied. We characterize systematic error, random error, saturation effects, and environmental influences. We correct for them statistically, or we redesign the detector.

Human risk assessors are subject to their own well-known biases:

• Availability bias (recent incidents loom larger)

• Anchoring (first number sticks)

• Overconfidence

• Loss aversion

• Organizational incentives

• Professional background bias (ops vs. audit vs. threat intel)

  
  

Yet in cybersecurity, we almost never attempt to measure these biases, let alone correct for them. We simply average opinions, escalate disagreements, or default to hierarchy.

That is not quantitative risk management. It is qualitative judgment wearing quantitative clothing.

Calibration Comes Before Quantification

If cybersecurity truly wants to be quantitative, the first step is not better math—it is calibration of human detectors.

In practice, this means:

• Giving multiple assessors known historical scenarios with known outcomes

• Measuring variance between assessors

• Identifying consistent over- and under-estimators

• Mapping individual and group bias statistically

• Tracking drift over time as experience, roles, and incentives change

  
  

Only then can we assign confidence intervals to risk estimates. Only then does it make sense to combine inputs into probabilistic models.

Without calibration, quantitative risk frameworks merely give us precisely calculated nonsense.

Disagreement Is Data

One of the most valuable insights from experimental science is that disagreement between measurements is not a failure—it is information. When detectors disagree, you do not average blindly; you investigate why.

The podcast example was illuminating precisely because the disagreement was so stark. It revealed:

• Different internal threat models

• Different assumptions about control effectiveness

• Different interpretations of likelihood

• Different implicit definitions of “impact”

  
  

Those differences should have been surfaced, measured, and reconciled. Instead, in most risk discussions, they are smoothed over to reach a decision.

That smoothing destroys information.

Toward Honest Risk Measurement

Quantitative risk management is not about replacing judgment with math. It is about making judgment measurable, bounded, and accountable.

That requires humility—accepting that:

• Our measurements are noisy

• Our confidence is often misplaced

• Our expertise does not make us accurate by default

  
  

In experimental physics, this humility is enforced by reality. The data does not care how senior you are. Cybersecurity has not yet reached that stage—but it will have to, if it wants to mature as a discipline.

Before we ask for better numbers, we must ask a harder question:

How well calibrated are the people producing them?

Until we answer that, quantitative risk will remain an aspiration—not a measurement.

The Path Forward in Cybersecurity

As we navigate the complexities of cybersecurity, it is essential to understand that the landscape is ever-evolving. The threats we face are not static; they change and adapt. This means our risk assessments must also be dynamic.

Embracing Continuous Learning

To improve our calibration, we must embrace a culture of continuous learning. This involves regular training and updates for all team members involved in risk assessment. By keeping abreast of the latest threats and mitigation strategies, we can refine our judgment and improve our accuracy.

Leveraging Technology

Technology can play a significant role in enhancing our risk measurement capabilities. Advanced analytics, machine learning, and artificial intelligence can help us process vast amounts of data and identify patterns that human assessors might miss. However, we must remember that technology is a tool, not a replacement for human judgment.

Building a Collaborative Environment

Encouraging collaboration among team members can lead to more accurate risk assessments. By fostering an environment where diverse perspectives are valued, we can uncover biases and assumptions that may skew our measurements. Open discussions about differing viewpoints can lead to richer insights and more robust risk evaluations.

The Importance of Documentation

Documenting our risk assessment processes is crucial. This not only helps in tracking changes over time but also provides a reference for future assessments. Clear documentation can help identify patterns in bias and improve calibration efforts.

Conclusion

In conclusion, the journey toward effective quantitative risk management in cybersecurity begins with calibration. By acknowledging the human element in our assessments and striving for continuous improvement, we can enhance our risk measurement practices. Only then can we hope to navigate the complex threats that organizations face today.

As we move forward, let’s remember that the phrase "calibration of human detectors" is not just a technical term; it’s a call to action. We must ensure our measurements are accurate and meaningful, paving the way for a more secure future.