When Cybersecurity Becomes an Audit Exercise

I don’t write alot of straight cybersecurity articles anymore.    However, I sometimes look at cybersecurity programs in large enterprises and think: we have somehow made the audit the security program.

And then, inevitably these days, someone tells me I’m doing security wrong.

Perhaps they’re right.

But increasingly, I find myself coming back to something Phil Venables has written about repeatedly (see his blog, philvenables.com): the distinction between security as a business and operational capability and security as a compliance and control-assurance exercise.

I don’t think the answer is that audits, frameworks, controls or regulators are bad. Quite the opposite. In financial services, they are essential.

The problem begins when we confuse evidence that security controls exist with evidence that the organization is actually secure.

They are two very different questions

Consider these two questions:

Can you demonstrate that the required security control exists and operates?

and:

How much cyber risk have we actually removed?

They sound similar., but they aren’t.

An audit-oriented program naturally produces a chain that looks something like:

Policy → Control → Evidence → Finding → Remediation → Closure

There is nothing inherently wrong with that.

But an attacker doesn’t operate that way.

The attacker is thinking much less formally and is more experimentally driven:

Identity → Privilege → Endpoint → Cloud → Lateral Movement → Data → Business Impact

The first model produces assurance artifacts.  The second produces security outcomes.  The danger is that an organization can become extremely good at the first without becoming proportionately better at the second.

Why does this happen?

The uncomfortable answer is that audit-driven security solves a lot of organizational problems.  Imagine that an audit discovers that 17% of privileged accounts don’t have the required quarterly recertification.  That’s a wonderfully manageable problem that can be simply stated and simply resolved.  You can assign an owner, create a remediation date, produce evidence of the percentage coming down, and based on that close the finding.  Did it really make you more secure?

Now compare that with:

“Our identity architecture creates excessive privilege concentration, and compromise of these three service accounts could provide an attacker with a path into six critical business systems.”

That’s a much harder problem.  It may require redesigning applications, changing identity models specifically as well as the cyberarchitecture in general.  That all requires replacing legacy technology, obtaining engineering resources and disrupting business processes.

One problem is a control deficiency.

The other is an architectural risk problem.

Organizations naturally become good at solving problems that are easy to define, assign, measure and close.

The framework isn’t the problem

This is an important distinction.  NIST, ISO, regulatory frameworks and control libraries are not inherently the problem. They provide useful structure and, importantly, allow boards, regulators and independent assurance functions to have a common language to talk about risk and with that to establish whether an organization has implemented expected practices.

The problem with the audit only approach occurs when the framework becomes the objective.

The sequence can gradually change from:

“We need a framework to help us manage cyber risk.”

to:

“We need to comply with the framework.”

And then:

“We need evidence for every control.”

And eventually:

“Our cybersecurity maturity is 92%.”

The assumption from a risk standpoint then, is that we are good, and we’ve achieved our risk reduction goals.  I’ve seen that happen at multiple financial institutions.  At that point I think we should step back and ask a very simple question:

92% of exactly what and what does it do to reduce our risk?

A high control-compliance score doesn’t necessarily tell us how difficult it is to compromise the organization, how far an attacker could move after initial access, whether critical services can withstand disruption, or whether the organization can recover from a major incident.

In the real world, a control can be operating exactly as designed while the architecture around it remains fundamentally vulnerable.

Audit is a necessary layer—not necessarily the strategy

This isn’t an argument for abandoning governance or assurance.  For a major financial institution, that would be absurd.  Boards need assurance.  Regulators need evidence.  Management needs accountability.  Security leaders need to know whether important controls are actually operating.  But the relationship should look something like this:

Critical business services

↓

Threats and failure modes

↓

Risk scenarios

↓

Security architecture

↓

Security engineering

↓

Detection and response

↓

Resilience and recovery

↓

Metrics demonstrating risk reduction

↓

Control assurance and audit

The audit function is incredibly important.  But it shouldn’t necessarily be driving the entire chain.

The measurement problem

There is another reason this happens.  Cybersecurity has borrowed heavily from disciplines where measurement is comparatively straightforward.

We can measure:

• percentage of endpoints patched

• percentage of privileged accounts reviewed

• percentage of employees completing training

• percentage of applications scanned

• number of vulnerabilities outstanding

• percentage of controls passing testing

• number of audit findings closed

These numbers are useful.  But they can also become seductive.  Because they are easily measurable, they become targets.  And once they become targets, people optimize for them.

That’s essentially Goodhart’s Law applied to cybersecurity:

When a measure becomes a target, it stops being a good measure.

A security organization can therefore become exceptionally efficient at improving its security metrics without necessarily improving its resistance to a determined attacker by the same amount.

What does “good security” actually look like?

For me, the more interesting questions are architectural, and fundamentally about whether we are measuring the right thing, and in the right way that is measuring a property that actually measures the security risk in a proposed architecture (that is my physicist coming through — those people will recognize the Heisenberg Uncertainty Principle and canonical variables here).    

• If an attacker compromises an employee’s credentials, what can they do?

• If they compromise an endpoint, how far can they move?

• If they obtain privileged credentials, what prevents catastrophic access?

• If a critical application is compromised, what contains the blast radius?

• If ransomware takes down a major service, how quickly can the business recover?

• If a cryptographic algorithm is broken, how quickly can we migrate to something stronger?

Those questions don’t eliminate the need for controls.

They give the controls a purpose.

And that’s the distinction I increasingly care about:

Are we implementing controls because the framework says we should, or because we understand the risk they are intended to reduce?

Ideally, the answer is both.

But if we can’t explain the connection between the control and the risk, we should at least be honest about what we’re measuring.

The career problem

There is also a human dimension to this.

If you’re a security leader in an organization that primarily asks:

“Can you demonstrate that we’re compliant?”

then the person who is exceptionally good at producing assurance is going to look successful.

The person asking:

“Can we fundamentally change the architecture so that this organization is harder to compromise?”

will look disruptive.

That second person may be asking for:

• architectural change

• engineering investment

• removal of legacy systems

• changes to business processes

• different identity models

• better segmentation

• stronger resilience

• difficult conversations with application owners

Those things are much harder to put on a quarterly dashboard, and require over the short term more time, money and resources.  Because of that they are much harder to get approved.

Perhaps the real job of the CISO

This has changed how I think about the role of the senior security leader.  The job has never been to become anti-compliance.  That’s a dead end.    Nor is it to become the organization’s chief control administrator.  The interesting role sits between the two, and the third, which is a technical risk implementator and strategist.

The CISO should be able to say to the board and regulator:

“Here is how we meet the required control expectations.”

But also:

“Here is what those controls actually do for our risk.”

(That assumes the CISO has a good way to measure and manage the risk, which is a whole other post.). And, critically:

“Here are the architectural changes we are making because the current risk is unacceptable.”

That requires security leaders who can move comfortably between regulation, risk, architecture, engineering and business outcomes.

So, am I doing security wrong?

Maybe sometimes.

I’m perfectly capable of missing a control, misunderstanding a regulatory expectation or underestimating the importance of governance.  But I’m increasingly uncomfortable with the idea that security is primarily the successful completion of a control checklist.  A mature security program should be able to connect the dots:

Threat → Risk → Architecture → Control → Detection → Resilience → Business outcome.

Architecture should be underlined, because that is where a CISO and Cybersecurity department setup a strategy and make a conscious risk management decision.  If we can’t make that connection, we may have built a very impressive assurance machine.   And an assurance machine is not the same thing as a security program.  The uncomfortable question isn’t whether our controls pass the audit.

It’s:

If a capable adversary attacked us tomorrow, what would actually happen—and what have we changed to make that outcome materially better than it would have been last year?

That’s real risk management, and the question I want my security program to answer.