Paul Mozaffari

The 2:00 AM Change Window Fallacy: Why AI Safety Breaks in Production

At 2:00 AM in a core switch migration or firewall cutover, not knowing the how is negligence. The step-by-step rollback script is written before the window opens, and if it isn’t, the Change Advisory Board (CAB) cancels the change. On deterministic infrastructure, demanding that certainty is what competence looks like.

One that still sticks with me was a firewall migration where a contractor proposed removing the old static routes immediately after the new routing policy came up. The rollback plan required the legacy routes to remain in place until end-to-end validation completed. Five minutes after cutover, asymmetric routing started blackholing traffic between data centres. Because the rollback procedure had been pre-written, we restored the original route set in under a minute and kept the outage to a small validation window instead of a major incident.

The mistake is carrying that playbook unchanged into agentic AI. Governance teams ask probabilistic systems for pre-execution certainty they cannot give, write policy to stand in for it, and hold deployments until the paper is signed. Two things happen at once. Approved teams stall, and unapproved agents reach production anyway with no runtime containment at all.

In the change window, demanding the rollback plan is competence. With agents, treating policy as a firewall is the illusion of control.


Why Paper Guardrails Fail Open

A failed API call throws an exception. The monitor fires and traffic moves to the failover pool. An agent that meets an injected instruction throws nothing. It carries on, completes the task it was steered toward, and reports success in fluent prose. The failure looks like normal operation, so nothing downstream treats it as a failure.

On 2 September, CISA added CVE-2026-59822 to its Known Exploited Vulnerabilities catalog. It is reportedly the first Model Context Protocol (MCP) implementation to land there. The flaw sits in LiteLLM’s MCP Streamable HTTP endpoint (CVSS 8.8, fixed in 1.84.0). When key validation failed, an OAuth2 passthrough fallback swapped the rejected authentication context for an empty one instead of terminating the request. A fabricated bearer token was enough to reach the MCP tools behind the gateway.

The gateway did not fail closed. It substituted a default and carried on.

An ISO/IEC 42001 certificate would not have stopped that, and neither would a signed acceptable use policy. The failure happened in a code path, at runtime, long after the paperwork was approved. Policy does not inspect headers.


Deterministic Controls Around a Non-Deterministic Core

A system prompt is a request, not a boundary. The model will usually follow it, and usually is not a security property.

The controls that hold are the ones the model cannot talk its way past, because they do not run on the model. None of them asks a model to certify its own output. The auditor may not be the author.

UNTRUSTED CONTENT  (user input, retrieved documents, tool output)
        │   treated as data, never as instruction
        ▼
[ LLM ]  ──►  proposed action, in a fixed schema
        │
        ▼
[ POLICY ENGINE ]  deterministic rules, outside the context window
        │
        ├── within policy and reversible ──►  scoped, single-use credential ──►  production
        ├── outside policy or irreversible ─►  held for human approval
        └── validation fails ──────────────►  request terminated

TheThree Runtime Boundaries

  1. Ingress: content is data. Nothing the agent reads may grant it authority. Classifiers that try to spot injected instructions are useful telemetry, but they are a model judging text, so they cannot be the control. The control is that authority arrives only on a channel content cannot write to: the policy engine, an operator console, a credential issued outside the context window.
  2. Execution: no standing keys. Agents never hold persistent admin credentials. A proposed action goes to the policy engine, which issues a scoped, time-limited, single-use credential for that action alone. The engine is code, not a model.
  3. Reversibility: autonomy follows undo. Grant autonomy based on your ability to reverse the action, not on the model’s benchmark accuracy. If you cannot undo it automatically, a human approves it first.

Five Questions For Your Engineering Leads

The OAIC’s automated decision-making transparency obligations under the Privacy Act take effect on 10 December, and SOCI Act risk management obligations already apply to critical infrastructure operators. These are the questions that decide whether you can answer a regulator with evidence rather than policy.

  1. Do any agents hold static, shared secrets? If compromising one agent yields persistent write access to a database, there is no execution boundary.
  2. Does your telemetry record what agents intended, or only that requests succeeded? A log full of HTTP 200s says nothing about what the agent was steered to do.
  3. When validation fails anywhere in the agent path, does the request die? Or does something substitute a default and carry on?
  4. Can you reconstruct any agent decision after the fact? That means the inputs it saw, the action it proposed, and the rule that allowed it.
  5. Can a compromised agent change production state without a human? If yes, which actions, and can each one be undone automatically?

The Rule

Never grant an agent more write privilege than your fastest automated rollback can undo.

If you cannot undo an action at machine speed, do not let an agent execute it at machine speed. Models will not become predictable enough to make this unnecessary. The containment has to be deterministic because the thing it contains never will be.

Pick one agent you have approved for production. What is the worst action it can take today, and has anyone tested undoing it?


Part of the AI Security collection. Related: The Reversibility Test · The Zero-Trust Agent · AI Safety Has Never Worked a Change Window.


I’m Paul Mozaffari. I have spent 28 years designing and securing enterprise network and security infrastructure across government and private sectors, now focused entirely on the AI layer. If your organisation is putting agentic AI into production and you want experienced, production-scarred eyes on your specific architecture, threat surface, and runtime boundaries, I run private working sessions for executive and technical leadership teams. Book a 20-minute scoping call.

#AI Security #Architecture #Enterprise #Governance