Skip to content

The Gate That Caught Its Own Author

I asked what guardrails we have, found half of them were decoration, and then my own audit report made the same mistake it was auditing: claiming a pull request needed approval after the approval had already landed. What it took to turn an advisory checklist into a gate that blocks.

The Gate That Caught Its Own Author

I asked a simple question at the start of this work: what guardrails do we actually have baked in? The inventory that came back was humbling. Some gates blocked. Several others were observe-only, wired to log warnings that nobody reads. One monitor had been false-alarming every fifteen minutes for so long that a real drift would have been indistinguishable from noise. And the register that was supposed to catalog every security check was failing its own enumeration, silently, because nothing in CI ever ran it.

That was the audit. The embarrassing part came later, and it was mine.

The failure the audit was auditing

A pull request sat waiting on a review. I reported it as awaiting the operator's approval. Then I reported it again. And again, across three working rounds. The approval had landed after the first ask. I know because it was in the pull request's review list the whole time, attributed, timestamped, on the exact commit I was describing.

The reason I missed it: I read one convenient field from the API instead of the list that actually matters. That field renders empty in a specific, documented situation, and I had treated empty as absent. When the operator pointed it out, I checked the thing I should have checked in the first place, merged the request, and then did what the discipline demands: swept the session logs for how often this had happened before.

Nineteen confirmed instances in thirty days. One of them, seventeen days earlier, was the same shape: an approval re-requested because the agent read the wrong field. Another was the trap I had just repeated, already written down in the memory bank from a prior investigation. The right procedure existed. It was documented. It was in memory. And none of that mattered, because nothing forced the procedure to run at the moment of the claim.

Why the fix is not a note to self

The obvious remediation is a checklist. Write it down, tell every future session to follow it. That is exactly what I had: a checklist, written into a skill, loaded into context. It failed the same way the memory entry failed, because advice arrives at the same place as every other instruction the model is juggling, and juggling is where claims go unverified.

So the checklist became a gate. One small script produces the authoritative state of any pull request: the review list, whether a reviewer was ever actually requested, the merge state, and the CI verdict, with an honest category for what the credentials cannot see. A ledger records every run. And a trip-wire watches what the agent writes; when a claim about a pull request has no fresh ledger entry behind it, the very next tool call the agent attempts is refused, with the remedy spelled out: run the checker, then restate the claim from its output.

The harness cannot stop an assistant from writing a false sentence. That constraint is measured, not assumed; the extension layer can veto tool calls but not prose. Blocking the next tool call is the strongest enforcement available, and it turns out to be enough: the false claim ships once, and then the agent is forced to reconcile before it can do anything else.

The gate earns its keep immediately

Three catches in the first day, all live, all documented. The gate caught me, because quoting its own error message inside a reply replanted the violation it was reporting. It caught its own builder, because the first version swept the agent's reasoning text as if it were claims and tripped on the builder's thinking about the gate. And when the pull request that carried the fix finally merged, the drift monitor it repaired caught a genuine divergence in the window between the code landing and the reinstall syncing it, and healed it in eight seconds. That last one mattered most: the monitor had only ever been proven against planted faults before, and its first real catch arrived within the hour.

The gate also wedged itself, twice, in ways its tests did not predict. A claim about a pull request that does not exist creates a trip the sanctioned remedy cannot clear, because there is nothing to read. The remedy command was itself a tool call, so the gate denied the very command that would have cleared the trip. Each wedge became a fix with an audited path: an acknowledge command that records what it cleared, an exemption so the remedy is never denied by its own gate, and a rule that a failed read clears the trip while leaving the claim unbacked. Every one of those was found by running the gate against the live session, not by reviewing the code.

What I would tell anyone building this

An advisory checklist is a suggestion. A blocked tool call is a decision. When an agent must not make a class of claim without evidence, write the instrument that produces the evidence, put the veto at the next action the agent takes, make the instrument the only way through, and then prove the gate can fail by letting it catch you. If the first thing your enforcement catches is not your own work, you did not test it honestly.

The deeper finding is about where discipline lives. The correct procedure was documented before this work started, and it failed anyway. Rules in prose compete for attention; rules in a veto do not. The gap between knowing the right procedure and running it is exactly the gap enforcement exists to close, and in an agentic estate that gap is where the quiet failures live.

Written by Todd Miller