Policy Drift Checker: An AI Agent That Tests Agent Rails Against Reality

In the AI agent rails method, the card is the claim and the ledger is the evidence. This agent tests one against the other, and the compliance, privacy and security playbook places it in the wider function.

Rail at a glance

The Policy Drift Checker's rail allows metadata alone, so it can compare what a run did without seeing what the run touched.

What it does and who uses it

The Policy Drift Checker tests a simple claim: the card says one thing, and the ledger should show the same. Infrastructure teams know the idea from configuration drift, where Terraform compares recorded state with real resources 2.

Illustrative run: a weekly job compares last week's ledger metadata with each rail card in force, and it raises three findings. A Reporting Summariser run called a query ID outside its fixed list, rated high. The Segment Drafter rail shows a new model version with no regression pass on record. The Journey QA Reviewer rail has had no named owner for three weeks. A notification service tells each rail owner, and the checker takes no action. Any demotion comes from written triggers in the control plane, not from this agent.

Most of the comparison is set arithmetic, so code should do it and the model should only explain the result. The mapping below is illustrative.

Why this level

Reading metadata and writing a report leaves production alone, so a wrong finding costs attention and nothing more. Read level (L1) is where the Policy Drift Checker stays.

The Rule of Two 1 count is short. Values in ledger metadata can originate with the agents under review, so treat them as untrusted input. The session holds no payloads and has no way to change state, which leaves one property of three.

It will never gain a pause button. A checker that can stop what it checks puts detection and enforcement in one place, so one bug or hijack could halt every rail at once. OWASP's agentic list names tool misuse and rogue agents among its risks 3, and an independent witness helps expose both. Keep enforcement in the systems that already hold it, and see the rail levels for how demotion triggers work.

Three ways it fails, and the control for each

Drift checks are only as honest as the record they read.

How to evaluate it

Drift detection can only be scored against drift you planted, so seed it deliberately. Write seeded entries to a test ledger, since a production ledger is append-only and should stay free of fake runs.

• Seeded drift found, broken down by out-of-scope call, gate bypass, limit breach and undocumented change.
• Time from drift to report, measured from the seeded event's timestamp.
• False findings at version boundaries. Include runs that straddle a card change, where naive comparisons go wrong.
• Audit coverage, the share of gateway calls with a matching ledger entry.
• Decision age, how long findings wait for an owner, reported per rail.

Someone else should check the checker, because no agent should mark its own work. A person samples its ledger entries against the seeded results.

Where it runs

Ledger metadata carries no customer payloads, so contract terms decide whether an external model is acceptable. Rail cards and tool names still reveal your control design, which favours a private model in regulated teams, as the boundary architecture patterns explain. PrivyCDP provides an immutable audit ledger that names the authorising actor on each AI-assisted action, which gives such a comparison something firm to read, but the checker is not part of the product.

Related

• AI agents for compliance, privacy and security teams: where this checker fits.
• Audit evidence packs with an AI agent: the nearest use case, on the same ledger.
• AI governance policy template: where logging rules are written down.
• Rail card template: the document this agent compares with reality.

Browse the agents this one watches from the AI agent rails hub. Without a written claim there is nothing to test, so start with the rail card template.