Audit Evidence Packs: Answer Auditors With an AI Agent on Rails

Who this is for: compliance leads, DPOs and internal auditors who get record-level questions about customer data. This job is one rail in the vendor-neutral AI agent rails method. Take advice on what your auditor or regulator will accept.

Before: the manual workflow

The manual evidence workflow starts with an email and ends with a document nobody can fully check. A question arrives from an auditor. Compliance asks the data team for exports, the marketing team for screenshots and the platform team for logs. Someone writes a narrative from what came back, and time zones and identifiers rarely line up.

Two problems recur. The narrative is written from memory of how the process should work. And gaps appear late, sometimes first noticed by the auditor. Tool logs also expire quickly: Hightouch keeps row-level debugger logs for seven days and run summaries for 30 1, and Fivetran Activations keeps Sync Tracking for 14 days in its interface 2. A question about last spring cannot be answered from those screens, so evidence must live in your own ledger.

What counts as audit evidence for an AI agent?

Audit evidence for an AI agent is a record that bears on the question and that a third party can check. ISO 19011:2018 describes audit evidence as relevant to the audit criteria and "verifiable" 3. A model's account of what happened fails that second test unless each statement points to a record. The questions auditors actually ask are concrete: what was decided, on which data and version, who approved it, and whether you can reproduce it.

Several rules put the burden on you, without saying what a pack must look like. GDPR Art. 5(2) requires controllers to be able to demonstrate compliance, Art. 7(1) requires them to demonstrate consent, and Art. 58(1) lets authorities order information and obtain access 456. For high-risk AI systems, EU AI Act Art. 26(6) requires deployers to keep the logs they control for at least six months unless other law says otherwise 7. Annex III obligations apply from 2 December 2027, and marketing segmentation is not high-risk per se. Retention beyond that floor is a decision for your compliance team.

The table maps common questions to sources. It also shows what each source cannot prove, which is what a good pack says out loud.

After: the job on rails, step by step

The evidence job on rails starts from a logged question and ends with a person's release.

1. Compliance logs the question. Period, population and rule go on record. The agent never acts directly on an auditor's email.
2. The agent reads (L1). It reads rail cards, policy versions and ledger views, and returns what exists, with references. For an internal question, this read-only answer may be all that is needed.
3. Rules test coverage. A query compares what should exist with what does, such as sends in the period against consent-check IDs. The result feeds the gaps section.
4. Rules test integrity. The chain is verified, and each cited entry's hash is recomputed.
5. The agent drafts the pack (L2). It writes the question, an evidence index, statements with ledger references, and gaps stated plainly.
6. Rules test the draft. Every reference must resolve, every figure must match its source, and uncited sentences or conclusion words such as "compliant" are rejected.
7. The compliance lead reviews and releases. Release binds to the pack hash. Counsel reads anything going to a regulator.
8. A separate service delivers it, and the ledger records entries cited, pack version and releaser.
9. Lessons feed back. A returned pack or a found gap becomes a test case and, where needed, a remediation owned by the rail owner.

What rules do and what the model does

Rules establish the facts in an evidence pack, and the model arranges them in readable form. This split follows one principle: rules enforce and models explain. A model's silence about a period does not mean that period is clean.

The Rule of Two also applies 8. An auditor's question is outside text, and past agent runs can contain injected instructions. The session reads that text and sees sensitive records, so it must not hold the third property, the ability to change state or send. Delivery runs as a separate service, and payloads stay out of the session where a reference will do.

How should an evidence pack state a gap?

An evidence pack should state each gap as a finding with a cause, a period and an effect on the answer. Absence of a record is not proof that nothing happened, and the pack must not imply it. A pack with an empty gaps section needs one line saying which coverage checks ran. The taxonomy below is ours, and the wording is illustrative.

Here is an illustrative answer. Asked whether every send in a quarter passed a consent check, the pack gives the count matched and names one destination with three weeks of missing check IDs. It says when the control began and what the data cannot show. The compliance lead adds the remediation and releases. Auditors tend to trust a pack more when it finds its own holes, though that is our judgement and not a measured result.

The rail card

The rail card below fixes what the Evidence Packer may do, and data views and tools enforce it.

The level is L1 for a read-only answer inside the team, and L2 when the agent drafts something meant for release. It never reaches L3, because release is a judgement about what to tell a regulator and cannot be undone.

What the ledger shows for one run

The ledger entry for one pack lets an auditor test the pack itself. This example is illustrative, with invented values.

{
  "question_hash": "c41e...",
  "scope": {"period": "<quarter>", "rule": "consent_check_on_send"},
  "checks": {"coverage_id": "cov_0092", "chain_verified": true, "citations_resolved": true},
  "entries_cited": 412,
  "gaps": [{"type": "never_logged", "destination": "ad_platform_b"}],
  "pack": {"version": 3, "hash": "9ad0..."},
  "released_by": {"role": "compliance_lead", "actor_ref": "usr_204"},
  "model": "model-id@version"
}

Three ways it fails, and the control for each

Each failure below comes from how evidence work goes wrong, and each control sits outside the model.

The last control needs care. Approval records can tempt a team to compare individual reviewers. AI Act Annex III point 4(b) lists monitoring and evaluating workers' performance as a high-risk employment use 9, so take advice before reusing the ledger that way.

Metrics that are honest

Honest evidence metrics show whether packs survive contact with an auditor, using definitions from the AI marketing operations roadmap where they apply.

• Time to answer, from logged question to release.
• Packs returned by auditors, with the reason recorded.
• Gaps found before an audit, compared with gaps an auditor found first.
• Seeded gaps reported, by gap type.
• Audit coverage: the share of AI-assisted actions with a complete ledger record. Anything below 100% is a defect.

How vendor tools compare

Compliance automation tools collect control evidence well, and for a SOC 2 or ISO audit they are usually the right answer. They mostly answer "is this control operating?", while this rail answers "what happened to this person or this send?". We read these pages on 1 November 2026.

A script or saved query is better than an agent when the same question recurs in the same form. Keep the coverage and integrity checks and drop the model. The agent earns its place when questions vary and the narrative takes real time. PrivyCDP's immutable audit ledger records each AI-assisted action with the human or system actor that authorised it. The Evidence Packer is a method you would build on a ledger like that, not a shipped feature. The boundary architecture patterns list the fields worth logging from day one, and withdrawal events come from your consent architecture.

Checklist

• ☐ Questions are logged with period, population and rule before the agent runs.
• ☐ The agent reads through fixed, read-only views.
• ☐ Coverage and integrity checks run outside the model.
• ☐ Every statement cites an entry, and broken citations fail the draft.
• ☐ A gaps section is mandatory and uses a fixed taxonomy.
• ☐ Conclusion words are rejected by rule.
• ☐ Release binds to the pack hash, and only the compliance lead releases.
• ☐ Delivery is a separate service.
• ☐ Reports use rails and roles, not named staff.
• ☐ Seeded gaps run before any model, prompt or tool change.

Evidence packs often follow an access request, and the wider function is in the compliance, privacy and security playbook. The method is on the AI agent rails hub. To write this rail for your team, download the rail card template.