Data Quality Triage: Sort and Fix Failures With an AI Agent on Rails

Who this is for: data stewards, analytics engineers and heads of data. This page belongs to the AI agent rails method and the data and analytics playbook.

Before: the manual workflow

The manual triage of data failures starts with a channel full of alerts and one person working out which matter. One late source can fail many downstream tests, so the on-call engineer traces lineage by hand. Then come the slower questions: which audiences use this table, who owns the source, and did this happen last month? Noisy tests teach people to ignore the channel, and the failure that matters looks like all the others.

After: the job on rails, step by step

The triage job on rails separates reading from acting, so the session that reads samples can never write.

1. Trigger. A data-quality test fails, or a data steward raises a request.
2. Group. A rule uses lineage to group failures that share an upstream model.
3. Read. The agent reads test results, schema, lineage and masked samples.
4. Classify. It labels the likely class: data, pipeline, test or real-world change. A seasonal spike needs no fix at all.
5. Rank. A rule scores downstream use, and the agent builds the ranked list. Anything on a consent or identity path goes straight to its owner.
6. Propose. It writes the failure, likely cause, affected uses, proposed fix and rollback plan, stating causes as hypotheses.
7. Approve. For a format-only fix, a data steward approves one batch. The approver is never the requester. Approval binds to the batch hash, and a timeout expires it.
8. Execute. A separate execution service runs the allow-listed transform with no model in the loop, stores before values and reruns the test.
9. Log. A system outside the agent records the test, proposal, approval, rows changed and rollback reference.

Anything else leaves the rail as a ticket for the owner. Schema changes go to the pipeline owner, usually with an explanation from the Schema Drift Sentinel. The L3 path is optional: many teams should stay at L2 and let engineers change code.

What counts as a format-only fix?

A format-only fix changes how a value is written and leaves what it means untouched, and a machine must be able to prove that. Keep an allow-list of named transforms, each with its own test. The examples are illustrative, with the traps on the right.

Identity keys stay out even when a change looks harmless. Lower-casing an email address is a good example. RFC 5321 says the local part of a mailbox must be treated as case sensitive, while discouraging anyone from relying on it 8. Changing a stored key can create or break a merge, so that decision belongs to identity review. Consent fields stay out because a wrong value can mean unlawful contact, as the suppression architecture explains.

Four checks prove a batch is value-preserving, and the execution service runs them with no model involved:

1. Round trip. Parsing the before and after values gives the same canonical value.
2. Distinct count. The number of distinct canonical values does not change.
3. Idempotence. Running the transform twice gives the same result as running it once.
4. Rollback. Before values are stored, and a drill has restored them.

What rules do and what the model does

Rules and tests decide whether data failed and what may change, while the model reads context and suggests why. A model's view never clears a failed test, and its silence does not mean the data is correct.

Statistical root-cause tools show the limit. Anomalo describes comparing failing and passing rows across segments to find where failures concentrate 5. That shows where, not why, and a language model's explanation is no more certain.

The rail card

The rail card below fixes what the Data Quality Triager may do, and systems enforce each limit, so a prompt cannot widen it.

The level reflects reversibility and blast radius, as the rail levels explain. A format fix on one approved table, with stored before values, has a small blast radius, so it may reach L3. It does not go to L4 here, because deciding that a change is format-only is exactly where mistakes hide. The Rule of Two sets the architecture 9. Masked samples can hold text typed by customers, which is untrusted, and they remain sensitive. So the reading session gets no state-changing tool. A private model is preferred, and an external one needs masking and contract cover, as the boundary patterns describe.

What the ledger shows for one run

The ledger entry for one run traces a fix from failed test to rollback reference. The JSON is illustrative: its fields and values are invented, and it is not a product API.

{
  "rail": "data-quality-triager",
  "correlation_id": "c-7f3a",
  "test_id": "customers.country_code_format",
  "failure_class": "data",
  "cause_hypothesis": "source export added trailing spaces",
  "proposed_fix": {"transform": "trim_spaces", "catalogue_version": "3", "batch_hash": "sha256:91ab..."},
  "approval": {"by_role": "data steward", "bound_to": "sha256:91ab...", "expires": "2027-01-12T17:00:00Z"},
  "execution": {"service": "fix-runner", "rows_changed": 1840, "retest": "pass", "rollback_ref": "rb-5521"},
  "model_version": "model-v3"
}

Three ways it fails, and the control for each

The Data Quality Triager fails in ways specific to fixing data, and each control below sits outside the model.

Metrics that are honest

Honest metrics for data triage measure whether causes get found and failures stop returning, using the roadmap's definitions where they exist. Baseline them before the agent starts.

• Time to triage: failed test to accepted ranking. Fast triage of the wrong cause is worse than slow triage.
• Recurring failures: the same test failing again after a fix, which is the best sign of symptom-fixing.
• Rollbacks: fix batches undone. Zero may mean nobody checks.
• Cause accuracy on historic failures, where the real cause is known.
• Alert precision, counted after grouping by root cause.
• Override rate. A rate near zero may mean rubber-stamping.
• Audit coverage. Anything below 100% is a defect.

Report by rail and role, not by named steward. Annex III point 4(b) of the EU AI Act lists monitoring workers' performance as a high-risk use 10, so take advice before building a per-person view.

How vendor tools compare

Data observability tools are often the better answer for triage, because they already hold lineage, history and alert routing. The table records each vendor's page, checked on 1 November 2026. "Not stated" means the page opened did not say, not that the feature is absent.

None of the pages opened describes an approved write path for fixing customer data. That gap is the main reason to build this rail, and many teams will not need it. If an engineer can fix format problems in transformation code, that is the better answer, because the fix lives upstream under code review. A script can group failures by upstream model with no model at all. Buy an observability tool when samples may leave under your contract, and choose a private rail when they may not. Doing nothing new is fair when failures are rare.

PrivyCDP's AI layer offers limits on data access and review of results, and its immutable audit ledger records each AI-assisted action with the human or system actor that authorised it.

Checklist

Use this checklist before the agent reads production samples.

• ☐ Every test has a named owner and a severity.
• ☐ A rule lists critical downstream uses, including consent and identity paths.
• ☐ Samples are masked before the session reads them.
• ☐ Consent and identity fields sit outside every fix path.
• ☐ Format transforms are allow-listed, and value-preserving checks run in the execution service.
• ☐ Approval binds to a batch hash, and the approver is not the requester.
• ☐ Before values are stored, and a rollback drill has passed.
• ☐ A repeat fix on the same test needs a source ticket.
• ☐ Historic failures and seeded defects are rerun on any model, prompt or tool change.

Back to the AI agent rails hub. To write this rail for your own team, download the rail card template.