ai
Data Quality Triager: An AI Agent That Sorts Data Failures
The Data Quality Triager is an AI agent that sorts failed data-quality tests on a customer data platform (CDP), ranks them by downstream harm and proposes fixes. Data stewards use it at propose level (L2), and at L3 it applies format-only fixes after approval. It never changes what a field means, deletes a record, or edits consent or identity fields.
Rail level: L2 to L3 · Risk: medium · Function: data and analytics
By Art Kukharenko · Published · Updated
All AI agent railsIts rail belongs to the AI agent rails method. The full workflow is in data quality triage with an AI agent.
Rail at a glance
The rail card below gives the Data Quality Triager two paths: triage, which changes nothing, and a fix path that only a data steward can open. The approval service and the execution service hold that split, not the prompt.
Scroll horizontally to see all columns.
| Field | Value |
|---|---|
| Job | Triage data-quality failures, propose fixes, and apply format-only fixes after approval. |
| Trigger | A failed data-quality test or a data-steward request. |
| Data scope | Test results, schema and masked samples, with no consent fields on the fix path. |
| Tool scope | Read: test results, schema, lineage, masked samples; propose: a ranked triage list and fixes; execute: one approved format-only fix, run by the execution service. |
| Output contract | Failure, likely cause, affected downstream uses, proposed fix and rollback plan. |
| Gate | A data steward approves each fix batch, and the approval binds to the batch hash. |
| Limits | Format-only changes, a maximum batch size, and approved tables only. |
| Ledger | Test ID, proposed fix, approval, rows changed, rollback reference. |
| Evaluation | Historic failures with known causes, plus seeded defects. |
| Owner and kill switch | Head of data or data steward; an Orchestrator pause disables the fix path while triage continues. |
What it does and who uses it
The Data Quality Triager gives a data steward one ranked list instead of a wall of red tests, using test results, lineage and masked samples.
Illustrative run: a nightly load from a loyalty vendor fails three tests. currency_code arrives in lower case on some rows, order_date switches to a slash-separated format, and store_region is empty on others. The triager groups all three under one likely cause, a new export template. It ranks store_region first, because lineage shows a revenue report and an audience sync read that column. It proposes upper-casing the currency codes as a format-only batch with a rollback file. It proposes nothing for the dates, since the vendor's notes do not say whether the day comes first. The steward approves the batch and sends the other two to the source owner.
The hardest call on this rail is where format ends and meaning begins. This illustrative table draws that boundary.
Scroll horizontally to see all columns.
| Proposed change | Class | Why |
|---|---|---|
Trim spaces, or upper-case gbp in a currency column | Format | The value and its meaning stay the same. |
Reorder 03/04/2026 into ISO date order | Meaning, unless the source format is documented | Day and month can swap silently, and the new value still passes the test. |
| Fill an empty country from a phone prefix | Meaning | The fix is an inference, not a reformat. |
| Lower-case an email address | Refused | Email is a match key, so it counts as an identity field. RFC 5321 also treats the local part as case-sensitive, while discouraging reliance on that 2. |
Map Y to true in an opt-in column | Refused | Consent fields sit outside the fix path. |
Why this level
Data Quality Triager proposals carry little risk, because nothing changes until a steward signs. A format fix earns L3 because a pre-image exists, and the table allow-list and batch size cap the blast radius. It stays below L4 because reversal is clean only inside the platform: by the time someone spots a bad batch, a sync may have copied the values out.
The Rule of Two 1 explains the split between the two paths. Masked samples and test messages carry text from source systems, which makes them untrusted input, and the samples still derive from private data. A triage session therefore holds two of the three properties and gets no write tool. The fix runs in a separate execution service with no model in it. That service accepts only an approved batch hash and a transform from an allow-list, and it refuses any column tagged as consent or identity.
New tables start at L2. A table moves to L3 on written evidence from replayed failures and the data owner's sign-off, as the rail levels describe. L3 is the ceiling.
Three ways it fails, and the control for each
Data-quality agents fail most dangerously when tests turn green and the data is still wrong.
Scroll horizontally to see all columns.
| Failure | What it looks like | Control |
|---|---|---|
| Meaning change dressed as format | A date "fix" swaps day and month on part of a table, and the format test passes. | Transforms come from an allow-list with no model in the path. Ambiguous patterns are refused, and the steward sees before and after values. |
| Treating the symptom | The same vendor bug returns weekly, and each approved batch hides it for another week. | Recurrence is counted by test ID. A repeat failure blocks the fix path and goes to the source owner. |
| Ranking by row count | A noisy table with many failures outranks a handful of bad rows feeding a suppression list. | Ranking uses lineage and a written list of critical uses. Test owners set warn and error thresholds 3, and the agent cannot alter them. |
How to evaluate it
Good triage puts the right failure in front of the right person first, and an approved fix stays fixed. Score the Data Quality Triager on historic failures with known causes, plus seeded defects:
- Time to triage, from failed test to an entry a steward can act on.
- Cause accuracy, on whether the known cause is named and related failures are grouped.
- Refusal on meaning traps. Seed ambiguous dates, unit changes and inferred values, and count any proposed as format-only.
- Rollbacks per approved batch, each marking a fix that passed review wrongly.
- Recurring failures, a sign the team is polishing symptoms.
Test the fix path separately by asking the execution service to touch a consent column, which it should refuse. Report results by rail and role, not by named steward.
Where it runs
Masked samples still come from customer records, so a private model is the preferred home for the Data Quality Triager. An external one would need masking and contract cover first, and the boundary architecture patterns set out the placements. PrivyCDP offers AI-assisted review of unusual events, while this rail is a method you write yourself.
Related
- Data quality triage with an AI agent: steps, ledger example and checklist.
- AI agents for data and analytics teams: how the data agents divide the work.
- Schema Drift Sentinel: the structure changes behind failed tests.
- Rail card template: a blank copy of this table.
The AI agent rails hub has the wider method. Download the rail card template to set a format-or-meaning boundary for your own tables.
Browse use casesFunction playbooksAgent roster
FAQ
Can an AI agent fix data quality issues on its own?
Only format-only fixes, and only after a data steward approves the exact batch. Anything that changes meaning, such as filling a missing value, stays with a person.
Why keep consent and identity fields off the fix path?
A small edit there can change who gets contacted or join two people's records. The execution service refuses them even when a batch looks harmless.
What happens when the fix path is paused?
Triage carries on, and failures still reach stewards as a ranked list. Stewards apply urgent fixes through their normal tooling.
Sources
Further reading
- AI Agents for Data Quality
- Data Quality Monitoring AI Agent: Triage
- Draft-approve-execute split — definition
- Agent rails
- AI Agent Permissions Template Download
Published: · Updated:
Editorial dates describe content changes. Confirm current sources before making legal, security or purchasing decisions.