ai
Metric Explainer: An AI Agent That Explains Why a Number Moved
The Metric Explainer is an AI agent that explains why a number moved on a customer data platform (CDP), using lineage, approved metric definitions and aggregates above a minimum group size. Analysts and data consumers read its answers at L1, the read level. It never changes a metric definition, never states a cause as fact and never reads individual records.
Rail level: L1 · Risk: low · Function: data and analytics
By Art Kukharenko · Published · Updated
All AI agent railsUnder the AI agent rails method, it answers one narrow question well. The data and analytics playbook shows where it fits beside the team's other agents.
Rail at a glance
The Metric Explainer's rail gives it fixed queries and an approved catalogue, so it cannot write its own SQL or reach a single record. The data view and the query allow-list hold those limits.
Scroll horizontally to see all columns.
| Field | Value |
|---|---|
| Job | Explain why a metric moved, using lineage and approved definitions. |
| Trigger | A question from a data consumer or an alert. |
| Data scope | Metric catalogue, lineage and aggregates above a minimum group size. |
| Tool scope | Read: metric catalogue, lineage, fixed aggregate queries; propose: nothing; execute: nothing. |
| Output contract | Plain explanation, contributing factors as hypotheses, query references. |
| Gate | None for answers, though the metric owner reviews a sample. |
| Limits | A minimum group size, and fixed queries only. |
| Ledger | Question hash, queries, answer version, model version. |
| Evaluation | Historic metric moves with known causes. |
| Owner and kill switch | Analytics lead; disabling the agent leaves the metric catalogue available. |
What it does and who uses it
The Metric Explainer answers the question analysts hear every week: why did this number move? It differs from the Reporting Summariser, which writes scheduled readouts, and the Anomaly Watcher, which responds to alerts on campaigns. It follows a set order of checks, because the dullest explanation is often the true one.
- Definition. Did the metric's version change? A shared definition layer such as the dbt Semantic Layer makes this answerable 3.
- Freshness. Did a load arrive late, or did a backfill restate earlier periods?
- Instrumentation. Does lineage show a tag, event or source change in the window?
- Mix. Did the share of each subgroup change while rates within them held? Simpson's paradox is the extreme case, where a trend reverses once a population is split 2.
- Outside events. What lines up in the calendar, such as a campaign or price change? These are always hypotheses.
Illustrative run: a head of e-commerce asks why the repeat-purchase rate fell two points last week. The explainer finds that metric version 7 went live on Thursday and excludes marketplace orders. Recomputed on version 6, most of the fall disappears. The remainder matches a promotion that brought in more first-time buyers, while rates within each cohort stayed flat. The answer lists both as hypotheses with query references, and it says pricing was not tested. The metric owner samples the answer later that week.
Why this level
The Metric Explainer stays at read level (L1) because its only output is an explanation that a person checks. No data changes, so reversal is not the concern. The blast radius is a decision made on a wrong story, such as cutting a budget after what was really a tracking bug.
The Rule of Two 1 is easy to satisfy here. The question is free text from a data consumer, so it counts as untrusted input. Aggregates above the group-size floor are designed not to reveal private data, and the agent holds no state-changing tool. That is one property of the three.
It has no promotion path. Proposing a definition change is the metric owner's job, and an agent that could edit definitions could make any number look better. A team that wants drafted commentary should use a different rail, with its own rail level and reviewer.
Three ways it fails, and the control for each
Metric explanations fail by sounding reasonable, so each control below makes a weak answer visible or impossible.
Scroll horizontally to see all columns.
| Failure | What it looks like | Control |
|---|---|---|
| Coincidence told as cause | A campaign launched the same week, so the answer says it drove the fall. | The output schema has a hypotheses field and no cause field. Each factor needs a query reference, and "cannot tell from this data" is an accepted answer. |
| Totals hide a reversal | The overall rate falls while each cohort's rate rises, and the answer blames every cohort. | A fixed mix query runs across catalogue dimensions before any text is written. A number check compares the text with the results. |
| Slicing past the group-size floor | Two nearly identical questions differ by one person, and subtraction reveals that person's value. | Free SQL is off the tool list. The floor applies at the data view, with further suppression on related cells, since one suppression often forces others 4. Repeated near-identical queries raise a flag. |
How to evaluate it
The test set for a metric agent is historic moves with known causes, and it should include boring ones. Add tracking bugs, backfills and definition changes alongside real shifts in customer behaviour.
- Accuracy on known cases: was the true cause listed, and did it rank above coincidences?
- Order discipline: did the answer rule out definition and freshness before offering a story?
- Share of answers with evidence, counting each claim that carries a query reference.
- Honest abstention. Seed questions the data cannot answer, where "cannot tell" is the only correct response.
- Number checks, run automatically on every figure in every answer.
- Owner sample, tracking how often the metric owner disagrees and why.
A new model, prompt or query means a fresh run of the whole set.
Where it runs
Aggregates leave room for an external model where contract terms allow it, though regulated teams may prefer a private one; the on-premise LLM guide explains when that cost is justified. PrivyCDP includes AI-assisted review of reports, whereas this rail stays a vendor-neutral method for you to write.
Related
- AI agents for data and analytics teams: the playbook this agent belongs to.
- Weekly summaries from an AI agent: the closest use case, which shares the number checks.
- Explainable AI for customer decisions: a separate topic, about explaining a decision on one person.
- Rail card template: where to record the fixed-query list.
See the AI agent rails hub for the full method, then download the rail card template to write down which questions this agent may answer.
Browse use casesFunction playbooksAgent roster
FAQ
Can AI work out why a metric changed?
It can test a set list of explanations quickly and show evidence for each. It can rarely prove a cause from observational aggregates, so its answers are hypotheses. An analyst or an experiment settles the question when the decision matters.
Why not let the agent write its own SQL?
Free queries can wander around a minimum group size, join tables nobody approved and produce figures no one can reproduce. Fixed queries are narrower, but each result is reviewed, repeatable and safe to cite.
Who owns a metric's definition?
A named metric owner, recorded in the catalogue. The agent reads definitions and reports when one changed. If a definition looks wrong, it says so in the answer, and the owner decides what happens through normal change control.
What should the agent say when it does not know?
"Cannot tell from this data", followed by what was checked and what would settle the question. That answer counts as a pass in evaluation, because a confident guess can cost far more than an admitted gap.
Sources
Further reading
- AI Agents for Data Quality
- AI Agent for Data Quality Triage: Data
- AI Identity Resolution Review Agent
- Agent rails
- AI Agent Permissions Template Download
Published: · Updated:
Editorial dates describe content changes. Confirm current sources before making legal, security or purchasing decisions.