ai
Onboarding Funnel Watcher: An AI Agent That Explains Funnel Drops
The Onboarding Funnel Watcher is an AI agent for onboarding funnel analysis that explains drops by step and cohort. It runs at read level (L1) for product owners and growth analysts, working from aggregates in a customer data platform (CDP) or product analytics tool. It reads no individual sessions, and it never changes a flow, targets a person or infers sensitive traits.
Rail level: L1 · Risk: low · Function: product and growth
By Art Kukharenko · Published · Updated
All AI agent railsThe AI agent rails method keeps this job narrow: a detector finds the change, the agent explains it, and the product owner decides.
Rail at a glance
Fixed queries and a separate detector bound the Onboarding Funnel Watcher, because it explains numbers it is given and cannot go looking for people.
Scroll horizontally to see all columns.
| Field | Value |
|---|---|
| Job | Watch onboarding funnels and explain drops by step and cohort. |
| Trigger | A schedule, or a detector alert on a funnel step. |
| Data scope | Funnel aggregates by step and cohort, and release notes. No individual sessions. |
| Tool scope | Read: funnel aggregates, release notes, data freshness. Propose: none. Execute: none. |
| Output contract | Step affected, size of change, and likely causes as hypotheses. |
| Gate | No approval applies. The product owner decides any change. |
| Limits | Minimum group size, and fixed queries only. |
| Ledger | Queries, alert, summary version. |
| Evaluation | Past funnel incidents with known causes. |
| Owner and kill switch | Sits with the head of product or growth. Stopping the schedule ends the summaries, and the team returns to its usual funnel reports. |
What it does and who uses it
The Onboarding Funnel Watcher turns a detector's "step 3 is down" into a short note a product owner can act on. It suits teams whose onboarding has many steps, several app versions and more cohorts than anyone checks each morning.
In this illustrative case, a digital bank's account-opening funnel triggers an alert on a Tuesday. All figures are invented.
Scroll horizontally to see all columns.
| Step | Previous four weeks | This week | Note |
|---|---|---|---|
| Application started | 100% | 100% | Entry step |
| Mobile number verified | 88% | 87% | Within normal range |
| ID document uploaded | 71% | 58% | Alert, Android only |
| Account opened | 64% | 52% | Follows the step above |
The note offers three hypotheses. Release notes show a new camera permission prompt in the Android version shipped on Monday, and the fall matches that version. The upload event may also have been renamed in that release, so the drop could be a tracking gap. A paid campaign started last week, yet the fall appears in every channel, which counts against it.
The note says the data was fresh to 07:00 and names the check that would separate the first two: compare upload events with back-end document counts. The product owner asks engineering to run it, then decides whether to roll back.
Why this level
The Onboarding Funnel Watcher stays at L1 because a wrong note wastes an afternoon of investigation and touches no customer. Aggregates and internal release notes keep the Rule of Two count low, with no outsider text, no individual data and no state-changing tools 6.
Detection stays outside the model. A statistical detector or threshold rule raises the alert. Amplitude's Anomaly + Forecast, for example, compares points with an expected range from Prophet, and offers Agile and Robust modes for different metrics 1. The model explains, and its silence never means the funnel is healthy.
L1 is the top of this rail, because rolling back a release or changing a flow is a product and engineering decision with its own change process. Cohort cuts come from an approved list, such as plan, channel, app version and region, so nobody can ask it to infer age, health or other sensitive traits.
Three ways it fails, and the control for each
Most false funnel stories come from the data, not from customers, so the Onboarding Funnel Watcher's first controls run on the queries.
Scroll horizontally to see all columns.
| Failure | What it looks like | Control |
|---|---|---|
| An immature cohort reads as a drop | Recent sign-ups have not had time to finish. Mixpanel's default conversion window is 7 days 2, and PostHog offers "Hide incomplete periods" because open windows pull a trend down 3. | Fixed queries exclude cohorts whose conversion window is still open. Each note states the date its data matures. |
| Broken tracking reads as behaviour | A renamed event or late-arriving data makes a step look abandoned while real completions carry on. | Freshness and event-volume checks run before the model writes. A failed check labels the alert "possible data problem". |
| Mix shift reads as a product fault | A new campaign brings less committed sign-ups. Overall conversion falls while each channel holds steady, a form of Simpson's paradox 4. | The evidence table shows overall and per-cohort figures side by side. A minimum group size removes tiny cohorts, where PostHog warns that correlations may not be accurate 5. |
How to evaluate it
Evaluate the Onboarding Funnel Watcher by replaying past funnel incidents with known causes, along with quiet periods where nothing went wrong.
- Time to notice a drop. Measure from the start of the real change to the note, and compare with when people noticed during the original incident.
- Alert precision. Count real issues as a share of alerts, as defined in the AI marketing operations roadmap. Many steps across many cohorts produce chance alerts, so split the count by source.
- Known cause in the hypotheses. Check whether the true cause appeared and how high it ranked. Tracking faults should carry their own label.
- Seeded traps. Insert a renamed event, an immature cohort and a mix shift. Each should produce its specific label, not a confident story.
- Causal language. Scan for sentences that state a cause as fact, because the output contract allows only hypotheses.
Where it runs
An external model is acceptable for the Onboarding Funnel Watcher where the contract allows, since it sees only aggregates, but regulated teams are better served by a private one. Watch the release notes, which can reveal unannounced features or security fixes, and strip them or keep the model private. The boundary architecture patterns explain both routes. PrivyCDP pairs AI-assisted review of unusual events with bring-your-own-model in the customer environment.
Related
- AI agents for product and growth teams: the playbook, with a recommended adoption order.
- Marketing anomaly detection: the closest worked use case, where the Anomaly Watcher separates detection from explanation in the same way.
- Rail card template: somewhere to write down your approved cohort list.
The AI agent rails hub sets out the full method. For your own funnels, download the rail card template and list the cohorts you will allow.
Browse use casesFunction playbooksAgent roster
FAQ
What can AI onboarding funnel analysis tell you?
It can say which step moved, for which cohorts, by how much, and what changed around that time. It offers likely causes as hypotheses, along with the check that would test each one. It cannot prove a cause from funnel data alone.
Can the agent read session replays of people who dropped off?
No, because it works from aggregates only and has no view of a person's session. If a team needs replays for debugging, that is a separate process with its own access controls and privacy review.
How is this different from anomaly detection in an analytics tool?
The analytics tool finds the unusual number. This agent adds release notes and cohort context, then writes an explanation with hypotheses and data caveats. Keep the statistical detection in the tool or a rule, and do not ask the model to judge what counts as an anomaly.
Sources
Further reading
- AI Agents for Growth Experiments: A
- AI A/B Test Analysis Agent:
- AI Feature Adoption Analysis: Feature
- Agent rails
- AI Agent Permissions Template Download
Published: · Updated:
Editorial dates describe content changes. Confirm current sources before making legal, security or purchasing decisions.