Failed Sync Retry: Recover Reverse ETL Failures With an AI Agent on Rails

Who this is for: platform leads and data engineers who run audience syncs under an approval process. This job is one rail in the vendor-neutral AI agent rails method. Tool behaviour below comes from vendor documentation read on 1 November 2026.

Before: the manual workflow

The manual workflow for a failed sync is an on-call engineer reading logs and deciding whether to press retry. An alert fires, or a marketer notices a low count. The engineer opens the run, guesses whether the error is transient, and re-runs it or forces a full resync. Nobody checks whether the audience has changed since it was approved. Duplicates appear days later as a complaint, or not at all.

Reverse ETL tools already retry a lot of this themselves. So the human question is narrower than it looks: what happens when the built-in retries stop, and does a re-run still match what someone approved?

When is a retry not a new activation?

A retry is not a new activation only when nothing a person approved has changed. The rule has four tests: same approved version, same audience hash, same destination, and consent re-checked. This table turns it into decisions.

Consent can only shrink a retry. Anything that grows the set voids the approval, because approval binds to a hash of the exact payload. This needs a frozen copy of the approved audience. If the sync recomputes from a live model, you cannot prove the audience is the same, so point it at a versioned snapshot table instead.

Why do retries cause duplicate deliveries?

Retries cause duplicates because the caller often cannot tell whether the first attempt worked. AWS's engineering guidance describes the core case: when a response is lost, the caller does not know the outcome, and a blind retry can repeat the effect 1. Batches make it worse. Airbyte states that it delivers at least once, and that a partly completed chunk may be re-sent on the next attempt 2. Hightouch and Fivetran Activations can both split a rejected batch and retry the parts 34.

In a warehouse, a duplicate is usually a clean-up task. In a destination, it can reach a person. So the destination's operation sets the highest level a retry may run at. The levels below are our recommendation.

An idempotency key closes the lost-response gap where the destination supports one. The same key goes on every attempt. AWS says a known token with different parameters should return a validation error, which is what you want if a payload has drifted 1. Stripe's documentation, a useful reference pattern though not a marketing destination, advises against personal data in keys and says keys can be pruned once at least 24 hours old 8. The lesson is general: the retry window must be shorter than the destination's key or dedupe window.

After: the job on rails, step by step

The failed sync job on rails puts a rules engine on the action path and a model on the explanation.

1. A failed sync event arrives. Include partial failures. Hightouch ends a run with rejected rows as "Completed with errors", not "Failed" 3, so an alert on failure alone misses them.
2. Rules classify the error. Timeouts, 5xx and 429 are transient. Authentication and validation errors are not, as both Hightouch and Fivetran's connector guidance treat them 39. The agent holds no keys, so a 401 goes to on-call.
3. Rules compare against the approval. They check approval hash, snapshot hash, destination, mapping version, operation type, retry count and window.
4. The consent service runs a fresh check. Withdrawn people drop out of the set, using the same enforcement as a send-time check. The check ID is stored.
5. The execution service retries. It reuses the key, backs off with jitter, and honours Retry-After 4. At L3 it waits for on-call to approve. At L4 it proceeds.
6. Rules reconcile. Accepted counts are compared with the expected set. A mismatch stops further retries.
7. The model explains. It writes the likely cause for on-call and proposes a fix for anything that is not a retry. It holds no retry tool.

Promotion from L3 to L4 needs written evidence and the platform lead's sign-off. Replay past failures in a test destination and find no duplicates. Show that a changed hash blocks the retry and that a withdrawal removes the person. Drill the kill switch. Any breach drops the rail back automatically, as the rail levels require.

L4 also expects reversibility, which limits this rail honestly. A retried list add can be removed using the approved sync's rollback reference. A sent message cannot. So on a failed reconciliation, retries stop and the rail falls to L3, and on-call decides on any removal. Destinations that cannot be reversed never run at L4.

What rules do and what the model does

Rules make every retry decision here, and the model only writes the explanation. Error text from a destination is outside input, and it could contain instructions. The Rule of Two says a session should not hold untrusted input, sensitive access and state-changing tools together 10. So the model reads logs without payload data and has no tools that change state. The action path has no model in it, and a hijacked explanation cannot trigger a retry.

The rail card

The rail card below fixes what Sync Medic may do. Retry limits and windows are yours to set, within the dedupe window above.

The first sync still belongs to the Audience Sync Steward, with a named approver bound to the segment hash, as in the segmentation walkthrough. Sync Medic only repeats what that approval covered.

What the ledger shows for one run

The ledger entry for one retry shows that nothing approved has changed. This example is illustrative, with invented values and no personal data.

{
  "sync_id": "sync_4471",
  "rail": "sync-medic@v2",
  "level": "L4",
  "error_class": "destination_5xx",
  "approval_hash": "9f2c...",
  "audience_hash": "71de...",
  "destination": "ad_platform_a",
  "mapping_version": "v12",
  "retry": {"count": 2, "limit": 3, "key_ref": "idem_8841"},
  "consent_check_id": "cc_7712",
  "removed_by_consent": 14,
  "reconciliation": "match",
  "outcome": "accepted"
}

Three ways it fails, and the control for each

Each failure below is a known pipeline problem, and each control sits outside the model.

Metrics that are honest

Honest retry metrics count harm as well as speed.

• Mean time to recover, from failure to reconciled delivery.
• Duplicate deliveries, target zero. Measure them in seeded replays and through reconciliation, because production duplicates are easy to miss.
• Failed retries, which show whether error classification is right.
• Retries refused by rule, split by reason. A rise may show approvals expiring too fast.
• Audit coverage. Any retry without a complete record is a defect.

How vendor tools compare

Reverse ETL and pipeline tools handle retries well, and their defaults are often all you need. We read these pages on 1 November 2026. They do not address approval binding, which is not their job.

One detail matters for approvals. Hightouch compares current model results with the previous run 12, so a later run may carry new rows alongside the retried ones. As we read it, that is closer to a new run than to a retry of what was approved. A frozen snapshot table avoids the question.

A vendor's retry plus an alert is enough when the destination upserts, the audience has no approval bound to it, and the data is low-risk. Where approval does apply, a script can do everything on the action path, because that path has no model anyway. Add the model only if on-call wants a written diagnosis. PrivyCDP's immutable audit ledger records each AI-assisted action with the human or system actor that authorised it; Sync Medic itself is a method you would build, not a shipped feature. The rule on approval gates still governs anything outside the bounds.

Checklist

• ☐ Alerts fire on partial failure as well as failed runs.
• ☐ Syncs read a frozen snapshot, versioned by audience hash.
• ☐ Rules classify errors, and authentication failures go to on-call.
• ☐ Hash, version, destination and mapping are compared before each attempt.
• ☐ A fresh consent check runs on every attempt, and it can only shrink the set.
• ☐ Each destination operation has a written highest level.
• ☐ Keys are reused across attempts and contain no personal data.
• ☐ The retry window is shorter than the key or dedupe window.
• ☐ One layer owns retries.
• ☐ Reconciliation follows every retry, and a mismatch stops the rail.

The wider function is in the platform and data pipeline operations playbook, and the method is on the AI agent rails hub. To write this rail for your team, download the rail card template.