How to Run a Customer Data Platform Proof of Concept With Synthetic Data

A customer data platform (CDP) proof of concept should test the hard behaviours, not the happy path: schema enforcement, duplicate and late events, identity merges with opt-outs, delivery to a real destination with retries, an audit trace of one decision and, for private deployments, a restore. Run it for four to six weeks on synthetic data so no production personal data reaches a vendor sandbox, and score it against pass criteria written before kickoff.

Who this is for: architects, security leads and procurement teams running the proof-of-concept (PoC) gate of a vendor selection. It follows the CDP RFP template; cost is covered in the 3-year total cost of ownership model. More in the guides hub.

Why use synthetic data in a customer data platform PoC

Synthetic is not automatically anonymous. The UK Information Commissioner's Office says synthetic data modelled on personal data may still reveal it if the information can be inferred, and names model inversion, membership inference and attribute disclosure as known vulnerabilities 2. For a PoC, generate from rules and schemas rather than training a generator on production tables, unless your privacy team signs off.

Limit. Synthetic data will not reproduce the dirty formats and odd joins of real systems. After the PoC, run a short shadow phase on a small approved slice inside your own boundary.

How to generate synthetic data

Recipe:

1. Write the event contract in a schema language such as JSON Schema 7 and list identity keys: email, phone, customer ID, device ID, cookie ID.
2. Generate profiles with a fixed seed. Use reserved domains such as example.com and example.org 8 and fictional phone ranges so nothing can reach a real person.
3. Generate event streams with realistic shape: daily cycles, bursts, bot traffic, several time zones, sized to your peak events per second.
4. Inject labelled fault cohorts. Suggested starting point: 2% duplicates (half within an hour, half after 48 hours), 1% late events (24 and 72 hours), 0.5% malformed, 1% on a drifted schema version, 1,000 identity pairs with conflicting attributes, and 500 profiles with consent withdrawals.
5. Write a ground-truth file with the expected outcome for every injected fault, so scoring is arithmetic, not opinion.
6. Version the generator, seed and dataset hash; give every vendor the same files.
7. Never generate realistic payment-card, national-ID or medical-record numbers; use obviously fictional patterns.

The acceptance workflow: six tests

Test 1: ingest with an explicit contract

Register the schema, then load the dataset including malformed events and the drifted version. Pass: every malformed event is quarantined with a reason code, none reaches a profile, and corrected events replay once.

Test 2: duplicate, late and failed events

Send duplicates inside and outside the vendor's deduplication window, late events, and a burst during which the endpoint returns errors. Get the key and window in writing. For example, Twilio Segment documents deduplication on messageId within an approximately 24-hour look-back window, described as 99% effective, and cannot deduplicate events that have no messageId 9. Pass: input equals processed plus quarantined plus dropped-with-reason, with zero silent drops, and behaviour outside the window matches the documentation.

Test 3: identity merge with opt-out

Load the conflicting pairs and the consent cohort. Pass: expected merges happen, conflicts follow the configured rule, an opted-out profile that merges with an opted-in one stays suppressed, and unmerge restores the previous state. Background: identity resolution explained and consent withdrawal and suppression architecture.

Test 4: activation to one real destination, with retries

Connect one destination you control, such as a sandbox account or an HTTP endpoint. Put a fault-injecting stub in front of it, for example WireMock, which supports fault and latency injection 10, and fail it for ten minutes. Pass: every eligible message arrives after recovery, retries and dead letters are visible, any duplicates carry the same idempotency key, and no message reaches a withdrawn profile.

Test 5: audit trace of one decision

Pick three sends at random and ask the vendor to trace each in front of you. Pass: inputs, rule or model version, consent state at send time, destination and timestamp are reconstructed in under five minutes each. Then try to alter the audit log; it should refuse or show tampering.

Test 6: restore test for private deployments

Take a backup, destroy a copy of the environment, restore into a clean one, and compare record counts, consent states and audit continuity. Pass: restored within the agreed recovery time, counts and consent checksums match, and keys held in your key management system worked as designed. NIST's contingency-planning guide is a standard reference for recovery planning 11. For SaaS, ask for the vendor's latest restore evidence and run an export instead (see the cloud exit strategy template).

Optional: load. Replay the stream at 2x your peak. Grafana k6 describes smoke, load, stress, spike and soak test types 12. Method: peak-load activation test plan.

Success criteria and scoring

Score each test 0–5 using the scale in the RFP template (0 absent, 3 meets with limits, 5 meets and is independently evidenced). A vendor passes if every gate scores at least 3 and the weighted total reaches 4.0 out of 5. Weights and thresholds are suggestions; fix yours before kickoff.

Two evaluators score independently; the vendor may re-run a failed test once; log every defect with severity and owner.

Timeline: four to six weeks

A four-week version runs tests 1–3 in weeks 1–2 and tests 4–6 in weeks 3–4, with one re-run allowed.

Common mistakes

• Using a production extract "just for the PoC".
• Testing only the happy path, or letting the vendor choose the data.
• Running in the vendor's demo tenant when you will buy a private deployment; deploy in your own cloud account or data centre instead (see on-premise and self-hosted platforms).
• Skipping the exit test. If you may migrate later, see migrating off a SaaS CDP.
• Changing criteria after seeing results.

---