industries
Peak-Load Activation Test Plan for a Customer Data Platform
By Art Kukharenko · Published · Updated
Browse this industry resourcesTo load test a customer data platform (CDP) for peak traffic, drive synthetic events and profiles at an open-model arrival rate, then measure four things separately: ingest throughput, segmentation lag, activation fan-out, and behaviour when destinations rate-limit you. Pass criteria are latency percentiles and backlog-drain times at a margin over your forecast peak (we suggest 2x), not average throughput. This plan gives the workload model, scenarios, metrics, pass criteria and tooling.
Who this is for: engineers and operations leads preparing a customer data platform for a sales event, launch or seasonal peak. It belongs to the high-load commerce and telecom hub.
What you are proving
A peak test is not a benchmark. It answers business questions:
- Do we accept every event without loss, and can we prove it?
- Does segment membership update fast enough for the campaign to be relevant?
- Do activations reach destinations within the promised time, without duplicates?
- When a destination slows or rejects traffic, do the other destinations stay healthy?
- After a spike, how long does the backlog take to drain?
- Can we replay or backfill during live traffic without breaking either?
- Is consent honoured under load?
Each question becomes a scenario and a pass criterion below.
Build the workload model first
Derive the model from your last real peak, not from a vendor datasheet. Take events per second, new profiles per second, segment count, destination count and the ratio of anonymous to identified traffic from logs. Then project forward.
A worked example, with assumptions stated (illustrative, not benchmarks):
Scroll horizontally to compare all columns ↔
| Step | Value |
|---|---|
| Daily active users | 2,000,000 |
| Events per active user per day | 40 |
| Daily events | 80,000,000 |
| Share of daily events in the peak hour (from your logs) | 12% |
| Events in the peak hour | 9,600,000 |
| Average events per second in the peak hour | about 2,667 |
| Burst factor within the hour (measure it; assumed 3x) | about 8,000 events per second |
Test at 1x, 2x and 3x of the burst figure. The cost side of the same traffic is in why per-profile pricing breaks at scale.
Synthetic generators
Events
Real traffic is not uniform, and uniform tests flatter systems. Your event generator should reproduce:
- A skewed distribution of events per user, where a small share of users produce most events.
- A realistic event mix (page views, product views, add-to-cart, checkout, purchase, identify).
- Anonymous-to-known stitching at your observed rate, including a person arriving on a second device.
- Late, duplicate and out-of-order events, and a small share of malformed payloads.
- Bot-like bursts from few identifiers.
- Seeded randomness, so a failing run can be repeated exactly.
Profiles
Performance depends on store size, so pre-load profiles before the test: the full production count, or at least 10 times the active set. Include merged identities, a range of consent states (including opt-outs) and segments that overlap. Use synthetic personal data only; never copy production personal data into a test environment. Where you need realistic structure, generate it. The synthetic-data proof of concept guide covers generators in more depth.
Open, not closed, load
Choose an open model: arrivals at a fixed rate regardless of how fast the system answers. k6's documentation explains that in a closed model new iterations begin only when previous ones finish, so when the system slows, the arrival rate tapers off and the test under-loads exactly when it should push hardest, the problem known as coordinated omission. Its arrival-rate executors (constant-arrival-rate and ramping-arrival-rate) implement the open model 1.
Scenarios
Scroll horizontally to compare all columns ↔
| # | Scenario | What you do | What you watch |
|---|---|---|---|
| S1 | Ingest spike | Ramp from 1x to 3x over two minutes, hold 30 minutes, drop back | Accept rate, p95 and p99 latency, errors, message-queue lag, events produced versus events stored |
| S2 | Segmentation lag | At 2x ingest, change user behaviour so that known users should enter and leave real-time segments | Event-to-membership latency, evaluation backlog, correctness spot checks, batch recompute duration |
| S3 | Activation fan-out | One event triggers delivery to several destinations (email, push, ad audiences, webhooks) | Event-to-destination latency per destination, per-destination queue depth, duplicates |
| S4 | Destination rate limits | Make mock destinations return HTTP 429 with a retry-after hint, then slow responses | Back-off behaviour, queue growth, isolation from healthy destinations |
| S5 | Replay under load | Replay 24 hours of events at 5x while live traffic is at peak | Live-path latency, duplicate sends, suppression correctness |
| S6 | Failure injection | Kill a node, partition the network, slow a disk, fill a queue | Recovery time, acknowledged-event loss, alerting |
| S7 | Consent withdrawal storm | Submit a burst of opt-outs during peak | Suppression propagation time to every channel |
| S8 | Soak | Run at 1x peak for 4 to 8 hours | Memory, disk growth, compaction and retention jobs |
Destination limits are real constraints
A related point from practitioners: stateless pipelines without replay lose data when a destination is down, which is one reason teams look for platforms that retain events and can replay them 3.
Metrics and pass criteria
Thresholds below are examples to adapt from your business objectives; they are not industry standards.
Scroll horizontally to compare all columns ↔
| Metric | How to measure | Example pass criterion at 2x peak |
|---|---|---|
| Ingest success rate | Accepted requests over total, excluding client errors you injected | At least 99.99% |
| Ingest latency | p50, p95, p99 at the collection endpoint | p99 under 250 ms |
| Event loss | Count produced by the generator versus count stored after backlog drains | Zero for acknowledged events |
| Duplicate rate | Idempotency key collisions | Below 0.01% |
| Event-to-segment latency | Timestamp of event versus segment-membership change | p95 under 5 seconds for real-time segments |
| Event-to-activation latency | Event timestamp versus destination acknowledgement | p95 under 30 seconds |
| Backlog drain | Time for consumer lag to return to normal after the spike | Under 10 minutes |
| Suppression latency | Opt-out timestamp versus last permitted send | p99 under 60 seconds |
| Destination isolation | Change in other destinations' latency while one is rate-limited | Under 5% |
| Recovery | Time to restore service after S6 faults | Under 15 minutes, no acknowledged loss |
| Headroom | CPU, memory, disk, network at 2x peak | CPU under 70% |
Record percentiles, not averages, and keep the raw histograms. Messaging benchmark tools report latency in windows (the OpenMessaging framework, for example, reports average, 50th, 99th and 99.9th percentiles per 10-second window plus a final aggregate) 4.
Tooling
Scroll horizontally to compare all columns ↔
| Tool | Good for | Notes |
|---|---|---|
| k6 | HTTP and event generators with arrival-rate (open-model) scenarios | ramping-arrival-rate takes a start rate, stages, time unit, and pre-allocated and maximum virtual users 15 |
| Locust | Python-defined user behaviour, distributed load, a live web interface, headless in CI | Open source under the MIT license 6 |
Kafka ProducerPerformance (kafka-producer-perf-test) | Pre-filling topics and measuring broker capacity | Options include --num-records, --record-size, --throughput; -1 disables throttling 7 |
| OpenMessaging Benchmark | Comparing messaging systems (Kafka, Pulsar, RocketMQ and others) under defined workloads | Reports publish latency percentiles and "pub delay" 4 |
| Mock destination servers | Emulating 429s, slow responses, partial failures | Write them to mirror each vendor's documented limits |
Environment and hygiene
Checklist
- Derive peak events per second, burst factor, profile count and destination list from real logs.
- Define pass criteria in business terms before testing.
- Build seeded generators with realistic skew, stitching and noise.
- Pre-load profiles at production scale with synthetic personal data.
- Use open-model arrival rates.
- Mock every destination with its documented limits.
- Run S1 to S8, at 1x, 2x and 3x where relevant.
- Reconcile produced, stored and delivered counts after every run.
- Record percentiles and keep raw histograms.
- Run each scenario three times and compare.
- Fix, then re-run; keep a regression suite.
- Schedule the full plan before each peak and after major releases.
For the architecture being tested, see customer data platform architecture and real-time decisioning; for suppression design, see consent withdrawal across channels. Reference deployments are in CDP on AWS, Azure and GCP.
Frequently asked questions
How many events per second should a customer data platform handle?
It depends on your traffic, so derive it. Take daily events, multiply by the share that occurs in your peak hour, divide by 3,600, then apply a burst factor measured from your logs. Test at least twice that figure. The worked example above gives about 2,667 events per second on average in the peak hour and about 8,000 at a 3x burst.
What tools can I use to load test a customer data platform?
k6 and Locust are common choices for generating HTTP or event traffic; k6 offers arrival-rate scenarios and Locust offers Python-defined behaviour and distributed runs 16. Kafka's producer performance tool and the OpenMessaging Benchmark help test the message-queue layer 47. Mock servers cover destinations.
How do I test real-time segmentation latency?
Emit events that should change a known user's segment membership, record the event timestamp, and poll or subscribe for the membership change. Report the p95 and p99 difference while the system is at 2x ingest, and check correctness on a sample.
Should I load test third-party destination APIs?
No, not production ones. Emulate them with mock servers that mimic each vendor's documented limits, including 429 responses and retry-after behaviour 2, or use a vendor-provided sandbox if its terms permit load. Sending synthetic traffic to live providers risks spam complaints, rate-limit penalties and charges on your real accounts.
What is coordinated omission?
It is the distortion that occurs when a load generator waits for a slow response before sending the next request, so the test sends less traffic exactly when the system is struggling and flatters the results. Open-model, arrival-rate executors such as those in k6 avoid it by starting iterations at a fixed rate 1.
Sources
- Grafana k6 docs, open and closed models Open source
- Braze, API rate limits Open source
- Hacker News, RudderStack vs Segment thread (no replay in stateless pipelines) Open source
- OpenMessaging Benchmark Framework Open source
- Grafana k6 docs, ramping-arrival-rate executor Open source
- Locust documentation, "What is Locust?" Open source
- Apache Kafka, ProducerPerformance tool source Open source
Put this evaluation into practice.
Assess your deployment requirementsFurther reading
- CDP for High-Load Commerce & Telecom
- Deployment readiness checklist
- Email Suppression List Management Across
- CDP Proof of Concept With Synthetic Data
- CDP on AWS, Azure and GCP
Published: · Updated:
Publication date recorded for the approved release. Editorial updates do not establish legal approval or a customer result.