industries

Peak-Load Activation Test Plan for a Customer Data Platform

By Art Kukharenko · Published · Updated

Browse this industry resources

To load test a customer data platform (CDP) for peak traffic, drive synthetic events and profiles at an open-model arrival rate, then measure four things separately: ingest throughput, segmentation lag, activation fan-out, and behaviour when destinations rate-limit you. Pass criteria are latency percentiles and backlog-drain times at a margin over your forecast peak (we suggest 2x), not average throughput. This plan gives the workload model, scenarios, metrics, pass criteria and tooling.

Who this is for: engineers and operations leads preparing a customer data platform for a sales event, launch or seasonal peak. It belongs to the high-load commerce and telecom hub.

What you are proving

A peak test is not a benchmark. It answers business questions:

  1. Do we accept every event without loss, and can we prove it?
  2. Does segment membership update fast enough for the campaign to be relevant?
  3. Do activations reach destinations within the promised time, without duplicates?
  4. When a destination slows or rejects traffic, do the other destinations stay healthy?
  5. After a spike, how long does the backlog take to drain?
  6. Can we replay or backfill during live traffic without breaking either?
  7. Is consent honoured under load?

Each question becomes a scenario and a pass criterion below.

Build the workload model first

Derive the model from your last real peak, not from a vendor datasheet. Take events per second, new profiles per second, segment count, destination count and the ratio of anonymous to identified traffic from logs. Then project forward.

A worked example, with assumptions stated (illustrative, not benchmarks):

Scroll horizontally to compare all columns ↔

Comparison — source references are listed below
StepValue
Daily active users2,000,000
Events per active user per day40
Daily events80,000,000
Share of daily events in the peak hour (from your logs)12%
Events in the peak hour9,600,000
Average events per second in the peak hourabout 2,667
Burst factor within the hour (measure it; assumed 3x)about 8,000 events per second

Test at 1x, 2x and 3x of the burst figure. The cost side of the same traffic is in why per-profile pricing breaks at scale.

Synthetic generators

Events

Real traffic is not uniform, and uniform tests flatter systems. Your event generator should reproduce:

  • A skewed distribution of events per user, where a small share of users produce most events.
  • A realistic event mix (page views, product views, add-to-cart, checkout, purchase, identify).
  • Anonymous-to-known stitching at your observed rate, including a person arriving on a second device.
  • Late, duplicate and out-of-order events, and a small share of malformed payloads.
  • Bot-like bursts from few identifiers.
  • Seeded randomness, so a failing run can be repeated exactly.

Profiles

Performance depends on store size, so pre-load profiles before the test: the full production count, or at least 10 times the active set. Include merged identities, a range of consent states (including opt-outs) and segments that overlap. Use synthetic personal data only; never copy production personal data into a test environment. Where you need realistic structure, generate it. The synthetic-data proof of concept guide covers generators in more depth.

Open, not closed, load

Choose an open model: arrivals at a fixed rate regardless of how fast the system answers. k6's documentation explains that in a closed model new iterations begin only when previous ones finish, so when the system slows, the arrival rate tapers off and the test under-loads exactly when it should push hardest, the problem known as coordinated omission. Its arrival-rate executors (constant-arrival-rate and ramping-arrival-rate) implement the open model 1.

Scenarios

Scroll horizontally to compare all columns ↔

Comparison — source references are listed below
#ScenarioWhat you doWhat you watch
S1Ingest spikeRamp from 1x to 3x over two minutes, hold 30 minutes, drop backAccept rate, p95 and p99 latency, errors, message-queue lag, events produced versus events stored
S2Segmentation lagAt 2x ingest, change user behaviour so that known users should enter and leave real-time segmentsEvent-to-membership latency, evaluation backlog, correctness spot checks, batch recompute duration
S3Activation fan-outOne event triggers delivery to several destinations (email, push, ad audiences, webhooks)Event-to-destination latency per destination, per-destination queue depth, duplicates
S4Destination rate limitsMake mock destinations return HTTP 429 with a retry-after hint, then slow responsesBack-off behaviour, queue growth, isolation from healthy destinations
S5Replay under loadReplay 24 hours of events at 5x while live traffic is at peakLive-path latency, duplicate sends, suppression correctness
S6Failure injectionKill a node, partition the network, slow a disk, fill a queueRecovery time, acknowledged-event loss, alerting
S7Consent withdrawal stormSubmit a burst of opt-outs during peakSuppression propagation time to every channel
S8SoakRun at 1x peak for 4 to 8 hoursMemory, disk growth, compaction and retention jobs

Destination limits are real constraints

A related point from practitioners: stateless pipelines without replay lose data when a destination is down, which is one reason teams look for platforms that retain events and can replay them 3.

Metrics and pass criteria

Thresholds below are examples to adapt from your business objectives; they are not industry standards.

Scroll horizontally to compare all columns ↔

Comparison — source references are listed below
MetricHow to measureExample pass criterion at 2x peak
Ingest success rateAccepted requests over total, excluding client errors you injectedAt least 99.99%
Ingest latencyp50, p95, p99 at the collection endpointp99 under 250 ms
Event lossCount produced by the generator versus count stored after backlog drainsZero for acknowledged events
Duplicate rateIdempotency key collisionsBelow 0.01%
Event-to-segment latencyTimestamp of event versus segment-membership changep95 under 5 seconds for real-time segments
Event-to-activation latencyEvent timestamp versus destination acknowledgementp95 under 30 seconds
Backlog drainTime for consumer lag to return to normal after the spikeUnder 10 minutes
Suppression latencyOpt-out timestamp versus last permitted sendp99 under 60 seconds
Destination isolationChange in other destinations' latency while one is rate-limitedUnder 5%
RecoveryTime to restore service after S6 faultsUnder 15 minutes, no acknowledged loss
HeadroomCPU, memory, disk, network at 2x peakCPU under 70%

Record percentiles, not averages, and keep the raw histograms. Messaging benchmark tools report latency in windows (the OpenMessaging framework, for example, reports average, 50th, 99th and 99.9th percentiles per 10-second window plus a final aggregate) 4.

Tooling

Scroll horizontally to compare all columns ↔

Comparison — source references are listed below
ToolGood forNotes
k6HTTP and event generators with arrival-rate (open-model) scenariosramping-arrival-rate takes a start rate, stages, time unit, and pre-allocated and maximum virtual users 15
LocustPython-defined user behaviour, distributed load, a live web interface, headless in CIOpen source under the MIT license 6
Kafka ProducerPerformance (kafka-producer-perf-test)Pre-filling topics and measuring broker capacityOptions include --num-records, --record-size, --throughput; -1 disables throttling 7
OpenMessaging BenchmarkComparing messaging systems (Kafka, Pulsar, RocketMQ and others) under defined workloadsReports publish latency percentiles and "pub delay" 4
Mock destination serversEmulating 429s, slow responses, partial failuresWrite them to mirror each vendor's documented limits

Environment and hygiene

Checklist

  1. Derive peak events per second, burst factor, profile count and destination list from real logs.
  2. Define pass criteria in business terms before testing.
  3. Build seeded generators with realistic skew, stitching and noise.
  4. Pre-load profiles at production scale with synthetic personal data.
  5. Use open-model arrival rates.
  6. Mock every destination with its documented limits.
  7. Run S1 to S8, at 1x, 2x and 3x where relevant.
  8. Reconcile produced, stored and delivered counts after every run.
  9. Record percentiles and keep raw histograms.
  10. Run each scenario three times and compare.
  11. Fix, then re-run; keep a regression suite.
  12. Schedule the full plan before each peak and after major releases.

For the architecture being tested, see customer data platform architecture and real-time decisioning; for suppression design, see consent withdrawal across channels. Reference deployments are in CDP on AWS, Azure and GCP.


Frequently asked questions

How many events per second should a customer data platform handle?

It depends on your traffic, so derive it. Take daily events, multiply by the share that occurs in your peak hour, divide by 3,600, then apply a burst factor measured from your logs. Test at least twice that figure. The worked example above gives about 2,667 events per second on average in the peak hour and about 8,000 at a 3x burst.

What tools can I use to load test a customer data platform?

k6 and Locust are common choices for generating HTTP or event traffic; k6 offers arrival-rate scenarios and Locust offers Python-defined behaviour and distributed runs 16. Kafka's producer performance tool and the OpenMessaging Benchmark help test the message-queue layer 47. Mock servers cover destinations.

How do I test real-time segmentation latency?

Emit events that should change a known user's segment membership, record the event timestamp, and poll or subscribe for the membership change. Report the p95 and p99 difference while the system is at 2x ingest, and check correctness on a sample.

Should I load test third-party destination APIs?

No, not production ones. Emulate them with mock servers that mimic each vendor's documented limits, including 429 responses and retry-after behaviour 2, or use a vendor-provided sandbox if its terms permit load. Sending synthetic traffic to live providers risks spam complaints, rate-limit penalties and charges on your real accounts.

What is coordinated omission?

It is the distortion that occurs when a load generator waits for a slow response before sending the next request, so the test sends less traffic exactly when the system is struggling and flatters the results. Open-model, arrival-rate executors such as those in k6 avoid it by starting iterations at a fixed rate 1.

Sources

  1. Grafana k6 docs, open and closed models Open source
  2. Braze, API rate limits Open source
  3. Hacker News, RudderStack vs Segment thread (no replay in stateless pipelines) Open source
  4. OpenMessaging Benchmark Framework Open source
  5. Grafana k6 docs, ramping-arrival-rate executor Open source
  6. Locust documentation, "What is Locust?" Open source
  7. Apache Kafka, ProducerPerformance tool source Open source

Put this evaluation into practice.

Assess your deployment requirements

Further reading

Published: · Updated:

Publication date recorded for the approved release. Editorial updates do not establish legal approval or a customer result.

Need help?

Ask the PrivyCDP Assistant