Kafka Replay Strategy Without Duplicate Events

A safe Kafka replay strategy starts from one uncomfortable fact: replaying a topic re-delivers events your consumers already processed, so duplicates are not a risk, they are a guarantee. The entire job of a replay plan is making sure those guaranteed duplicates do no harm. That means idempotent consumers first, deliberate offset handling second, and isolation from live side effects third.

Replay is one of Kafka’s best features. It lets you rebuild a broken read model, reprocess after a bug fix, or seed a new service from history. It is also one of the easiest ways to double-charge a customer or send a million duplicate emails if you treat it casually.

Why a replay strategy matters

Kafka retains events, which means the log is a source of truth you can read again. That is powerful: you can rebuild a corrupted projection, backfill a new feature, or recover from a consumer bug by reprocessing from before it shipped.

The danger is that “reprocess from before” also means re-running every side effect those events triggered. Without a plan, a replay that fixes your database also re-sends notifications, re-calls payment APIs, and re-emits downstream events. This post is part of the Distributed systems patterns series, and the dedup problem here is exactly what idempotency keys for distributed systems solve at the consumer; pair it with backpressure design for real-time systems when a replay floods downstream.

Does replaying a Kafka topic create duplicate events?

Yes, always. A replay rewinds the consumer to an earlier offset and re-delivers every message from that point, including ones already processed. There is no replay mode that magically skips what a consumer saw before; duplicate delivery is the defining behavior. Your only defense is making reprocessing idempotent.

This is not a flaw to fix; it is the nature of an append-only log. Kafka’s own “exactly-once semantics” reduce duplicates within a Kafka-to-Kafka pipeline, but they do nothing about the external side effects your consumer performs. The moment your consumer calls an email API or writes to an external system, exactly-once is your responsibility, not Kafka’s.

How do you avoid duplicates when replaying Kafka events?

Make consumers idempotent so processing an event twice yields the same result as processing it once. The two reliable approaches are tracking a unique event ID and skipping IDs you have already applied, or designing every write as an upsert keyed by the event ID so a repeat simply overwrites with identical data.

Idempotency is the foundation, and it is worth designing in before you ever need a replay. A consumer that is idempotent can be replayed fearlessly; one that is not turns every replay into a careful, risky operation.

On each event:
  if event.id in processed_ids:        # dedup table / cache
      skip
  else:
      apply(event)                     # idempotent write (upsert by key)
      record(event.id)                 # atomically with the write if possible

The subtlety is atomicity: recording “I processed event X” and applying its effect should happen together, or you can crash between them and either reprocess (fine, if idempotent) or lose the record (bad). Where possible, fold the dedup key into the same transaction as the write. This is the same idempotency discipline that protects any distributed write.

How do you reset a Kafka consumer offset to replay?

Stop the consumer group, reset its committed offsets to the desired position (a timestamp, a specific offset, or the earliest offset), then restart it. Resetting offsets while the group is live triggers rebalances and races; the clean path is always stop, reset, resume.

The common reset targets are:

• To earliest: reprocess the entire retained history. Use for rebuilding a projection from scratch.
• By timestamp: rewind to just before a bug shipped. The most common surgical replay.
• To a specific offset: precise control when you know exactly where to start.

Should you replay into the same topic or a new one?

Prefer replaying through an isolated consumer group or a separate environment rather than the live group that drives production side effects. Replaying into the group that sends emails or calls payment APIs re-fires all of them unless every side effect is idempotent. Isolation contains the blast radius.

There are three common patterns, in increasing safety:

• Replay the live group in place. Only safe if every consumer and every downstream side effect is idempotent. Fastest, riskiest.
• Replay through a parallel consumer group. A second group reads the same topic and rebuilds a shadow read model, which you swap in once verified. Side effects are disabled in the replay group.
• Replay into a separate environment. Reprocess in staging or a dedicated rebuild cluster, validate the result, then promote. Safest, slowest.

The right choice depends on what the consumer does. A pure read-model projection can often replay in place. A consumer that triggers irreversible external actions should never replay against live side effects.

How long should you retain Kafka data for replay?

Retain as far back as you would ever need to rebuild from, which is a deliberate capacity decision, not a default. Replay can only go back as far as the log is retained, so a 7-day retention means a 7-day replay ceiling. If you need to rebuild a read model from all history, you need either long (or infinite) retention or a compacted topic that keeps the latest value per key.

The two retention models serve different replay goals. Time/size-based retention keeps a rolling window, which is fine for “reprocess the last few days after a bug” but useless for “rebuild from the beginning of time.” Log compaction keeps the latest record per key forever, which is ideal for replaying current state into a new consumer but does not preserve the full event history.

The cost tradeoff is real: longer retention means more storage, which is exactly what tiered storage addresses by moving older log segments to cheap object storage while keeping them replayable. If replay-from-history is a requirement, decide the retention model up front, because you cannot replay events that retention already deleted.

A Kafka replay checklist

Before you rewind a single offset in production:

• Every consumer in the path is idempotent, verified, not assumed.
• Side effects (emails, charges, downstream events) are either idempotent or disabled for the replay.
• You are resetting offsets on a stopped group, with a known target (timestamp/offset/earliest).
• You have estimated the reprocessing volume and the lag it will create for live traffic.
• The replay runs in an isolated group or environment unless in-place is provably safe.
• You can stop the replay midway and know the system is still consistent.
• You have a way to verify the replayed result before it serves real users.

Can Kafka’s exactly-once semantics eliminate replay duplicates?

Only within Kafka, and only for Kafka-to-Kafka flows. Exactly-once semantics (EOS) make a read-process-write pipeline that stays inside Kafka idempotent and transactional, so a consumer that reads from one topic and writes to another will not double-apply across that boundary. That is genuinely useful, and it is not the same as protecting external side effects.

The boundary that EOS does not cross is the one that matters most during a replay: the call to a payment API, the email send, the write to an external database. The moment your consumer does something outside Kafka, exactly-once guarantees end, and your own idempotency is the only thing standing between a replay and a duplicate charge. This is why “we have exactly-once enabled” is not a substitute for idempotent consumers; it covers a narrower surface than people assume.

Treat EOS as a helpful layer for internal stream processing and idempotency as the durable guarantee for everything that touches the outside world. The two work together: EOS keeps your Kafka-internal pipeline clean, and consumer-side idempotency keeps your external effects safe when you rewind.

What are the legitimate reasons to replay?

Replay is a capability, not a repair tool, and the reason matters because it determines the safe procedure. Four cases cover almost everything, and they are not equally risky.

Reason	Shape	Risk
Rebuild a read model	Reprocess history into a projection or search index	Low — the output is derived and disposable
Backfill a new consumer	A new service reads history to build its initial state	Low — nothing downstream existed before
Reprocess after a bug fix	Corrected logic re-applied to affected events	High — live consumers and side effects are in play
Recover from data loss	Rebuild state after a bad migration or deletion	High — usually under time pressure

The two low-risk cases share a property worth exploiting: the destination is new or disposable. Nothing downstream is watching, so a mistake costs a re-run rather than an incident. These should be routine operations you are comfortable performing, and if they are not, that discomfort is a signal that your consumers are not as idempotent as you believe.

The two high-risk cases share the opposite property: they replay into a live system where consumers are running and side effects fire. That is where the procedure below matters, and where the instinct to move fast — always strongest during recovery — is most dangerous.

The strategic conclusion: design so that the high-risk cases become low-risk ones. If a bug fix can be applied by rebuilding a projection into a new index and then swapping, rather than by reprocessing into the live one, you have converted a dangerous operation into a safe one. Architecturally, that means preferring derived, rebuildable read models over state mutated in place, which is one of the underrated practical benefits of event-driven design — and it is only available if you retain enough history to rebuild from, which is a retention decision made long before the incident.

What makes a consumer genuinely replay-safe?

“Make consumers idempotent” is the correct instruction and too abstract to implement from. Concretely, a replay-safe consumer has four properties, and most consumers have two of them.

It deduplicates on a stable event ID, not on arrival. Every event carries an identifier assigned by the producer that survives replay unchanged. The consumer records which IDs it has applied and skips repeats. Deriving the ID from partition and offset breaks immediately on replay into a new topic, because the offsets differ.

Its writes converge. An UPDATE ... SET status = 'shipped' applied twice is harmless. An UPDATE ... SET count = count + 1 applied twice is a bug. Where the domain permits it, prefer absolute assignment over relative mutation, and upserts over inserts — this eliminates whole categories of replay damage without any deduplication machinery at all.

It separates deciding from emitting. A consumer that processes an event and then sends an email has an irreversible side effect in the middle of a replayable pipeline. Moving the emission behind a flag, or into a separate stage that a replay can skip, is what makes the pipeline safe to re-run at all.

It tolerates out-of-order arrival. A replay can deliver an old event after a newer one has already been processed. If applying the old event overwrites newer state, the replay corrupts data that was correct beforehand. The usual guard is a version or timestamp on the record, with the consumer refusing to apply an event older than the state it holds.

That last property is the one most often missing and the most damaging, because it fails quietly. Deduplication protects you from applying the same event twice; nothing protects you from applying an older event after a newer one unless you check. A replay that reprocesses a week of history into a live system with no ordering guard will faithfully roll records backward to their historical values, and the consumer will report complete success.

The general mechanics of dedup, key scoping, and storage are covered in Idempotency Keys for Distributed Systems; what replay adds is the ordering requirement, because normal operation rarely delivers a week-old event to a caught-up consumer and replay does it constantly.

How do you run a replay safely in production?

Replay is one of the few operations that can corrupt data across many services at once, and it is usually performed under pressure after something has already gone wrong. That combination deserves a procedure rather than improvisation.

The sequence that keeps it safe:

1. Write down the exact scope first. Which topic, which partitions, which offset or timestamp range, and which consumer group. “Replay yesterday” is not a scope; it is how you reprocess three weeks by accident.
2. Confirm every consumer in the path is idempotent. Not “should be” — verified. A single non-idempotent consumer downstream turns a replay into duplicated side effects, and it is frequently a consumer whose team does not know a replay is happening.
3. Identify side effects that cannot be undone. Emails, payments, push notifications, webhooks to third parties. These need to be disabled, routed to a sink, or filtered out before the replay starts, because no amount of idempotency downstream helps once an email has been sent.
4. Dry-run against a copy. Replay into a separate consumer group that writes to a scratch destination, and compare the result against expectations. This is the step most often skipped and the one that most often prevents an incident.
5. Rate-limit the replay. Historical data reprocesses far faster than it originally arrived — potentially days of events in minutes — which can overwhelm databases and downstream services sized for real-time rates. Throttle deliberately.
6. Replay, then verify against a known quantity. Row counts, checksums, or a spot-check of records you can independently confirm.

Two decisions worth making in advance rather than mid-operation. Prefer a new consumer group over resetting an existing one, because a new group leaves the live group’s offsets untouched, so live processing continues and rollback is simply “stop the replay.” Resetting the production group’s offsets couples your recovery to your live traffic and has no clean undo.

And prefer replaying into a new topic when the transformation itself changed. If the reason for replaying is that consumers now interpret events differently, writing the reprocessed results to a new topic lets you validate the new output alongside the old before cutting over — a migration rather than an in-place mutation.

The rate-limiting point deserves emphasis because it is the most common way a well-planned replay causes an outage. A consumer that comfortably handles 500 messages per second in production will happily pull 50,000 per second from a historical backlog, and the database behind it will not. That is the same backpressure problem described in Backpressure Design for Real-Time Systems, arriving through a door nobody was watching.

What retention do you need to make replay possible?

Replay is only available if the data still exists, and retention is decided long before anyone wants to replay. The default of a few days is frequently too short for the cases that matter.

Work backwards from what you want to be able to rebuild. If a projection must be reconstructable from scratch, retention has to cover its entire history, which usually means infinite retention on a compacted changelog topic rather than a time-bounded one. If replay exists to recover from bugs, the window has to exceed your realistic time-to-detection — and a subtle logic bug frequently goes unnoticed for longer than a week.

Three practical rules. Deletion-sensitive topics deserve longer retention, so no consumer can miss a deletion event across an outage window and be left holding data it was supposed to purge. Compaction and replay interact: a compacted topic retains only the latest value per key, so it can rebuild current state but cannot replay the sequence of changes — if history matters, you need a non-compacted topic or a separate archive. And retention is a cost decision with a recovery consequence, so make it deliberately rather than accepting the broker default, which was chosen by someone who did not know your recovery requirements. The full treatment is in Kafka Partitions, Retention, and Compaction.

A final habit that pays for itself: practise a replay before you need one. Rebuilding a non-critical projection from history once a quarter proves the whole chain works — retention is sufficient, consumers are genuinely idempotent, the offset tooling behaves as documented, and downstream systems survive the rate. Every one of those assumptions decays silently, and the moment you actually need a replay is the worst possible time to discover which one stopped being true. Treat the rehearsal as you would a backup restore drill: cheap, boring, and the only thing that converts a documented capability into one you can actually rely on.

What I’d do differently

The mistake that teaches this lesson is replaying a topic to fix a read model and discovering, an hour later, that you also re-sent every notification in the window. The damage is not the replay; it is that the consumer’s side effects were never idempotent, so replay was never actually safe.

If I were designing the consumer from scratch, I would make idempotency a day-one property, not a thing I bolt on when a replay goes wrong. Assign every event a stable ID at production time, dedup on it at every consumer, and keep irreversible side effects behind an idempotency guard. Do that, and replay stops being a high-stakes operation and becomes the routine, boring recovery tool it should be.

Sources

• Apache Kafka, Consumer offsets and kafka-consumer-groups tool: kafka.apache.org/documentation/#basic_ops_consumer_group
• Apache Kafka, Exactly-once semantics: kafka.apache.org/documentation/#semantics
• Confluent, Message delivery guarantees: docs.confluent.io/kafka/design/delivery-semantics.html