Table of Contents
- The Ambiguous Failure
- Exactly-Once Delivery Does Not Exist
- Idempotency Keys in Practice
- Natural Idempotency Where Available
- Retry Behaviour That Does Not Amplify
- Consumer-Side Deduplication
- Which Operations Need Protection
- Testing for Duplicates
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: A failed request has two indistinguishable explanations — it never arrived, or it succeeded and the response was lost. Since you cannot tell which, every retry of a state-changing operation must be safe to apply twice.
The Ambiguous Failure
A client sends a payment request. The connection times out. What happened?
Possibility one: the request never reached the server. Retrying is correct and necessary.
Possibility two: the server received it, charged the card, and the response was lost on the way back. Retrying charges the customer twice.
The client cannot distinguish these. The information required to tell them apart is precisely the information that was lost.
This is the fundamental problem in distributed systems communication, and it does not have a clever solution that removes it. The only available response is to make the operation safe to repeat, so that the ambiguity stops mattering.
Teams frequently discover this through a production incident. Retry logic is added for resilience, which is correct, and the retries duplicate side effects, which produces duplicate charges, duplicate emails, duplicate orders, or duplicate inventory decrements. The retry logic was not wrong — it was incomplete, because it was added without the idempotency that makes retrying safe.
Exactly-Once Delivery Does Not Exist
Message queues advertise delivery guarantees, and the terminology causes persistent confusion.
At-most-once. The message may be lost and will never be duplicated. Achieved by not retrying. Unacceptable for anything that matters.
At-least-once. The message will arrive and may arrive several times. Achieved by retrying until acknowledged. This is what real systems provide.
Exactly-once delivery. Not achievable across an unreliable network. This is a theoretical result rather than an engineering limitation — any protocol attempting it reduces to one of the above.
What some systems do provide is exactly-once processing, which is a different claim. It means the effect is applied once even though the message may be delivered several times, and it is achieved through deduplication or transactional coupling between consumption and effect. That is genuinely useful and it is a property of the consumer rather than of delivery.
The practical consequence: assume every message may arrive more than once, and design consumers to tolerate that. A consumer that is safe to run twice makes the delivery guarantee irrelevant, which is a much better position than depending on a guarantee the network cannot provide.
Idempotency Keys in Practice
The standard mechanism for making arbitrary operations safe to retry.
The client generates a unique key per logical operation and includes it with the request. The server records the key alongside the result. A second request with the same key returns the recorded result rather than performing the operation again.
POST /payments
Idempotency-Key: 8f14e45f-ea4c-4f7b-9b3e-2c1a5d7e9f01
{ "amount": 4999, "currency": "GBP", "source": "card_x" }
First request: process payment, store (key → result), return 201
Retry same key: look up key, return stored result, no second chargeThe details that determine whether this actually works:
The key must be generated for the logical operation, not per attempt. A key generated fresh on each retry provides no protection whatsoever. This is the most common implementation error.
Storage must be atomic with the operation. Recording the key and performing the effect must succeed or fail together. Otherwise a crash between them leaves the operation done and the key unrecorded, and the retry duplicates it.
Concurrent requests with the same key need handling. Two simultaneous retries must not both proceed. A unique constraint on the key, or a lock, is required — the second should either wait or receive a conflict response.
Retention must exceed the maximum retry window. Keys retained for an hour do not protect against a client retrying the next day. Twenty-four hours is a common floor.
Request parameters should be validated against the stored key. The same key with different parameters indicates a client bug and should be rejected rather than silently returning the previous result.
That final point catches real problems. A client reusing a key across genuinely different operations will otherwise receive the wrong result, and the failure is silent.
Natural Idempotency Where Available
Some operations are inherently safe to repeat, and designing for that avoids needing keys at all.
Setting an absolute value. set balance = 100 is idempotent. add 50 to balance is not. Where the domain permits absolute assignment, prefer it.
Create with a client-supplied identifier. If the client generates the identifier, a duplicate create attempt collides with the existing record and can return it rather than creating a second.
Conditional updates. Applying a change only if the current state matches an expected value. The second attempt finds the state already changed and does nothing.
Deletion. Deleting an already-deleted record can return success. Repeated deletion is naturally idempotent if you accept that semantics.
State machine transitions. Moving a record to a specific state is idempotent; incrementing through states is not.
The design principle: prefer declarative operations that express the desired end state over imperative operations that express a change. “Ensure this is true” is repeatable; “do this” is not.
Where the domain genuinely requires a relative operation — appending to a log, incrementing a counter, charging an amount — idempotency keys are the answer. Recognising which category an operation falls into is the first step.
Retry Behaviour That Does Not Amplify
Retries protect against transient failure and can convert a partial outage into a complete one.
The mechanism: a service becomes slow, clients time out and retry, retries increase load, the service becomes slower, more clients time out. This is a positive feedback loop that saturates a struggling service precisely when it needs less traffic.
The controls that prevent it:
Exponential backoff with jitter. Doubling delays between attempts, plus randomisation so that synchronised clients do not retry in unison. The jitter matters more than it appears — without it, a service recovering faces a coordinated retry wave.
A bounded attempt count. Three to five attempts, then fail and record it. Unbounded retries are how a transient failure becomes an indefinite load source.
Retry budgets. Cap retries as a proportion of total requests, perhaps ten percent. Under widespread failure this prevents the retry volume itself from becoming the problem.
Circuit breakers. After a threshold of failures, stop attempting entirely for a period. This gives the downstream service room to recover rather than being held under load.
Retry only what is retryable. A 500 or a timeout may succeed on retry. A 400 will not. Retrying deterministic failures is pure waste.
Deadline propagation. Pass a deadline through the call chain so downstream services know how long the caller will wait. Work performed after the caller has abandoned the request is wasted capacity.
That last item is underused and valuable. Without it, a chain of services continues processing a request nobody is waiting for, consuming capacity during exactly the overload conditions that caused the timeout.
Consumer-Side Deduplication
For message consumers, deduplication is the mechanism that makes at-least-once delivery acceptable.
A processed-message table. Record each message identifier as it is handled, in the same transaction as the effect. A duplicate finds the identifier present and skips. Simple, reliable, and requires a transactional store.
Transactional coupling. Where the message store and the effect store are the same database, consuming and applying in one transaction gives exactly-once processing directly.
Version-based rejection. Where messages carry a version or sequence, a consumer that tracks the last applied version discards anything not newer. This handles duplicates and out-of-order delivery together.
Natural idempotency in the handler. The best option when available, since it requires no bookkeeping.
A bounded deduplication window. Retaining identifiers indefinitely is impractical, so choose a window comfortably exceeding the maximum possible redelivery delay.
Two operational points. The deduplication table needs pruning, or it becomes a growing table with an ever-larger index. And ordering guarantees are usually weaker than assumed — many queues guarantee order only within a partition, so a consumer processing messages from several partitions must not depend on global ordering.
Which Operations Need Protection
Not everything requires the same treatment, and applying idempotency keys universally adds cost without benefit.
| Operation | Risk if duplicated | Protection needed |
|---|---|---|
| Read | None | None |
| Payment or charge | Severe — financial | Idempotency key, mandatory |
| Sending email or notification | Moderate — user annoyance | Key or deduplication |
| Creating a record | Moderate — duplicate data | Client-supplied identifier |
| Incrementing a counter | Moderate — wrong values | Key, or absolute set |
| Setting a field to a value | None | Naturally idempotent |
| Deleting | None if tolerant | Naturally idempotent |
| Publishing an event | Depends on consumers | Consumer-side dedup |
| Inventory decrement | Severe — overselling | Key, mandatory |
The rows worth prioritising are the financial and inventory ones, where duplication has direct external consequence. Notifications are the most commonly overlooked — duplicate emails are not catastrophic and they visibly signal to users that something is wrong with your system.
The event publishing row is worth noting because responsibility is distributed: a publisher that may emit duplicates is acceptable if every consumer deduplicates, and it is a hazard if any consumer assumes single delivery. That assumption needs to be documented rather than left implicit.
Testing for Duplicates
Duplicate-handling bugs are difficult to find because they require specific timing that does not occur during normal testing.
Send the same request twice, deliberately. The most basic test and frequently absent. Assert that the second returns the stored result and produces no additional effect.
Send duplicates concurrently. Two simultaneous requests with the same key. This finds the missing unique constraint or lock, which is the most common gap.
Inject failure after the effect, before the response. Simulating the lost-response case directly, then retrying. This is the actual scenario idempotency exists for.
Crash between the effect and the key record. Verifies that the two are genuinely atomic. If they are not, this test finds it.
Replay a message stream. For consumers, replaying messages should be a no-op. If it is not, the consumer is not idempotent.
Test the retention boundary. A retry arriving just after the key expired should be handled predictably, whatever that means for your domain.
The concurrency test is the highest-value one. Sequential duplicate handling is usually implemented correctly; concurrent duplicate handling is frequently not, because it requires a database constraint rather than application logic.
Common Pitfalls
Generating a new key per retry attempt. Provides no protection at all.
Recording the key non-atomically with the effect. A crash between them causes the duplicate you were preventing.
No unique constraint on the key. Concurrent retries both proceed.
Retention shorter than the retry window. Late retries are unprotected.
Unbounded retries without backoff. Converts a degradation into an outage.
Retrying non-retryable errors. Wasted work on deterministic failures.
Assuming exactly-once delivery. It does not exist; design consumers accordingly.
Conclusion
A timeout is ambiguous, permanently. The request either did not arrive or arrived and succeeded with a lost response, and no protocol distinguishes them. Since retrying is necessary for resilience, every state-changing operation must be safe to apply twice.
Prefer natural idempotency where the domain allows it — absolute assignment over relative change, client-supplied identifiers on creation, conditional updates that no-op when already applied. These require no bookkeeping and cannot be implemented incorrectly.
Where relative operations are unavoidable, use idempotency keys with three properties that determine whether they work: generated once per logical operation rather than per attempt, recorded atomically with the effect, and protected by a unique constraint so concurrent retries cannot both proceed.
Then bound the retries. Exponential backoff with jitter, a capped attempt count, a retry budget, and circuit breakers — because retry logic without these converts a slow service into an unavailable one.
Frequently Asked Questions
Should every endpoint accept an idempotency key? Every state-changing endpoint where duplication has consequence. Reads need nothing, and operations that are naturally idempotent need nothing.
Who generates the key? The client, once per logical operation. A server-generated key cannot work, because the client would not know it when retrying.
How long should keys be retained? Longer than any client’s maximum retry window. Twenty-four hours is a common minimum; some payment systems retain considerably longer.
What should happen when the same key arrives with different parameters? Reject with a conflict error. It indicates a client bug, and silently returning the previous result produces a confusing failure.
Is exactly-once delivery genuinely impossible? Across an unreliable network, yes. Exactly-once processing is achievable through deduplication or transactional consumption, which is a property of the consumer rather than of delivery.
How are concurrent duplicate requests handled? A unique constraint on the key in the database. The second insert fails, and that handler either waits for the first to complete or returns a conflict. Application-level checks alone have a race window.
Does this apply to internal service calls? Yes, and it is frequently overlooked there. Internal networks fail too, internal retries duplicate too, and internal duplicate effects are just as real as external ones.
