Key takeaway: Building exactly-once delivery is genuinely hard and most systems do not actually provide it, whatever the marketing implies. The dependable and achievable target is at-least-once delivery paired with an idempotent handler, which produces the same practical outcome without the underlying complexity.
Why At-Least-Once Is What You Actually Get
A worker picks up a job, begins processing, and crashes or loses connectivity before acknowledging completion to the queue. The queue, having received no acknowledgement, redelivers the job to another worker, which then processes it again — and the first worker may also have completed the actual side effect before crashing, meaning the effect now happens twice even though the queue behaved exactly as designed.
This is not a queue malfunction. It is the fundamental trade-off in distributed message delivery: guaranteeing a message is never lost requires accepting that it might occasionally be delivered more than once, because the alternative — guaranteeing it is never duplicated — risks losing it entirely if an acknowledgement is lost at the wrong moment.
What Goes Wrong Without Idempotency
| Job type | Consequence of running twice |
|---|---|
| Charge a customer’s payment method | Customer charged twice |
| Send a notification email | Customer receives duplicate email |
| Increment a counter or balance | Value becomes incorrect |
| Create a record | Duplicate record created |
| Send a webhook to a third party | Third party receives and may act on it twice |
Charging a payment method twice is the case with the most direct and visible consequence, and it is also one of the most common places this bug actually surfaces in production, because payment processing jobs are exactly the kind of critical, side-effect-heavy operation that gets queued for reliability in the first place — the very reliability mechanism introduces the duplication risk it is meant to protect against.
Making a Handler Idempotent
The general pattern is recording that a specific job, identified by a unique key, has already been processed, and checking that record before performing the job’s actual side effect — if the key is already recorded as processed, the handler returns successfully without repeating the side effect, rather than executing it again.
def process_payment_job(job):
if already_processed(job.idempotency_key):
return # safe no-op, already done
charge_payment(job.amount, job.customer_id)
mark_processed(job.idempotency_key)The idempotency check and the marking of completion need to be part of the same atomic operation as the side effect itself where possible, or protected against the same race condition the job system introduced in the first place — a check-then-act pattern implemented naively can itself be run twice concurrently by two workers picking up what they believe is the same unprocessed job.
Where the Idempotency Key Comes From
The key needs to uniquely and stably identify the specific unit of work — not the job’s queue message ID, which can differ between the original delivery and a redelivery of logically the same work, but something derived from the actual business operation, such as an order ID combined with an operation type, that remains identical across any redelivery of that same logical job.
Using a random identifier generated fresh each time a job is enqueued defeats the purpose entirely, since a redelivered job would then appear to have a different key and the idempotency check would incorrectly treat it as new work rather than recognising it as a duplicate.
Downstream Idempotency Too
For jobs that call external services — sending a webhook, calling a third-party API — passing an idempotency key to that external service, where the service supports it, extends the same protection past your own system boundary. Many external APIs, particularly payment providers, explicitly support an idempotency key parameter for exactly this reason, and using it closes the same duplication gap on their side of the call.
The Bottom Line
Design every job handler to be idempotent using a stable key derived from the actual business operation, not the queue’s own message identifier, and treat this as a required property of any job with a real side effect rather than an edge case to handle only if duplication is actually observed in production. Redelivery will happen eventually regardless of how reliable the queue infrastructure is, because at-least-once is the honest guarantee almost every practical system provides.
