How idempotency keys work

A key is a promise the client makes and the server has to keep, which means every hard decision about it is really a decision about who is allowed to forget.

The mechanism looks small. The client attaches an identifier to a request; the server records that identifier alongside the work; a request arriving with an identifier that is already recorded is answered from the record instead of being executed. Four sentences, and it is genuinely the whole idea.

The difficulty is entirely in the parameters. Who generates the identifier, what it is compared against, what happens when it is reused wrongly, how long the record lives, and what the record contains. Each of those has a wrong answer that works in testing.

The client generates it, and that is not negotiable

The key has to be generated by whatever is capable of retrying, and the server cannot do that by definition: if the server could recognise the second request on its own, it would not need a key.

This rules out most of the tempting alternatives. A hash of the request body is not a key, because two genuinely distinct requests can be byte-identical: a customer buying the same coffee twice in a minute produces two identical payloads and expects two charges. A database sequence is not a key, because it is allocated after the request arrives. A timestamp is not a key, because a retry happens at a different time than the original.

What works is an identifier created at the point where the intent is formed, before the first attempt, and reused unchanged by every retry of that intent:

• A UUID generated when the user opens the checkout form, not when they press the button.
• The message id the producer assigned, for a consumer.
• A composite of stable business identifiers, where one genuinely exists.

The rule that holds all three together: a new key means new work. If a retry can generate a fresh key, the mechanism is off. This is the most common implementation bug, and it usually appears as key generation living inside the retry loop rather than outside it.

The header it arrives in has a name people already expect. Stripe shipped Idempotency-Key, the rest of the industry copied it, and an IETF httpapi working group draft, The Idempotency-Key HTTP Header Field, writes the convention down. That draft is not a standard: it expired without being published as an RFC. Use the name because every client library and every API your callers already integrate with uses it, not because a specification obliges you to.

The two-layer version catches people out. A mobile client generates a key, sends the request, and an HTTP client library three layers down retries after a timeout. That retry reuses the key and is protected. But if the user gives up and presses the button again, the application generates a new key, and it should: that is a new intent, not a retry, and suppressing it would be wrong.

Business identifiers versus opaque keys

Where a real identifier exists, it is often better than a UUID, and the choice deserves a decision rather than a default.

An opaque UUID protects against exactly one thing: the same request arriving twice. A business key such as order-8891-refund protects against something stronger, because it is derived from what the work is rather than from the attempt to do it. Two different code paths that both try to refund order 8891 will collide on it, even though neither knows about the other.

The cost is that a business key can be wrong in a way a UUID cannot. If the identifier is not as unique as you believe, two distinct pieces of work share a key and the second is silently suppressed. user-123-daily-report looks stable until the day someone needs it regenerated. The failure is silent, which makes it expensive.

A reasonable default: opaque keys at a public API boundary, where you do not control the caller and cannot enforce a naming scheme; business keys internally, where you do, and where collision is a design decision rather than an accident.

The key alone is not the identity

This is the part most homegrown implementations get wrong, and the bug it produces is the worst kind.

Two services consume the same event stream. The same message id arrives at both. Each must process it once, which means two executions in total and two records, not one. If the record is keyed on the message id alone, the second service’s first delivery is recognised as a duplicate of the first service’s work and skipped.

Nothing fails. No exception, no log line, no alert. Work that was supposed to happen did not happen, and it is discovered later by whoever notices the missing outcome.

So the identity of a record is the key together with something that names the unit of work: a handler, a consumer, a workflow step. Call it a scope. A record is identified by the pair, never by the key alone.

Scope also solves a problem in the other direction. It gives you a namespace, so a key from one part of the system cannot collide with a key from another, and two teams can adopt the same convention without coordinating.

Underneath “which unit of work” sits a second question: whose key is this? Keys live in one namespace in the store unless you do something about it, so two callers that both pick order-1 collide, and the loser either gets the other’s stored response replayed at it or has its own work silently suppressed. Put the caller into the identity, by prefixing the key (userId:clientKey) or by folding the tenant into the scope. Said plainly: a guessable key in a shared namespace is a way to read someone else’s response body, and closing it costs one string concatenation.

Choosing the scope is a small design decision with a long tail. Derive it from a class and method name and it is convenient and stable until someone renames the class, at which point every in-flight record is orphaned and every retry re-executes. Write it by hand and it survives refactoring but can be typo’d into two scopes that were meant to be one. Neither is free; the refactoring hazard is the one that catches people, because renaming a class does not look like a change to runtime behaviour.

Reuse with a different body

A client sends key abc with a €10 charge. Later it sends key abc with a €500 charge.

Replaying the first response is clearly wrong: the caller asked for something different and would be told it succeeded. Executing the second is also wrong: the key says this is a retry, and if the first attempt also ran, money moves twice.

The correct answer is to refuse. This is a client bug, and the only safe response is to say so, loudly, with a status the client cannot mistake for success. Stripe returns a 400; 422 Unprocessable Content is also defensible. What matters is that it is not a 200 and not silent.

To detect it you have to record something about the original request. Storing the whole body is possible and expensive; the usual approach is a hash, often called a fingerprint:

fingerprint = sha256(canonical_form_of(request_body))

canonical_form_of is where the effort goes. If your serialiser orders JSON keys differently between two runs, or includes a timestamp, or renders 1.0 and 1.00 differently, the fingerprint changes between a request and its own retry, and you reject a legitimate retry as a mismatch. That failure is loud rather than silent, which is the right way round, but it is still an outage.

One asymmetry has to be designed in deliberately. A record without a fingerprint cannot be contradicted. If the original was stored before fingerprinting was added, or by a path that does not compute one, an incoming request that has a fingerprint should be treated as a duplicate rather than a mismatch. Two requests conflict only when both carry a fingerprint and the two differ. Any other rule makes enabling the feature an incident.

The duplicate that arrives while the original is running

Almost every naive implementation is this:

if (alreadyProcessed(key)) {
    return storedResult(key);
}
doTheWork();
record(key);

There is a window between the check and the record that is as wide as the work takes, and a duplicate landing inside it passes the check and executes. For a payment taking two seconds against a caller that retries after one, that window is not a rare race. It is the common case.

Closing it means the record has to be created before the work runs, not after, which means a record has three states rather than two: absent, in progress, and complete. And the in-progress state immediately raises the question the two-state version never had to answer: what happens to the second caller?

Three answers, all legitimate:

• Wait, briefly, for the first to finish, then replay its result. Best for an HTTP caller that wants an answer.
• Refuse immediately: for HTTP, a 409 with a Retry-After; for a consumer, decline the delivery. Best for a message consumer, where blocking a thread from a small pool is how a consumer group stops making progress.
• Fail the second outright. Simplest, and correct when duplicates are genuinely rare.

The second question is harder: what happens when the first caller never finishes? A process killed mid-work leaves an in-progress record with no owner. If that record is permanent, the key is now poisoned and no retry will ever succeed. If it expires too quickly, a slow but healthy execution is declared dead while it is still running, and a second execution starts alongside it.

The usual shape is a lease: the record is held for a bounded period, and the holder extends it periodically while it is alive. Stop extending and the lease expires and can be claimed. It is the same mechanism as a distributed lock with a heartbeat, and it has the same unavoidable caveat, which is that a sufficiently long pause is indistinguishable from death.

What to store as the result

The record needs to carry enough for the duplicate to be answered rather than merely refused. A caller that retried because it never saw the response still needs the response.

For an HTTP API that means the status, the body and the headers that matter. Two consequences follow immediately. The first is size: response bodies in a records table grow faster than anyone expects, and a retention policy is not optional. The second is sensitivity: whatever was in that response is now at rest in a second place, with a second set of access controls, and it will outlive the request by however long the retention window is. If the response contains personal data, the records table is now in scope for every rule that applies to it.

For a consumer, the useful thing to store is usually not a return value but what the first run produced: the id of the row it inserted, the ids of the messages it published. That way a duplicate can reference the first run’s work instead of redoing it or pretending it did not happen.

Errors deserve an explicit decision. If a request fails and the failure is recorded as the result, every retry gets the same failure back for as long as the record lives, which is correct for a validation error and badly wrong for a transient database outage. The workable rule is that a thrown failure releases the record and a returned failure completes it: an exception means the work did not happen and the key is free, while deliberately returning an error status means this error is the final answer for that key.

How long to keep records

Retention is where the promise finally expires. Keep a record and duplicates are recognised; drop it and the next retry is a fresh execution.

Bound it from below by how long retries can plausibly arrive. An HTTP client retries for seconds. A message broker’s dead letter queue can be replayed days later. A support engineer re-runs a job next Tuesday. Twenty-four hours is a common default that covers client retries comfortably and covers a Monday-morning replay not at all.

Bound it from above by storage and by what the records contain. Response bodies at API volume add up quickly, and a long window on sensitive data is a liability rather than a safety margin.

Whatever you choose, the expiry has to actually run. A retention policy that is configured but never scheduled is the same as no retention policy, except that everyone believes otherwise. A table that grows without bound degrades slowly and then all at once, and the incident does not look like an idempotency problem when it arrives.

The parameters, in one place

None of these is exotic. What makes the problem hard is that there are nine of them, each has a plausible wrong answer, and several of the wrong answers fail silently.

If you are on Spring, the options for actually building this are compared in idempotency in Spring Boot.

The mechanics of scope, keys, fingerprints and leases as one implementation are in the documentation.