Idempotency at Scale: More Than a Request Header

FinTech Engineering

I start with the business effect, not the header. Idempotency is a concurrency contract that gives repeated requests one durable operation; it is not duplicate detection after the fact.

HTTP semantics are only the starting point

RFC 9110 defines an idempotent method as one whose intended server effect is the same for multiple identical requests as for one. It classifies PUT, DELETE, and safe methods as idempotent, while warning clients not to retry non-idempotent methods automatically unless they know the application semantics are idempotent.

Payment creation is usually POST because it creates a new operation. The application therefore needs a stronger contract: a client supplies an idempotency key; the server binds that key to one canonical operation, one payload fingerprint, and one outcome. Stripe’s idempotent request documentation is a useful public example: it replays the first result, checks that reused keys have matching parameters, and documents a retention window. Those details are Stripe-specific, not properties of HTTP.

Define the namespace before the table

abc-123 is not globally unique simply because a client generated it. Build a namespace such as (tenant_id, api_operation, idempotency_key). A key used for CreatePayment should not collide with the same text used for CreateRefund, and one tenant must not be able to discover another tenant’s operation.

Canonicalize the request fields that determine the financial effect, then hash them. Exclude transport-only metadata, but include amount, currency, destination, capture mode, and any account whose change would create a different operation. Store both the fingerprint and enough protected request metadata to investigate mismatches. Reuse with a different fingerprint should return a conflict, never silently replay the old result.

Claim atomically under concurrency

The hardest race occurs when two identical requests arrive before either has written a result. A read-then-insert implementation allows both workers to execute. Use a unique database constraint and an atomic insert/claim. The winner creates an operation record; the loser reads it.

Here is how I would model the durable record:

• namespace and idempotency key;
• payload fingerprint;
• canonical operation ID;
• state such as in_progress, succeeded, failed_final, or outcome_unknown;
• response status and stable response body or resource reference;
• lease owner and expiry for recoverable work;
• creation and expiry timestamps.

When a duplicate sees in_progress, the API can return 202 Accepted with the operation URL, briefly wait for completion, or return a documented conflict/retry response. It should not start another provider call. A lease helps recover an abandoned worker, but lease expiry alone does not prove that an external side effect did not happen.

Close the crash window

Imagine the worker successfully sends AED 75 to a provider, then crashes before saving the response. Reassigning the lease and sending again can duplicate the payment. The correct recovery depends on the downstream contract:

1. pass a stable downstream idempotency key derived from the internal operation;
2. persist the provider request reference before or with submission where possible;
3. after an ambiguous timeout, query provider status or await a signed callback;
4. keep the internal state outcome_unknown until evidence resolves it.

This is why a response cache placed after the provider is insufficient. The durable operation must exist before the effect, and recovery must understand the effect’s financial semantics.

Decide what gets replayed

Validation failures that occur before execution need not reserve a key forever, but the API contract should say so. Once execution starts, replaying the same stable response—including some failures—can be safer than running again. Stripe documents that its API v1 saves the first status and body, including 500 responses; its newer API behavior differs, so never generalize a vendor rule without checking the API version.

Some results should be represented by a resource reference instead of storing a large response blob. Reconstruct the response from an immutable operation and version the representation. Otherwise a later serialization change can make the “same response” impossible to explain.

Retention, regions, and scale

Retention must cover realistic client retry, offline queue, and webhook-redelivery windows. Expiring too early permits a delayed replay to create a new effect; retaining forever increases storage and privacy obligations. After expiry, a tombstone or business uniqueness rule may still be required for operations that cannot safely repeat.

Multi-region active-active writes complicate atomic claiming. Eventual replication can allow both regions to win. Options include routing a namespace to one home region, using a globally linearizable store, or accepting provisional requests while one authority assigns the operation. Measure the consistency cost instead of claiming “global idempotency” from a local unique index.

Operational metrics should include claim contention, in-progress age, fingerprint conflicts, unknown outcomes, expired replays, storage growth, and hot keys. Load tests must send concurrent duplicates, not only unique traffic.

The contract to publish

My design rule is to publish the contract—supported operations, namespace, retention, mismatch behavior, in-progress response, and error replay—and enforce the same identity downstream. At scale, I earn idempotency with one atomic claim and one recoverable operation, not with the presence of a request header.

Review the contract with failure injection, not only API examples. Pause a worker after its provider call, race one hundred identical requests, reuse a key with one changed field, fail over between regions, and replay after expiry. Inspect both the external side effect and the internal operation. The pass condition is one explainable effect; a collection of identical HTTP responses can still hide duplicated money movement.

Related reading

• Designing an Event-Driven Ledger That Auditors Can Trust
• Payment Orchestration Without the Spaghetti

References

• IETF RFC 9110 §9.2.2: Idempotent Methods
• Stripe: Idempotent requests
• Stripe: Handling duplicate webhook events