Idempotency Key Storage, TTL and Eviction

This page belongs to Idempotency Key Implementation in the API Design Fundamentals & Architecture reference. Generating and accepting a key is the easy half. The half that decides whether the guarantee actually holds under failure is where the key is stored, how long it survives, and what happens when two copies of the same request arrive at once.

The decision trigger

Symptom Cause
A duplicate charge after a client retried the next morning key expired before the retry
A duplicate created during a network blip no lock; two requests executed concurrently
A replayed response describes a different order key reused with a different body, replayed anyway
Keys survive the effect after a rollback key stored outside the effect’s transaction
The key store grew unboundedly no eviction policy
A cache failover produced duplicates keys in a non-durable store

Spec snippet: the stored record

-- The key lives in the SAME database as the effect, so both commit together.
CREATE TABLE idempotency_keys (
    key                text        PRIMARY KEY,
    api_key_id         text        NOT NULL,
    request_method     text        NOT NULL,
    request_path       text        NOT NULL,
    request_fingerprint bytea      NOT NULL,          -- sha256 of the canonical body
    state              text        NOT NULL,          -- 'in_progress' | 'completed'
    response_status    int,
    response_body      jsonb,
    locked_at          timestamptz,
    created_at         timestamptz NOT NULL DEFAULT now(),
    expires_at         timestamptz NOT NULL
);

CREATE INDEX idempotency_keys_expiry ON idempotency_keys (expires_at);

-- Scoping by caller stops one tenant's key colliding with another's.
CREATE UNIQUE INDEX idempotency_keys_scope
    ON idempotency_keys (api_key_id, key);

Four columns carry the weight. request_fingerprint detects a key reused with a different payload; state distinguishes “in flight” from “finished”; locked_at supports lock expiry when a process dies mid-request; and expires_at drives eviction. The response body is stored so a retry can be answered without re-executing anything.

Idempotency key lifecycle A request with an unseen key inserts an in-progress record and executes. A duplicate arriving while in progress receives a conflict. Once completed the stored response is replayed. A mismatched fingerprint is rejected. After expiry the key becomes absent again. absent first use insert in_progress locked, executing commit completed response stored duplicate while in flight → 409 retry → replay stored response different fingerprint → 422 keyreused after expires_at → evicted, key becomes absent again

Step-by-step

Step 1 — Commit the key with the effect

The single most important property: the key record and the side effect must be atomic with respect to each other. If they are not, a crash between them produces either a duplicate (effect without key) or a phantom (key without effect).

// create-payment.ts — key and effect in one transaction
import { createHash } from "node:crypto";

const TTL_HOURS = 24;

export async function createPayment(req: Request, res: Response) {
  const key = req.header("Idempotency-Key");
  if (!key) return sendMissingKey(res);

  const fingerprint = createHash("sha256")
    .update(canonicalJson(req.body))          // stable member ordering
    .digest();

  const outcome = await db.transaction(async tx => {
    const existing = await tx.one(
      `SELECT * FROM idempotency_keys WHERE api_key_id = $1 AND key = $2 FOR UPDATE`,
      [req.apiKeyId, key],
    );

    if (existing) {
      if (!existing.request_fingerprint.equals(fingerprint)) {
        return { kind: "fingerprint_mismatch" as const };
      }
      if (existing.state === "in_progress") return { kind: "in_flight" as const };
      return { kind: "replay" as const, status: existing.response_status, body: existing.response_body };
    }

    await tx.none(
      `INSERT INTO idempotency_keys
         (key, api_key_id, request_method, request_path, request_fingerprint, state, expires_at)
       VALUES ($1, $2, $3, $4, $5, 'in_progress', now() + interval '${TTL_HOURS} hours')`,
      [key, req.apiKeyId, req.method, req.path, fingerprint],
    );

    const payment = await createPaymentRow(tx, req.body);   // the effect, same tx

    await tx.none(
      `UPDATE idempotency_keys
          SET state = 'completed', response_status = 201, response_body = $3
        WHERE api_key_id = $1 AND key = $2`,
      [req.apiKeyId, key, payment],
    );

    return { kind: "created" as const, body: payment };
  });

  switch (outcome.kind) {
    case "created":  return res.status(201).json(outcome.body);
    case "replay":   return res.status(outcome.status).json(outcome.body);
    case "in_flight":return sendInFlight(res);                  // 409
    case "fingerprint_mismatch": return sendKeyReused(res);     // 422
  }
}

SELECT … FOR UPDATE is what serialises concurrent duplicates: the second request blocks until the first commits, then sees completed and replays. Without it, two requests can both find no row and both insert.

Step 2 — Choose the store deliberately

Store Durability Best for Failure mode
Primary database, same transaction strong money, contracts, anything unreconcilable slower writes
Primary database, separate transaction medium effects that are cheap to reconcile crash between commits duplicates
Redis with persistence medium high-volume, low-value operations failover can lose recent keys
Redis without persistence weak rate-limit style dedup only restart loses everything
Distributed KV (DynamoDB, etc.) strong multi-region APIs conditional writes needed

The rule follows the value of the effect. A duplicated “mark notification as read” is noise; a duplicated payment is an incident and a refund. Match the store’s durability to the cost of a duplicate, not to what is convenient.

Step 3 — Set the TTL from client behaviour

Retention must outlast the longest client retry Automatic retries occur within seconds. A queue drain retry can occur hours later. A manual replay by an operator can occur the next day. A one-hour retention window covers only the first two, leaving the manual replay to create a duplicate. time since the original request → automatic retry seconds queue drain minutes–hours operator replay next morning reconciliation job next day TTL 1h — too short TTL 24h+ — covers every realistic retry path

Twenty-four hours is the common floor for payment-style operations because it covers an operator replaying a failed batch the following morning. Where a client’s queue can be paused over a weekend, seven days is more honest. Storage is cheap: a key record with a stored response is a few kilobytes, so a million operations a day at seven days’ retention is single-digit gigabytes.

Step 4 — Handle stuck locks

A process that crashes mid-request leaves an in_progress row that would block retries forever. Reclaim it on a bounded timeout rather than never:

-- Reclaim keys whose owner died: older than the maximum request duration.
UPDATE idempotency_keys
   SET state = 'absent'
 WHERE state = 'in_progress'
   AND locked_at < now() - interval '5 minutes'
RETURNING key;

Set the reclaim interval above your longest possible handler duration, including downstream timeouts. Reclaiming too eagerly reintroduces exactly the double-execution the lock exists to prevent.

Step 5 — Evict on a schedule you control

-- Batch eviction: bounded, index-supported, safe to run every few minutes.
DELETE FROM idempotency_keys
 WHERE key IN (
   SELECT key FROM idempotency_keys
    WHERE expires_at < now()
    ORDER BY expires_at
    LIMIT 10000
 );

Deleting in bounded batches keeps the statement from holding long locks on a busy table. Avoid relying on a database TTL feature you cannot observe: you want a metric for “keys evicted” and “oldest live key”, because an eviction job that quietly stops is invisible until the table is enormous.

Choosing a store is really choosing what happens during a failover, and the honest way to decide is by the cost of one duplicate:

Matching the key store to the cost of a duplicate Operations that move money need the key in the same transaction as the effect. Cheap, reconcilable operations can tolerate a cache. The failure mode of each store is stated. match durability to what a duplicate costs payments, contracts a duplicate needs a refund same transaction as the effect provisioning, messaging a duplicate is visible but fixable durable store, separate transaction notifications, cache warms a duplicate is noise persistent cache is acceptable read deduplication no side effect at all in-memory is fine a cache failover silently loses recent keys — that is the whole decision

Standard compliance

Concern Standard Clause Note
Idempotency-Key header IETF draft (httpapi-idempotency-key-header) — Not yet an RFC; widely deployed
Idempotent methods RFC 9110 §9.2.2 PUT/DELETE are idempotent by definition
409 Conflict RFC 9110 §15.5.10 For a request already in flight
422 Unprocessable Content RFC 9110 §15.5.21 For a key reused with a different body
Problem body RFC 9457 §3 Error shape for both cases

Note the asymmetry the draft header exists to solve: PUT and DELETE are idempotent by definition, so they need no key, while POST is not, which is why the key applies almost exclusively to creation and command endpoints.

Concurrency and safety implications

The lock turns a race into a queue. Without it, two simultaneous requests with the same key can both check for an existing record, both find nothing, and both execute — the exact duplicate the key was meant to prevent, now harder to diagnose because the key store looks correct afterwards. Returning 409 to the second request is usually better than making it wait: a client that fired two copies of the same request concurrently has a bug, and blocking hides it while 409 surfaces it. The classification of that 409 as retryable-after-delay belongs with the rules in Retryable vs Non-Retryable Errors.

SDK / codegen downstream effect

  // Generated client with idempotency support in the wrapper layer
  export async function createPayment(body: PaymentCreate, opts?: { idempotencyKey?: string }) {
-   return api.createPayment({ paymentCreate: body });
+   const key = opts?.idempotencyKey ?? crypto.randomUUID();
+   // The SAME key must be reused across retries of this logical operation.
+   return withRetries(() => api.createPayment({ paymentCreate: body, idempotencyKey: key }));
  }

The subtle requirement is that the key is generated outside the retry loop. A key generated inside it changes on every attempt, which makes the mechanism useless while looking correct in code review — the single most common client-side idempotency bug, covered further in Generating Idempotency Keys in Node.js Express APIs.

Common mistakes

Mistake Correct approach
Key written in a separate transaction from the effect Commit both atomically
TTL shorter than the client’s retry window Retain 24 hours or more; document the window
No request fingerprint Hash the canonical body and reject mismatches
No lock on first use SELECT … FOR UPDATE or an equivalent conditional write
Stuck in_progress rows never reclaimed Reclaim after a bounded timeout above the max handler duration
Unbounded key table Batch-evict expired rows and alert on eviction lag
Keys in a non-durable cache for financial operations Use the transactional store
Generating the key inside the retry loop Generate once per logical operation

FAQ

How long should idempotency keys be retained?

Long enough to cover the slowest path by which the same logical request can arrive twice — which is almost never the automatic retry. Automatic retries land within seconds; the dangerous cases are a queue that was paused and drained hours later, or an operator replaying a failed batch the next morning. Twenty-four hours is the usual floor, and seven days is defensible where clients batch. Retention costs storage measured in gigabytes; premature expiry costs a duplicate side effect and a refund process. Whatever you choose, state it in the API reference so client authors can size their own retry windows against it.

Can idempotency keys live in Redis rather than the primary database?

Yes for low-value deduplication, no for anything unreconcilable. The problem is not Redis’s speed but its failure semantics: a failover to a replica that has not received the last few writes silently loses recent keys, and the retry that follows creates a duplicate. If the effect moves money, issues credentials, or creates a record a human would have to unpick, keep the key in the same transactional store as the effect so that both commit or neither does. A reasonable hybrid is Redis as a fast-path check in front of a durable record — but the durable record remains the authority.

What happens when the same key arrives with a different request body?

Reject it, loudly. Replaying the stored response would return a result describing a different request, which is worse than an error because the client believes its new request succeeded. Compare a fingerprint — a hash over a canonical serialisation of the body, with member ordering normalised — and return 422 with a problem document explaining that the key was already used for a different payload. This is nearly always a client bug, such as a key derived from a timestamp truncated to the minute or a key cached too broadly, and surfacing it immediately is a kindness.

Back to Idempotency Key Implementation