Handling Idempotency Key Conflicts

This guide belongs to Idempotency Key Implementation, part of the API Design Fundamentals & Architecture reference. Once you are generating idempotency keys in Node.js Express APIs, the hard part is no longer minting keys — it is deciding what to do when the same key arrives twice, especially when the second request does not match the first. This page covers replay detection, 409 on payload mismatch, request fingerprinting, in-flight locking, and key retention.

The symptom that triggers this decision

A client sends POST /payments with Idempotency-Key: abc, times out before the response, and retries. Three things can happen and only one is correct. If the retry is byte-for-byte the same, you must replay the original response — not charge the card again. If the retry reuses abc but changes the amount from $10 to $1000, you must reject it, because the key already names a different operation. And if two retries arrive concurrently while the first is still processing, you must not let both execute. An idempotency layer that only handles the happy “same request twice, sequentially” case will still double-charge under concurrency or silently accept a mismatched replay. This is why conflict handling — not key storage — is the real work.

The three outcomes

Request fingerprinting

To tell a genuine replay from a conflicting reuse, store a fingerprint of the original request: a hash of the canonicalized body (and any headers that change semantics, such as the target account). On replay you compare the incoming fingerprint to the stored one.

Canonicalization matters — {"a":1,"b":2} and {"b":2,"a":1} are the same request but different bytes. Serialize with sorted keys before hashing so semantically identical bodies hash identically.

Node:

import { createHash } from 'node:crypto';

function canonicalize(value) {
  if (Array.isArray(value)) return value.map(canonicalize);
  if (value && typeof value === 'object') {
    return Object.keys(value).sort().reduce((acc, k) => {
      acc[k] = canonicalize(value[k]);
      return acc;
    }, {});
  }
  return value;
}

export function fingerprint(body) {
  const canonical = JSON.stringify(canonicalize(body));
  return createHash('sha256').update(canonical).digest('hex');
}

Python:

import hashlib, json

def fingerprint(body: dict) -> str:
    canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

Storage schema

Persist one row per key with its fingerprint, a status, the cached response, and an expiry. A unique constraint on the key is what makes the in-flight lock atomic.

CREATE TABLE idempotency_keys (
  key              TEXT        PRIMARY KEY,
  fingerprint      TEXT        NOT NULL,      -- sha256 of canonical request
  status           TEXT        NOT NULL,      -- 'in_flight' | 'completed'
  response_status  INTEGER,                   -- cached HTTP status
  response_body    JSONB,                     -- cached response body
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at       TIMESTAMPTZ NOT NULL       -- retention window end
);

CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at);

Step-by-step conflict handling

Step 1 — Claim the key atomically (the in-flight lock)

Insert the key with status in_flight. If the row already exists, the INSERT fails on the primary key and you fall through to the replay path. The atomic insert is the lock: exactly one concurrent request wins it.

INSERT INTO idempotency_keys (key, fingerprint, status, expires_at)
VALUES ($1, $2, 'in_flight', now() + interval '24 hours')
ON CONFLICT (key) DO NOTHING
RETURNING key;

If RETURNING yields a row, you won the lock — process the request. If it returns nothing, an entry already exists; go to Step 2.

Step 2 — Compare fingerprints on an existing key

async function handleIdempotent(req, res, next) {
  const key = req.get('Idempotency-Key');
  if (!key) return res.status(400).json({ title: 'Idempotency-Key header required', status: 400 });

  const fp = fingerprint(req.body);
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, fingerprint, status, expires_at)
     VALUES ($1, $2, 'in_flight', now() + interval '24 hours')
     ON CONFLICT (key) DO NOTHING RETURNING key`,
    [key, fp],
  );

  if (claimed.rowCount === 1) {
    return next();                       // we own the lock — process
  }

  const existing = (await db.query(
    `SELECT fingerprint, status, response_status, response_body
     FROM idempotency_keys WHERE key = $1`, [key],
  )).rows[0];

  if (existing.fingerprint !== fp) {
    return res.status(409).type('application/problem+json').json({
      type: 'https://api-contract.com/problems/idempotency-key-reuse',
      title: 'Idempotency key reused with a different request',
      status: 409,
      detail: 'This Idempotency-Key was already used for a different payload. Generate a new key.',
    });
  }

  if (existing.status === 'in_flight') {
    res.set('Retry-After', '1');         // original still processing
    return res.status(409).type('application/problem+json').json({
      type: 'https://api-contract.com/problems/request-in-progress',
      title: 'A request with this idempotency key is already in progress',
      status: 409,
    });
  }

  // Completed with matching fingerprint → replay the cached response.
  return res.status(existing.response_status).json(existing.response_body);
}

Step 3 — Persist the response and release the lock

After the handler succeeds, write the cached response and flip the status to completed. A later replay with the same fingerprint now returns this stored result verbatim.

Python (FastAPI dependency + finalize):

async def finalize(key: str, status_code: int, body: dict):
    await db.execute(
        """UPDATE idempotency_keys
           SET status = 'completed', response_status = :s, response_body = :b
           WHERE key = :k""",
        {"s": status_code, "b": json.dumps(body), "k": key},
    )

async def idempotent_create(key: str, body: PaymentInput):
    fp = fingerprint(body.model_dump())
    won = await db.execute(
        """INSERT INTO idempotency_keys (key, fingerprint, status, expires_at)
           VALUES (:k, :f, 'in_flight', now() + interval '24 hours')
           ON CONFLICT (key) DO NOTHING RETURNING key""",
        {"k": key, "f": fp},
    )
    if not won:
        row = await db.fetch_one(
            "SELECT fingerprint, status, response_status, response_body "
            "FROM idempotency_keys WHERE key = :k", {"k": key})
        if row["fingerprint"] != fp:
            raise HTTPException(409, "Idempotency key reused with a different request")
        if row["status"] == "in_flight":
            raise HTTPException(409, "Request already in progress")
        return JSONResponse(row["response_body"], status_code=row["response_status"])

    result = await charge_card(body)         # the real side effect, run once
    await finalize(key, 201, result)
    return JSONResponse(result, status_code=201)

RFC alignment

Standard callout — RFC 9110 §15.5.10 and the IETF Idempotency-Key draft. 409 Conflict indicates the request conflicts with the current state of the target resource; reusing an idempotency key for a semantically different request is exactly such a conflict. The IETF “Idempotency-Key Header Field” specification defines the header as an opaque client-generated token and directs servers to detect resource replays and to reject a reused key whose request payload differs from the original. 422 is sometimes used for validation-level mismatches, but 409 is the correct code for a key conflict.

Idempotency, safety, and caching implications

Idempotency keys exist to make a non-idempotent method — almost always POST — safe under retries, which is precisely the gap left when you choose POST over PUT for resource creation. The in-flight lock is what upgrades “idempotent when retried sequentially” to “idempotent under concurrency,” which is the case that actually breaks in production when a client fires parallel retries.

Never cache an idempotency-conflict response in a shared cache: a 409 for one client’s key must not be served to another. Set Cache-Control: no-store on all conflict and replay responses. The cached response lives in your idempotency store, keyed by the client’s token, not in HTTP caches.

Key retention window

Keys must outlive the client’s retry horizon. Store an explicit expires_at and reclaim expired rows on a schedule. Too short a window and a slow retry after a network partition re-executes the side effect; too long and the store grows unbounded. A 24-hour to 7-day window covers almost all client retry policies. Document the window so clients understand that a key is only deduplicated within it.

DELETE FROM idempotency_keys WHERE expires_at < now();

SDK and codegen downstream effect

Declaring Idempotency-Key and the 409 response in the OpenAPI contract lets a generated SDK send a key automatically and surface the conflict as a typed error:

# openapi.yaml (OpenAPI 3.1.0) — excerpt
parameters:
  - name: Idempotency-Key
    in: header
    required: true
    schema: { type: string, format: uuid }
responses:
  '409':
    description: Idempotency key reused with a different request, or a request in progress.
    content:
      application/problem+json:
        schema: { $ref: '#/components/schemas/Problem' }

// Generated client attaches a key and retries safely
async function createPayment(input: PaymentInput, key = crypto.randomUUID()) {
  return http.post('/payments', input, { headers: { 'Idempotency-Key': key } });
  // Retries reuse the SAME key; a 409 means the SDK must mint a fresh key for a new payment.
}

The critical SDK rule: a retry must reuse the same key, and a genuinely new operation must use a fresh key. Generators that mint a new key per attempt defeat deduplication entirely — assert single-key-per-logical-request in a contract test.

Common mistakes

FAQ

Related

• Idempotency Key Implementation — the parent reference on designing idempotent write endpoints
• API Design Fundamentals & Architecture — the top-level reference for contract-first API design
• Generating Idempotency Keys in Node.js Express APIs — minting and validating the keys this page deduplicates
• Choosing Between POST and PUT for Resource Creation — when a key is needed versus getting idempotency from PUT