Idempotency Key Implementation

Idempotency is a non-negotiable contract for distributed APIs handling financial transactions, resource provisioning, or any state-mutating workflow where network failures are inevitable. Without an enforced idempotency contract, client retries produce duplicate charges, orphaned resources, and impossible-to-debug state corruption. This page covers the complete implementation surface: OpenAPI contract definition, atomic server-side storage, client-side auto-injection, CI enforcement, and production observability. Part of the API Design Fundamentals & Architecture reference.

Problem Framing

While GET, PUT, and DELETE are inherently idempotent under the HTTP specification, POST and PATCH are not — the server may create or mutate state on every call. In distributed systems, a payment that times out at the load balancer will be retried by the client without knowing whether the server committed it. The Idempotency-Key header is the contract that lets the server detect and suppress the duplicate execution while still returning a correct response to the client.

Without an enforced idempotency contract, the failure modes are silent: double charges process, inventory decrements twice, and audit logs contain phantom records. The HTTP Method Mapping Guidelines cover which HTTP verbs are safe and idempotent by default; this page covers how to make POST and PATCH safe through explicit key tracking.

Spec Definition

Define the Idempotency-Key header as a reusable OpenAPI 3.1 component. This anchors the contract and drives downstream validation, mock server generation, and SDK auto-injection.

# openapi.yaml (OpenAPI 3.1.0)
components:
  parameters:
    IdempotencyKey:
      name: Idempotency-Key
      in: header
      required: true
      description: >
        Client-generated UUID (v4 or v7) that identifies this logical operation.
        The server returns the original response for any duplicate request carrying
        the same key within the key's TTL window.
      schema:
        type: string
        format: uuid
        pattern: '^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'
        examples:
          - 550e8400-e29b-41d4-a716-446655440000

  responses:
    IdempotencyConflict:
      description: >
        A request with this key is already in-flight. The client must wait
        and retry rather than submit a new key.
      content:
        application/problem+json:
          schema:
            $ref: '#/components/schemas/Problem'

paths:
  /v1/payments:
    post:
      operationId: createPayment
      parameters:
        - $ref: '#/components/parameters/IdempotencyKey'
      responses:
        '200':
          description: Payment created (or replayed from cache)
          headers:
            X-Idempotency-Replayed:
              schema:
                type: boolean
              description: true when returning a cached result
        '409':
          $ref: '#/components/responses/IdempotencyConflict'

RFC / Standard Alignment

Implementation Walkthrough — Server Side

Atomic check-and-write is the critical invariant. A non-atomic GET-then-SET creates a race window where two simultaneous retries both observe a cache miss and both execute, defeating the entire contract.

Redis — Lua Script (atomic)

// idempotency-middleware.ts
import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const IDEMPOTENCY_TTL_SECONDS = 86400; // 24 hours

const checkAndSetScript = `
  local existing = redis.call('GET', KEYS[1])
  if existing then
    return existing
  end
  redis.call('SET', KEYS[1], ARGV[1], 'EX', ARGV[2])
  return false
`;

export async function idempotencyMiddleware(req, res, next) {
  const key = req.headers['idempotency-key'];
  if (!key) {
    return res.status(400).json({ type: '/errors/missing-idempotency-key' });
  }

  // Namespace by method + path to prevent cross-endpoint collisions
  const storageKey = `idemp:${req.method}:${req.path}:${key}`;

  // Reserve slot: marks key as IN_FLIGHT while the handler runs
  const placeholder = JSON.stringify({ status: 'IN_FLIGHT' });
  const existing = await redis.eval(
    checkAndSetScript,
    { keys: [storageKey], arguments: [placeholder, String(IDEMPOTENCY_TTL_SECONDS)] }
  );

  if (existing) {
    const cached = JSON.parse(existing as string);
    if (cached.status === 'IN_FLIGHT') {
      return res.status(409).json({
        type: '/errors/idempotency-key-in-flight',
        title: 'Request already in progress',
        detail: 'A request with this key is currently being processed. Retry after a short delay.'
      });
    }
    // Replay the stored response
    res.set('X-Idempotency-Replayed', 'true');
    return res.status(cached.statusCode).json(cached.body);
  }

  // Capture the response so it can be stored for future replays
  const originalJson = res.json.bind(res);
  res.json = async (body) => {
    await redis.setEx(
      storageKey,
      IDEMPOTENCY_TTL_SECONDS,
      JSON.stringify({ statusCode: res.statusCode, body })
    );
    return originalJson(body);
  };

  next();
}

For a complete walkthrough of middleware ordering, header normalisation, and error edge cases in Express, see Generating Idempotency Keys in Node.js Express APIs.

DynamoDB — Conditional Write (AWS)

# idempotency_store.py
import boto3
import json
import time
from botocore.exceptions import ClientError

dynamodb = boto3.client('dynamodb')
TABLE = 'IdempotencyKeys'
TTL_SECONDS = 86400

def check_and_reserve(storage_key: str) -> dict | None:
    """
    Returns None if key was successfully reserved (first execution).
    Returns cached item dict if key was already processed.
    Raises ConditionalCheckFailedException if key is in-flight.
    """
    ttl_epoch = int(time.time()) + TTL_SECONDS
    try:
        dynamodb.put_item(
            TableName=TABLE,
            Item={
                'PK':     {'S': storage_key},
                'Status': {'S': 'IN_FLIGHT'},
                'TTL':    {'N': str(ttl_epoch)},
            },
            ConditionExpression='attribute_not_exists(PK)',
        )
        return None  # Reserved — proceed with execution
    except ClientError as e:
        if e.response['Error']['Code'] != 'ConditionalCheckFailedException':
            raise
        # Key exists — fetch and return cached result
        response = dynamodb.get_item(TableName=TABLE, Key={'PK': {'S': storage_key}})
        return response.get('Item')

def store_result(storage_key: str, status_code: int, body: dict):
    ttl_epoch = int(time.time()) + TTL_SECONDS
    dynamodb.update_item(
        TableName=TABLE,
        Key={'PK': {'S': storage_key}},
        UpdateExpression='SET #s = :s, ResponseBody = :b, #t = :t',
        ExpressionAttributeNames={'#s': 'Status', '#t': 'TTL'},
        ExpressionAttributeValues={
            ':s': {'S': 'COMPLETE'},
            ':b': {'S': json.dumps({'statusCode': status_code, 'body': body})},
            ':t': {'N': str(ttl_epoch)},
        },
    )

Align the PK naming convention with Resource Modeling Best Practices to namespace keys consistently — POST:/v1/payments:<uuid> prevents collisions if the same UUID is reused across different endpoints.

Implementation Walkthrough — Client Side

Manual header injection on every call site is error-prone and breaks retries when a fresh UUID is generated per attempt. Auto-inject at the HTTP client layer so retry logic sees the same key on every attempt.

TypeScript — Axios Interceptor

// idempotent-client.ts
import axios, { AxiosInstance } from 'axios';

const NON_IDEMPOTENT_METHODS = new Set(['post', 'patch']);

export function addIdempotencyInterceptor(client: AxiosInstance): AxiosInstance {
  client.interceptors.request.use((config) => {
    const method = (config.method ?? '').toLowerCase();
    if (NON_IDEMPOTENT_METHODS.has(method)) {
      // Preserve existing key if caller already set one (e.g. for manual retry flows)
      if (!config.headers['Idempotency-Key']) {
        config.headers['Idempotency-Key'] = crypto.randomUUID();
      }
    }
    return config;
  });
  return client;
}

// Retry wrapper that preserves the key across attempts
export async function idempotentPost<T>(
  client: AxiosInstance,
  url: string,
  data: unknown,
  maxRetries = 3
): Promise<T> {
  const key = crypto.randomUUID();
  let delay = 500;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      const { data: result } = await client.post<T>(url, data, {
        headers: { 'Idempotency-Key': key },
      });
      return result;
    } catch (err: any) {
      // 409 IN_FLIGHT: brief wait then retry with same key
      if (err?.response?.status === 409 && attempt < maxRetries) {
        await new Promise((r) => setTimeout(r, delay));
        delay *= 2;
        continue;
      }
      // 5xx: retry with same key (server may not have committed)
      if (err?.response?.status >= 500 && attempt < maxRetries) {
        await new Promise((r) => setTimeout(r, delay));
        delay *= 2;
        continue;
      }
      throw err;
    }
  }
  throw new Error('Max retries exceeded');
}

Python — Requests Session

# idempotent_session.py
import uuid
import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

NON_IDEMPOTENT = {'POST', 'PATCH'}

class IdempotentSession(requests.Session):
    """Automatically injects Idempotency-Key on POST/PATCH and preserves it on retries."""

    def request(self, method: str, url: str, **kwargs):
        if method.upper() in NON_IDEMPOTENT:
            headers = kwargs.setdefault('headers', {})
            # Only generate a new key if the caller hasn't provided one
            if 'Idempotency-Key' not in headers:
                headers['Idempotency-Key'] = str(uuid.uuid4())
        return super().request(method, url, **kwargs)


def build_idempotent_session(retries: int = 3, backoff_factor: float = 0.5) -> IdempotentSession:
    session = IdempotentSession()
    retry = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=[500, 502, 503, 504],
        allowed_methods=['POST', 'PATCH', 'GET'],  # urllib3 must allow POST retries
        raise_on_status=False,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('https://', adapter)
    session.mount('http://', adapter)
    return session

For configuring exponential backoff with correct retry budgets on 5xx responses, see Configuring Exponential Backoff for 5xx Errors.

Edge-Case Handling

Bulk operations. When a POST /v1/payments/batch creates multiple resources atomically, the idempotency key covers the entire batch — not individual items. Store the complete batch response under a single key. If partial failures can occur, document this in your OpenAPI spec explicitly so clients understand that replaying the key returns the original partial result.

In-flight collision (409 storms). Under high concurrency, many clients retrying the same operation will all see IN_FLIGHT and get a 409. Implement jitter in the client backoff so retries stagger. On the server, set a maximum in-flight TTL (e.g. 30 s) so that if the handler crashes mid-execution, the key does not stay locked forever — use a background cleanup job or TTL on the IN_FLIGHT placeholder.

Header stripping by proxies. CORS preflight and some reverse proxies drop non-standard headers. Explicitly whitelist Idempotency-Key in Access-Control-Allow-Headers and in proxy configuration. Test with curl -v through the full proxy chain to confirm the header reaches the application.

Key reuse across API versions. A key generated against /v1/payments must not be treated as equivalent for /v2/payments. Include the path (not just the UUID) in the storage namespace. When issuing API versions, ensure the idempotency store is partitioned by version prefix.

Conditional requests and ETags. If an endpoint uses If-Match / ETag for optimistic locking alongside Idempotency-Key, the idempotency semantics take precedence on replay — return the original response even if the ETag has since changed, because the operation already completed.

Validation and Testing Patterns

Spectral Rule — Enforce Header on POST/PATCH

# spectral.yaml
rules:
  idempotency-header-required:
    message: 'POST and PATCH operations must require an Idempotency-Key header'
    severity: error
    given: '$.paths[*][post,patch]'
    then:
      function: schema
      field: parameters
      functionOptions:
        schema:
          type: array
          contains:
            type: object
            properties:
              name:
                const: Idempotency-Key
              required:
                const: true
            required: [name, required]

  idempotency-replay-header-documented:
    message: 'POST 200 response should document X-Idempotency-Replayed header'
    severity: warn
    given: '$.paths[*].post.responses.200.headers'
    then:
      function: truthy
      field: X-Idempotency-Replayed

GitHub Actions Contract Gate

# .github/workflows/api-contract.yml
name: API Contract Validation
on: [pull_request]

jobs:
  validate-idempotency:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7-alpine
        ports: ['6379:6379']

    steps:
      - uses: actions/checkout@v4

      - name: Lint OpenAPI spec
        run: npx @stoplight/spectral-cli@6 lint openapi.yaml --ruleset spectral.yaml --fail-severity error

      - name: Start mock server
        run: |
          npx @stoplight/prism-cli@5 mock openapi.yaml --port 4010 &
          sleep 2

      - name: First request — expect 200 fresh execution
        run: |
          KEY=$(python3 -c "import uuid; print(uuid.uuid4())")
          STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST http://localhost:4010/v1/payments \
            -H "Idempotency-Key: $KEY" \
            -H "Content-Type: application/json" \
            -d '{"amount": 100, "currency": "USD"}')
          [ "$STATUS" = "200" ] || (echo "Expected 200, got $STATUS" && exit 1)
          echo "IDEMPOTENCY_KEY=$KEY" >> $GITHUB_ENV

      - name: Duplicate request — expect 200 with replay header
        run: |
          REPLAYED=$(curl -s -D - -X POST http://localhost:4010/v1/payments \
            -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
            -H "Content-Type: application/json" \
            -d '{"amount": 100, "currency": "USD"}' | grep -i "x-idempotency-replayed")
          echo "Replay header: $REPLAYED"

      - name: Missing key — expect 400
        run: |
          STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST http://localhost:4010/v1/payments \
            -H "Content-Type: application/json" \
            -d '{"amount": 100, "currency": "USD"}')
          [ "$STATUS" = "400" ] || (echo "Expected 400 for missing key, got $STATUS" && exit 1)

SDK Generation Impact

When Idempotency-Key is defined as required: true in the OpenAPI spec, generated clients behave differently across generator targets:

The Go generator quirk means you must add an additional contract test to confirm missing-key requests return 400 — the type system alone will not catch it. For all generators, pair the required header with an interceptor (as shown above) that auto-injects a UUID when the caller omits it, so SDK consumers are not burdened with UUID generation on every call.

Anti-Patterns

FAQ

Related

• API Design Fundamentals & Architecture — parent reference covering the full design surface
• Generating Idempotency Keys in Node.js Express APIs — step-by-step Express middleware, header normalisation, and response caching
• HTTP Method Mapping Guidelines — which verbs are safe and idempotent by default, and when you need POST with a key
• Configuring Exponential Backoff for 5xx Errors — retry budget, jitter strategies, and backoff ceiling alignment
• RFC 7807 Problem+JSON Implementation — standard error body format for 409 and 400 responses