Skip to content

Idempotency

1. What is it? (2-3 lines)

Idempotency is the property of an operation where executing it multiple times with the exact same input produces the exact same system state and output result as executing it once (\(f(f(x)) = f(x)\)).

2. Why does it exist?

In distributed networks, transient timeouts, automatic retries, and At-Least-Once message delivery guarantees make duplicate requests inevitable. Without idempotency, retrying a payment POST request or re-consuming a Kafka message causes double-charging, duplicate order creation, or corrupted database balances.

3. Where is it used?

  • Financial Payment APIs: Stripe / PayPal checkout endpoints (Idempotency-Key HTTP header).
  • Messaging Consumers: Kafka / SQS / RabbitMQ consumers processing retried messages.
  • REST APIs: Naturally idempotent HTTP methods (GET, PUT, DELETE).

4. How does it work? (simple flow)

  • Idempotency Key Pattern (REST API / Kafka Consumer):
  • Client Generation: Client attaches a unique UUID Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000 to the request payload.
  • Server Check-and-Lock: Server executes atomic Redis check: SET idempotency:550e8400 IN_PROGRESS NX EX 86400.
    • If Key Exists & Completed: Server skips execution and returns the stored cached JSON response immediately!
    • If Key Exists & In-Progress: Server returns HTTP 409 Conflict (or blocks/waits).
    • If Key New: Proceed to step 3.
  • Business Processing: Execute DB mutation inside a local ACID transaction.
  • Save Result: Update Redis key state to COMPLETED and cache the final JSON response payload.

5. Advantages

  • Converts unsafe At-Least-Once network delivery into mathematically safe Exactly-Once Processing.
  • Safe network retries: Mobile apps and microservices retry failed HTTP requests safely without side-effects.
  • Protects database data integrity against network glitches and message broker re-deliveries.

6. Limitations

  • Requires state storage (Redis / DB Unique Indexes) to track historical idempotency keys and cached response payloads.
  • Requires managing key TTL expiration windows (e.g. 24-72 hours memory retention).

7. Common interview questions (10)

  1. What is the mathematical definition of Idempotency (\(f(f(x)) = f(x)\))?
  2. Which HTTP methods are naturally idempotent according to RFC 7231 (GET, PUT, DELETE vs POST)?
  3. How does Stripe's Idempotency-Key HTTP header protocol work end-to-end?
  4. How do you implement idempotent processing in a Kafka consumer reading At-Least-Once messages?
  5. Why is checking a key in Redis before starting processing vulnerable to race conditions if not done atomically (SET NX)?
  6. What happens if a duplicate request arrives while the initial request is still IN_PROGRESS?
  7. How does a database UNIQUE constraint (UNIQUE(tenant_id, idempotency_key)) act as a secondary safety net?
  8. What is the difference between Idempotent Delivery (Broker setting) and Idempotent Processing (Consumer logic)?
  9. How do Kafka Idempotent Producers (enable.idempotence = true) deduplicate batches at the broker layer?
  10. How do you handle idempotency when an API calls third-party downstream APIs (e.g. Stripe)?

8. Common misconceptions

  • Misconception: POST requests can never be idempotent. Fact: POST is not naturally idempotent by spec, but incorporating an explicit Idempotency-Key makes POST endpoints fully idempotent.
  • Misconception: Returning an error on retries is idempotent. Fact: Retries of an idempotent operation should return the exact same successful response as the original call!

10. 30-second cheat sheet

  • Formula: \(f(f(x)) = f(x) \implies 1\text{ execution} = 100\text{ retries}\).
  • HTTP Verbs: GET, PUT, DELETE = Naturally Idempotent; POST = Requires Idempotency-Key.
  • Server Execution Flow:
  • Atomic SET key IN_PROGRESS NX EX 86400 in Redis.
  • If exists \(\to\) Return cached response or 409 Conflict.
  • If new \(\to\) Process DB \(\to\) Set key COMPLETED + Store JSON response.
  • Database Safety: Add UNIQUE INDEX (tenant_id, idempotency_key) to DB schema as fallback.