Core Concepts
| Item | Description |
|---|
Idempotency | f(f(x)) = f(x). Applying an operation multiple times produces the same result as applying it once. No additional side effects on retry. |
Idempotency Key | Unique identifier (UUID) sent with each request. Server stores key + response. On retry, server returns stored response instead of re-processing. |
Exactly-Once Semantics | The holy grail: each message processed exactly once. Achieved by combining at-least-once delivery with idempotent processing on the receiver. |
Deduplication | Detecting and ignoring duplicate requests. Can be done at the application layer (idempotency keys) or infrastructure layer (message broker dedup). |
At-Least-Once Delivery | Message brokers guarantee delivery ≥ 1 time. Consumer MUST be idempotent. RabbitMQ, Kafka, SQS all default to at-least-once. |
Exactly-Once Delivery (Kafka) | Kafka transactions + idempotent producer: enable.idempotence=true. Guarantees no duplicates within the producer session. |
HTTP Idempotency
| Method | Idempotent? | Safe? | Semantics |
|---|
| GET | Yes | Yes | Retrieve resource. Multiple identical GETs = same response, no side effects. |
| HEAD | Yes | Yes | Same as GET but no body. Useful for checking existence. |
| PUT | Yes | No | Full replacement. Same body N times = same end state. Required by RFC 7231. |
| DELETE | Yes | No | Delete resource. First call deletes, subsequent calls return 404. End state is the same. |
| OPTIONS | Yes | Yes | Describe communication options. No side effects. |
| POST | No | No | Create resource. Each POST creates a NEW resource. NOT idempotent. Use Idempotency-Key header. |
| PATCH | No | No | Partial update. May or may not be idempotent depending on patch content (use with care). |
Implementation Strategies
| Item | Description |
|---|
Stripe-Style Idempotency Key | Client generates UUID. Sends in `Idempotency-Key` header. Server: (1) check key in DB, (2) if found, return stored response, (3) if new, process + store response + key. Keys expire after 24h. |
Database Unique Constraints | Natural idempotency: INSERT ON CONFLICT DO NOTHING. Use order_id as primary key — second INSERT with same order_id is a no-op. |
Version-Based Updates | UPDATE items SET qty=qty-1, version=version+1 WHERE id=5 AND version=7. If version already incremented, second call affects 0 rows. |
Transactional Outbox | Write event + mark operation as processed in one DB transaction. On retry, check if already processed before producing duplicate events. |
Idempotent Consumer (Kafka) | Store consumed offsets. On rebalance, reprocess from last committed offset. Use message key as dedup key in a processed-messages table. |
Request Fingerprint | Hash(request body + endpoint + timestamp window). If hash seen before → duplicate. Less storage than full key-value store but potential collisions. |
Common Pitfalls
| Item | Description |
|---|
POST without idempotency key | The #1 cause of duplicate orders and double charges. Always include Idempotency-Key on POST that creates resources. |
Idempotency key reuse | Client must generate a NEW key for each distinct request. Reusing the same key for different payloads = first payload wins forever. |
Key expiration too short | If keys expire before the retry window closes, a legitimate retry creates a duplicate. Match expiration to your retry policy max window. |
Non-atomic check-then-act | Race condition: two concurrent requests both see key as new and both process. Use DB-level uniqueness constraints: INSERT fails on key collision. |
Partial failure side effects | Request succeeds partially (payment captured, email not sent). Idempotency key stores success response — retry returns 'success' without sending the email. |
Pro Tip: The golden rule of distributed systems: assume every request will be delivered at least once, and design your handlers to be safe when called multiple times. Idempotency keys are the cheapest distributed safety net you can buy.