Idempotency and race conditions in payment APIs: how to test them properly

A payment API has one job it must never get wrong: move the right amount of money exactly once. Networks make that hard. A mobile client sends a payment request, the connection drops before the response arrives, and the client has no idea whether the payment happened. If it retries, it might pay twice. If it doesn't, it might not pay at all.

Idempotency is the standard answer, and almost every payment API now claims to support it. In testing engagements, though, idempotency is still where many serious payment defects turn up. A sequential happy-path test passes, while two requests arriving 3 milliseconds apart create two charges. This article covers what correct behaviour looks like and how to test it with techniques that actually produce concurrency.

What idempotency means for an API

In HTTP terms (RFC 9110), a method is idempotent if repeating the same request has the same intended effect on the server as sending it once. GET, PUT and DELETE are idempotent by definition. POST is not, and payment creation is almost always a POST.

APIs close that gap with an idempotency key: a unique value the client generates (usually a UUID) and sends with the request. The server stores the key together with the result of the first request. Any later request with the same key gets the stored result instead of being executed again. Stripe popularised the pattern. The IETF HTTPAPI working group's draft The Idempotency-Key HTTP Header Field standardises the header name and the main error cases. At the time of writing it is still an Internet-Draft, not an RFC, so treat its status codes as recommendations.

POST /v1/payments HTTP/1.1
Host: api.example.test
Authorization: Bearer <token>
Content-Type: application/json
Idempotency-Key: 9f1c2d4e-7b8a-4c3e-9d21-5a6b7c8d9e0f

{"amount": "25.00", "currency": "EUR", "destination": "acct_test_123"}

HTTP/1.1 201 Created
Content-Type: application/json

{"id": "pay_8XK2", "status": "pending", "amount": "25.00", "currency": "EUR"}

That looks simple, but the semantics hide several decisions your API has to make and document. Each one is a test case.

The semantics you need to pin down

• Replay: a second request with the same key and the same payload returns the original response: same status, same resource ID, ideally byte-identical body. It doesn't create a second resource or return a fresh "200 OK" with different content.
• Key reuse with a different payload: the same key with a different body (for example, a different amount) is almost always a client bug. The server should reject it rather than replay the first response. The IETF draft suggests 422 Unprocessable Content.
• Concurrent duplicates: while the first request is still being processed, a duplicate arrives. The server must not process it in parallel. The draft suggests 409 Conflict. Waiting and then replaying is also valid if documented.
• Missing key: if the endpoint requires a key, a request without one gets 400 Bad Request.
• Scope: keys must be scoped per client or tenant. Two different merchants that happen to send the same UUID must not see each other's responses. That would be both a correctness bug and a data leak.
• Retention: how long keys are stored. Stripe, for example, documents that keys can be removed once they are at least 24 hours old. A retry after expiry will create a new payment, so clients need to know the window.
• Error caching: if the first attempt fails, is the failure stored? Validation errors (4xx) usually should be replayed. For a 5xx where processing may or may not have started, the API must decide whether a retry is allowed to execute, and that decision has to stay consistent with what actually happened downstream.

Sequential tests: the baseline

Start with deterministic, sequential tests for each rule. They are fast, stable and belong in CI on every build:

# pytest -q test_idempotency.py   (needs: pip install pytest httpx)
import os
import uuid

import httpx
import pytest

BODY = {"amount": "25.00", "currency": "EUR", "destination": "acct_test_123"}


@pytest.fixture
def client():
    with httpx.Client(
        base_url=os.environ["BASE_URL"],
        headers={"Authorization": f"Bearer {os.environ['API_TOKEN']}"},
        timeout=30,
    ) as c:
        yield c


def post_payment(client, key, body=BODY):
    headers = {"Idempotency-Key": key} if key else {}
    return client.post("/v1/payments", json=body, headers=headers)


def test_replay_returns_original_response(client):
    key = str(uuid.uuid4())
    first = post_payment(client, key)
    second = post_payment(client, key)

    assert first.status_code == 201
    assert second.status_code == first.status_code
    assert second.json()["id"] == first.json()["id"]


def test_same_key_different_payload_is_rejected(client):
    key = str(uuid.uuid4())
    assert post_payment(client, key).status_code == 201

    changed = {**BODY, "amount": "250.00"}
    r = post_payment(client, key, changed)
    # The IETF draft suggests 422; what matters is that your API documents
    # a specific status and never silently returns the first payment.
    assert r.status_code == 422


def test_missing_key_is_rejected_when_required(client):
    r = post_payment(client, key=None)
    assert r.status_code == 400


def test_new_key_creates_new_payment(client):
    a = post_payment(client, str(uuid.uuid4()))
    b = post_payment(client, str(uuid.uuid4()))
    assert a.json()["id"] != b.json()["id"]

These tests catch missing implementations, wrong status codes and replays that return different IDs. They don't catch race conditions, because each request finishes before the next one starts. Most real duplicate-payment incidents are races.

Testing concurrent duplicates

A typical broken implementation looks like this: check whether the key exists → if not, process the payment → store the key. Two requests arriving together both pass the check before either stores the key, and both process the payment. This is a classic check-then-act race, and a sequential test can't see it.

To test it, you need requests that genuinely overlap on the server. Firing them in a loop isn't enough. Use a barrier, so all requests wait at the starting line and are released together:

"""Fire N identical requests with the same Idempotency-Key at (nearly) the same
instant and assert that at most one payment was created.
Requires Python 3.11+ (asyncio.Barrier) and: pip install httpx"""
import asyncio
import os
import uuid

import httpx

BASE = os.environ["BASE_URL"]
TOKEN = os.environ["API_TOKEN"]
N = 20


async def main() -> None:
    key = str(uuid.uuid4())
    body = {"amount": "25.00", "currency": "EUR", "destination": "acct_test_123"}
    headers = {"Authorization": f"Bearer {TOKEN}", "Idempotency-Key": key}
    barrier = asyncio.Barrier(N)

    limits = httpx.Limits(max_connections=N, max_keepalive_connections=N)
    async with httpx.AsyncClient(base_url=BASE, timeout=30, limits=limits) as client:
        # Warm up N connections first so TLS handshakes don't stagger the burst.
        await asyncio.gather(*(client.get("/health") for _ in range(N)))

        async def one() -> httpx.Response:
            await barrier.wait()  # all N coroutines are released together
            return await client.post("/v1/payments", json=body, headers=headers)

        responses = await asyncio.gather(*(one() for _ in range(N)))

    statuses = sorted(r.status_code for r in responses)
    ids = {r.json()["id"] for r in responses if r.status_code in (200, 201)}
    print("statuses:", statuses)
    print("distinct payment ids:", ids)

    assert len(ids) <= 1, f"duplicate payments created: {ids}"
    assert set(statuses) <= {200, 201, 409}, f"unexpected statuses: {statuses}"


if __name__ == "__main__":
    asyncio.run(main())

Details that make this test reliable:

• Warm connections first. Opening a TLS connection takes tens of milliseconds, often longer than the race window. Without warm-up, the first request is effectively alone.
• One connection per request. With HTTP/1.1, requests on a single connection are serialised. The client's connection limit must be at least N.
• Repeat it. A race that shows up 1 time in 30 is still a production incident at scale. Run the burst 50–100 times with fresh keys and fail on any duplicate.
• Verify the side effects, not just the responses. The responses may look fine while the ledger has two entries. After the burst, query the ledger, the PSP sandbox or the database and assert exactly one movement of money for the key.

For quick manual checks, the shell version works surprisingly well:

# 20 parallel POSTs with the same key; prints a count per status code
KEY=$(uuidgen)
seq 1 20 | xargs -P 20 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST "$BASE_URL/v1/payments" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"amount":"25.00","currency":"EUR","destination":"acct_test_123"}' \
  | sort | uniq -c

A correct implementation prints something like 1 201 and 19 409 (or 20 201 if concurrent duplicates wait and replay). If you see 20 201, check whether all twenty responses carry the same payment ID. If they don't, you have found a duplicate-payment bug.

When the race window is very narrow, a dedicated technique helps. James Kettle's 2023 research on single-packet attacks showed that sending several HTTP/2 requests so they complete in the same TCP packet removes most network jitter. Burp Suite's Repeater ("send group in parallel") and the Turbo Intruder extension implement it. It's useful for security-focused testing of idempotency and limits.

Beyond duplicates: races on shared state

Idempotency keys protect against the same request arriving twice. They do nothing for different requests competing for the same resource. The classic case is a double spend: two withdrawals, each individually valid, race against one balance.

"""Two different withdrawals race against one balance.
Account starts at 100.00 EUR; each withdrawal is 80.00 EUR.
Exactly one may succeed, and the final balance must be 20.00 EUR."""
import asyncio
import os
import uuid

import httpx

BASE, TOKEN = os.environ["BASE_URL"], os.environ["API_TOKEN"]
ACCOUNT = os.environ["TEST_ACCOUNT_ID"]  # seeded with a balance of 100.00


async def main() -> None:
    barrier = asyncio.Barrier(2)
    auth = {"Authorization": f"Bearer {TOKEN}"}
    async with httpx.AsyncClient(base_url=BASE, headers=auth, timeout=30) as client:

        async def withdraw() -> httpx.Response:
            await barrier.wait()
            return await client.post(
                f"/v1/accounts/{ACCOUNT}/withdrawals",
                json={"amount": "80.00", "currency": "EUR"},
                headers={"Idempotency-Key": str(uuid.uuid4())},  # distinct keys!
            )

        r1, r2 = await asyncio.gather(withdraw(), withdraw())
        balance = (await client.get(f"/v1/accounts/{ACCOUNT}")).json()["balance"]

    ok = [r for r in (r1, r2) if r.status_code == 201]
    print(r1.status_code, r2.status_code, "balance:", balance)
    assert len(ok) == 1, "both or neither withdrawal succeeded"
    assert balance == "20.00", f"lost update or overdraft: balance={balance}"


if __name__ == "__main__":
    asyncio.run(main())

Note that the two requests use different keys. From the API's point of view they are legitimate, distinct operations. If both succeed and the balance ends at −60.00, the system has a lost-update or overdraft bug. Common causes are a read-modify-write on the balance without row locking, and a balance check that runs in a separate transaction from the debit.

Other shared-state races worth testing in payment systems:

• Limits: daily spending limits and velocity rules checked against counters that are updated after the payment.
• Refunds: two partial refunds that together exceed the original amount.
• Capture/void: a capture and a void arriving together for the same authorisation.
• Coupons and one-time tokens: a single-use code redeemed twice in parallel.
• Webhooks: the same provider event delivered twice (providers retry by design), processed by two workers at once. Deduplicate on the provider's event ID with a unique constraint, not an in-memory set.

What correct server-side handling looks like

As a tester you don't have to implement the fix, but knowing the robust pattern helps you aim tests at the weak spots. The reliable approach is to let the database decide who wins, atomically:

CREATE TABLE idempotency_keys (
  tenant_id     uuid        NOT NULL,
  key           text        NOT NULL,
  request_hash  bytea       NOT NULL,          -- hash of method + path + body
  status        text        NOT NULL DEFAULT 'in_progress',  -- or 'completed'
  response_code int,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, key)
);

-- First writer wins. A concurrent duplicate gets zero rows back and must
-- then read the existing row: in_progress -> 409, completed -> replay.
INSERT INTO idempotency_keys (tenant_id, key, request_hash)
VALUES ($1, $2, $3)
ON CONFLICT (tenant_id, key) DO NOTHING
RETURNING key;

The first request inserts the row and proceeds. Concurrent duplicates hit the primary-key conflict, get no row back, read the existing row, and return 409 while it is in_progress or the stored response once it is completed. The request hash enables the "same key, different payload" check.

That suggests targeted tests:

• Crash in the middle. Kill the worker (or inject a fault) after the key is stored as in_progress but before completion. Does a retry stay stuck on 409 forever? A robust implementation has a recovery path, such as a lock timeout or reconciliation against the downstream provider.
• Timeout then retry. Make the downstream PSP stub respond after the client's timeout. The client retries with the same key. The result must be one payment, and the retry must eventually return the original outcome.
• Downstream idempotency. When your service calls a PSP, does it pass its own idempotency key on? Otherwise your API is idempotent but your provider charges twice when your service retries.
• Key scope. Send the same key from two tenants. Both must succeed independently and neither may see the other's response.

Don't forget the client side

Server-side idempotency only helps if clients use it correctly, and client bugs are common. Review and test the SDKs and apps that call the API:

• Generate the key once per logical operation, before the first attempt, and reuse it for every retry of that operation. A client that generates a fresh UUID on each retry has turned idempotency off without anyone noticing.
• Persist the key if the operation can outlive the process: a mobile app that is killed mid-payment should retry with the same key after restart.
• Retry only what is safe to retry: network errors, timeouts, 409 and 5xx with backoff and jitter. Don't retry 422 or other 4xx errors blindly.
• Never derive keys from payload content alone. Two genuinely separate payments of 25.00 EUR to the same payee would collide.

Putting it into a test plan

A compact plan for a payment endpoint:

1. Sequential semantic tests (replay, payload mismatch, missing key, scope, expiry) in CI on every build.
2. Concurrent-duplicate bursts with warm connections, repeated 50–100 times, with ledger assertions. Run nightly or before each release.
3. Shared-state race tests for balances, limits, refunds and capture/void.
4. Fault injection: worker crash mid-request, downstream timeout, duplicate webhooks.
5. Reconciliation after a load test: ledger totals versus PSP sandbox reports, mismatches listed per transaction.

None of these tests need exotic tooling. They need deliberate concurrency, assertions on side effects rather than status codes, and enough repetition to catch a race that only shows up occasionally. If your current suite has none of them, the duplicate-payment bug may already be there, just not found yet.