IDs, idempotency keys and deduplication

IDs, idempotency keys and deduplication answer one question in three places: how does a design recognise the same thing when it arrives again? An ID names one object for its whole life, and its format decides whether IDs sort by creation time, can be read out over a phone or can be guessed. An idempotency key names one attempt at an operation: the client sends it with a request and again with every retry, and the server answers a retry with the stored reply instead of doing the work twice. Deduplication drops an event that was already handled, by remembering recent event IDs or the last sequence number of each sender, for as long as copies can still arrive.

Why a design meets the same request twice

Networks and people repeat themselves. A phone that sends a payment and hears nothing for 30 seconds cannot tell whether the request was lost or only its reply, so it sends the request again. A message broker that loses an acknowledgement delivers the same event a second time, and people press Pay twice. Retrying is the only safe move for a client that does not know the outcome, so the design has to make the repeat harmless. capture_twice.py sends one card payment twice, because the first reply was lost:

"""One card payment, captured by a client whose first reply was lost, so it sends the same request again.

The trusting service charges every request it receives. The keyed service files each request under the
idempotency key the client sent with it: a retry with the same key gets the stored reply, and a request
that reuses a key for a different payment is refused. Amounts are integer minor units (paise)."""


class Gateway:
    """Stands in for the card network: every call moves money."""

    def __init__(self):
        self.charges = []

    def charge(self, card, amount):
        self.charges.append((card, amount))
        return f"ch_{len(self.charges)}"


class KeyReused(Exception):
    """The key was first used for a different request."""


class TrustingCapture:
    def __init__(self, gateway):
        self.gateway = gateway

    def capture(self, key, card, amount):  # the key is ignored
        return {"charge": self.gateway.charge(card, amount), "amount": amount}


class KeyedCapture:
    def __init__(self, gateway):
        self.gateway = gateway
        self.replies = {}  # idempotency key -> (the request it was first used for, the reply)

    def capture(self, key, card, amount):
        request = (card, amount)
        if key in self.replies:
            first, reply = self.replies[key]
            if first != request:
                raise KeyReused(f"key {key} was first used for amount {first[1]}")
            return reply  # a retry: the stored reply, and no second charge
        reply = {"charge": self.gateway.charge(card, amount), "amount": amount}
        self.replies[key] = (request, reply)
        return reply


for service in (TrustingCapture(Gateway()), KeyedCapture(Gateway())):
    first = service.capture("k-7f3a", "card-4242", 49_900)  # this reply is lost on the way back
    retry = service.capture("k-7f3a", "card-4242", 49_900)  # so the client sends the request again
    taken = sum(amount for _, amount in service.gateway.charges)
    print(f"{type(service).__name__}: first {first['charge']}, retry {retry['charge']}, "
          f"{len(service.gateway.charges)} charge(s), {taken} paise taken")

try:
    service.capture("k-7f3a", "card-4242", 59_900)  # same key, different amount
except KeyReused as refused:
    print("Reused key with another amount: refused,", refused)

TrustingCapture: first ch_1, retry ch_2, 2 charge(s), 99800 paise taken
KeyedCapture: first ch_1, retry ch_1, 1 charge(s), 49900 paise taken
Reused key with another amount: refused, key k-7f3a was first used for amount 49900

The trusting service charged the customer twice. The keyed service charged once, answered the retry with the reply it had stored, and refused a request that reused the key for another amount, because one key names one request. Idempotency keys only work when the service looks the key up before it does any work:

A design that stays correct under repeats rests on a few choices, and each one has a plain job:

• Random IDs (UUID version 4) carry 122 random bits, so any process can make one alone, and they reveal nothing.
• Time-ordered IDs (UUID version 7) start with the creation time in milliseconds, so new IDs sort after old ones.
• Readable codes use a short alphabet without look-alike characters, and each new code is checked for uniqueness.
• Access codes, the ones that open something, come from a secure random generator.
• Request fingerprints show a server that a known key has arrived with a different request.
• Key claims record a key before the work starts, so a second copy in flight finds it.
• Event-ID windows remember recent event IDs and drop copies that arrive within that time.
• Sequence windows keep the highest sequence number of each sender, plus a short record below it.

Random or time-ordered IDs: what an index sees, measured Pro: opens with a Pro, Premium or Ultimate pass