Race conditions, TOCTOU and double-spend

A wallet holds 100 credits, and two requests to spend 80 arrive a few milliseconds apart. Each one checks the balance, sees 100 and goes ahead, so the wallet pays out 160 credits and ends at −60. Race conditions like this one let the order in which two operations happen decide the result, and some orders break a rule that each request kept on its own.

You will see why a single-threaded JavaScript server races too, reproduce a race on purpose in a test, and fix each kind of race with the control that fits it.

A race is a check and an act with a gap between them

MITRE’s list of weaknesses gives the pattern a name: CWE-367, time-of-check to time-of-use, or TOCTOUTOCTOU: Time-of-check to time-of-use (CWE-367): code checks the state of something, then uses it, and the state changes in between, so the check no longer holds when the code acts on it.. The code checks the state of something, then uses it, and the state changes in between. TOCTOU is a child of CWE-362, the general race conditionrace condition: A weakness in which the result depends on the order in which two operations run, and some orders break a rule, such as two withdrawals that both pass the balance check before either subtracts (CWE-362)., in which two operations share a resource without proper synchronisation. A double-spenddouble-spend: The same balance, coupon or ticket used twice because two requests passed the same check before either recorded its use. It is the money form of a race condition. is the money version: the same balance, coupon or ticket spent twice because two requests passed the same check.

JavaScript runs one piece of code at a time, which makes race conditions look impossible in Node.js. They are not: every await hands control back to the event loop, which runs other requests while this one waits for its database. The program below starts two withdrawals with Promise.all, first with an await between the check and the act, then with one call that checks and changes together:

// Two withdrawals of 80 credits from a balance of 100, started at the same time with Promise.all.
// The store is in memory, but every call is async, as a database call is, so each await lets the other request run.
function makeStore(balance) {
  const log = [];
  return {
    log,
    async read(who) {
      log.push(`${who} reads ${balance}`);
      return balance;
    },
    async recordPayout(who, amount) {
      log.push(`${who} records a payout of ${amount}`);
    },
    async subtract(who, amount) {
      balance -= amount;
      log.push(`${who} subtracts ${amount}: balance ${balance}`);
    },
    // Checks and changes in one step, with no await in between: what UPDATE ... WHERE balance >= ? does.
    async subtractIfAtLeast(who, amount) {
      const ok = balance >= amount;
      if (ok) balance -= amount;
      log.push(`${who} ${ok ? `subtracts ${amount}: balance ${balance}` : `is refused: balance ${balance}`}`);
      return ok;
    },
    get balance() {
      return balance;
    },
  };
}

async function weakWithdraw(store, who, amount) {
  const balance = await store.read(who); // check ...
  if (balance < amount) return 'insufficient';
  await store.recordPayout(who, amount); // ... a gap ...
  await store.subtract(who, amount); // ... act
  return 'ok';
}

async function fixedWithdraw(store, who, amount) {
  if (!(await store.subtractIfAtLeast(who, amount))) return 'insufficient';
  await store.recordPayout(who, amount);
  return 'ok';
}

for (const [name, withdraw] of [['weak', weakWithdraw], ['fixed', fixedWithdraw]]) {
  const store = makeStore(100);
  const results = await Promise.all([withdraw(store, 'A', 80), withdraw(store, 'B', 80)]);
  console.log(`${name}: ${results.join(', ')}; balance ${store.balance}`);
  for (const line of store.log) console.log(`  ${line}`);
}

weak: ok, ok; balance -60
  A reads 100
  B reads 100
  A records a payout of 80
  B records a payout of 80
  A subtracts 80: balance 20
  B subtracts 80: balance -60
fixed: ok, insufficient; balance 20
  A subtracts 80: balance 20
  B is refused: balance 20
  A records a payout of 80

The weak run paid both withdrawals and ended at −60, a double-spend of 80 credits; the fixed run paid one and ended at 20. Step through the weak run one message at a time, then the fix:

Neither request did anything wrong on its own. Each checked the balance it had read, and each read was true when it happened; the rule broke only because the two ran together. So the fix is never a better check, but a check that nothing can come between. Every fix closes the gap, and four controls cover most races:

• A conditional updateconditional update: One database statement that checks and changes together, such as UPDATE accounts SET balance = balance - 80 WHERE id = 1 AND balance >= 80. Its row count says whether the change happened. checks and changes in one database statement, and its row count says whether it happened.
• A unique constraint makes the database refuse a second row for a rule such as once per student.
• A transaction with a row lock makes the second request wait until the first has finished.
• An idempotency keyidempotency key: A unique value a client sends with every attempt of one operation, so that the server can recognise a retried request and return the first answer instead of doing the work again. lets the server recognise a retried request and return its first answer.

The fixed run above is a conditional update in miniature: one call that subtracts only while the balance still covers the amount. A unique constraint and a row lock are kept by the database itself, whichever request reaches it first. An idempotency key answers a different duplicate, the same request sent again by a retry or a double tap.

Reproducing a race on purpose, every time Pro: opens with a Pro, Premium or Ultimate pass