Two-way generators: next, return, throw, yield*

A generator’s yield works in both directions. It sends a value out to the caller, and when the caller resumes the generator with next(value), the paused yield expression evaluates to that value inside the generator. The two other methods of a generator object resume it differently: return(value) makes the paused yield act like a return statement, so the generator finishes after running its finally blocks, and throw(error) makes it act like a throw statement, so the generator can catch the error or let it out. yield* hands the whole conversation to another generator, or walks any other iterable. Together these turn a generator from a value factory into a small process that a caller can talk to: a running total, a parser that can put a token back, or a state machine driven by events. This lesson traces where every value lands, reproduces the cleanup that never happens, and builds a crossing light.

Sending values in with next(value)

This generator keeps a running total of the amounts it is sent. Read const amount = yield total as “hand out the total, then wait; whatever the next next() brings is the amount”:

// next(value) resumes the generator, and the paused yield evaluates to that value.
function* till() {
  let total = 0;
  console.log('  till opened');
  for (;;) {
    const amount = yield total; // hands out the total, then waits for the next amount
    total += amount;
    console.log(`  received ${amount}`);
  }
}

const register = till();
console.log(register.next('ignored')); // starts the body: no yield is waiting for this value
console.log(register.next(250));
console.log(register.next(120));
console.log(register.next(30));

  till opened
{ value: 0, done: false }
  received 250
{ value: 250, done: false }
  received 120
{ value: 370, done: false }
  received 30
{ value: 400, done: false }

The first call, next('ignored'), started the body: it printed “till opened” and ran to the first yield total, answering with 0. Its argument was lost, because no yield was paused yet to receive it. That is why code that sends values into a generator usually makes one empty next() call first, to bring the body to its first yield. Every later next(value) resumes the paused yield, which evaluates to the value, and the call returns whatever the generator yields next. So the answer to next(250) is already the new total, 250, and next(120) gives 370.

yield binds loosely, like an assignment: yield total + 1 yields total + 1. Inside a larger expression, put it in parentheses, as in (yield total) * 2.

Finishing from outside: return() and throw()

return(value) and throw(error) resume a paused generator just as next() does, but the paused yield behaves as if a return or a throw statement stood in its place:

// return() and throw() resume a paused generator too: one finishes it, the other raises an error at the yield.
function* steps() {
  try {
    yield 'step 1';
    yield 'step 2';
    yield 'step 3';
  } finally {
    console.log('  finally runs');
  }
}

const a = steps();
console.log(a.next());
console.log(a.return('stopped')); // the finally block runs first
console.log(a.next());

const b = steps();
b.next();
try {
  b.throw(new Error('disk full')); // nothing inside catches it, so it comes back out
} catch (error) {
  console.log('caller caught:', error.message);
}
console.log(b.next());

// A generator that catches the error decides what happens next.
function* retries() {
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      yield `attempt ${attempt}`;
    } catch (error) {
      console.log(`  attempt ${attempt} failed: ${error.message}`);
    }
  }
}
const c = retries();
console.log(c.next().value);
console.log(c.throw(new Error('timeout')).value); // caught inside: the reply is the next yield
console.log(c.next());

// Before the first next(), the body has not started, so there is no try block to run.
console.log(steps().return('never started'));
try {
  steps().throw(new Error('too early'));
} catch (error) {
  console.log('thrown straight back:', error.message);
}

{ value: 'step 1', done: false }
  finally runs
{ value: 'stopped', done: true }
{ value: undefined, done: true }
  finally runs
caller caught: disk full
{ value: undefined, done: true }
attempt 1
  attempt 1 failed: timeout
attempt 2
{ value: 'attempt 3', done: false }
{ value: 'never started', done: true }
thrown straight back: too early

• a.return('stopped') finished the generator at its first yield. The finally block ran first, and the call returned { value: 'stopped', done: true }. After that, next() only ever answers done: true.
• b.throw(new Error('disk full')) raised the error at the paused yield. Nothing inside caught it, so the finally block ran, the error came back out of the throw() call to the caller, and the generator was finished.
• retries catches the error around its yield, so the generator decides what happens next. Its loop continues, and the throw() call returns the next yielded value, attempt 2. A caller can use this to tell a generator that the last value it handed out failed.
• Before the first next(), the body has not started, so there is no try block to run. return() finishes the generator at once, and throw() throws the error straight back to the caller.

On a generator that has already finished, return(value) returns { value, done: true } and throw(error) throws the error.

Cleanup belongs in finally

A generator that opens something, such as a file, a database cursor or a lock, must close it whether the consumer reads to the end or stops early. A finally block does that, because return() runs it:

// Cleanup in finally runs whenever the consumer stops early, as long as someone calls return().
function* readRows(file) {
  console.log(`  open ${file}`);
  try {
    for (let row = 1; row <= 1000; row++) yield `${file} row ${row}`;
  } finally {
    console.log(`  close ${file}`);
  }
}

for (const row of readRows('orders.csv')) {
  console.log(row);
  if (row.endsWith('row 2')) break; // break calls return(), which runs finally
}

const [header] = readRows('users.csv'); // destructuring closes it too
console.log(header);

try {
  for (const row of readRows('refunds.csv')) throw new Error(`cannot parse ${row}`);
} catch (error) {
  console.log('caught:', error.message);
}

const abandoned = readRows('audit.csv');
console.log(abandoned.next().value);
console.log('the program moves on: audit.csv is never closed');

  open orders.csv
orders.csv row 1
orders.csv row 2
  close orders.csv
  open users.csv
  close users.csv
users.csv row 1
  open refunds.csv
  close refunds.csv
caught: cannot parse refunds.csv row 1
  open audit.csv
audit.csv row 1
the program moves on: audit.csv is never closed

Three consumers stopped early, and all three closed their file. The orders.csv loop broke after two rows, and for…of called return(). The destructuring of users.csv took one row and closed the generator before printing it, which is why “close users.csv” appears before the header. The refunds.csv loop was left by an exception, and for…of closed the iterator before the error reached the catch.

The fourth generator was never closed. The program took one row from audit.csv with next() and moved on, and nothing calls return() for it later: a generator that becomes garbage simply disappears, and its finally block never runs. Whoever drives a generator by hand owns its cleanup. Call return() when you stop, ideally from your own finally, or consume the generator with for…of, which does it for you.

Two rules keep finally blocks honest. Never yield inside one: return() would then stop at that yield and answer done: false instead of finishing. And keep cleanup short and safe, because it also runs while an error from the loop body is on its way out.

A state machine driven by events

A generator remembers where it paused, so the position in its code can be the state of a machine, and each next(event) is a transition. This pedestrian crossing turns green, then waits for its button:

// A pedestrian crossing as a generator: the light's state is the place where the generator is paused.
function* crossingLight() {
  for (;;) {
    try {
      let event = yield 'green';
      while (event !== 'walk') event = yield 'green'; // cars keep going until someone presses the button
      yield 'amber';
      yield 'red: cross now';
      yield 'red: cross now';
    } catch (fault) {
      let event = yield `flashing amber (${fault.message})`;
      while (event !== 'repaired') event = yield 'flashing amber';
    }
  }
}

const light = crossingLight();
const show = (label, result) => console.log(`${label.padEnd(9)} → ${result.value}`);

show('start', light.next());
for (const event of ['tick', 'walk', 'walk', 'tick', 'tick']) show(event, light.next(event));
show('fault', light.throw(new Error('lamp failed')));
show('tick', light.next('tick'));
show('repaired', light.next('repaired'));
show('tick', light.next('tick'));

start     → green
tick      → green
walk      → amber
walk      → red: cross now
tick      → red: cross now
tick      → green
fault     → flashing amber (lamp failed)
tick      → flashing amber
repaired  → green
tick      → green

Read the generator from top to bottom and it describes the light’s behaviour as a story. It stays green, whatever ticks arrive, until a walk event; then it shows amber for one event and red for two, and the loop starts again. The second walk arrived during amber and changed nothing, because no code at that point looks at the event. A fault is sent with throw(): the error lands at whichever yield the light is paused at, the catch block switches to flashing amber, and only a repaired event brings it back to green.

Compared with a state variable and a switch over it, the generator version keeps sequences readable and needs no transition table. Its weakness is that its state is invisible: a paused generator cannot be inspected, saved to storage or sent over the network, and a page reload loses it. When a machine’s state must be shown, stored or restored, keep it in a plain variable.

Delegating with yield*

yield* iterable yields every value of another iterable, and its own value is that iterable’s return value. With a generator on the other side it is more than a loop: while the inner generator runs, every next(value), throw() and return() of the caller goes straight to it.

// yield* hands the conversation to another generator, or walks any other iterable.
function* inner() {
  const reply = yield 'inner asks';
  console.log(`  inner got: ${reply}`);
  return 'inner result';
}

function* outer() {
  yield 'outer starts';
  const result = yield* inner(); // next(value), throw() and return() go to inner while it runs
  console.log(`  yield* evaluated to: ${result}`);
  yield* ['a', 'b'];
  yield* 'hi';
}

const g = outer();
console.log(g.next().value);
console.log(g.next().value);
console.log(g.next('hello from outside').value);
console.log([...g]);

outer starts
inner asks
  inner got: hello from outside
  yield* evaluated to: inner result
a
[ 'b', 'h', 'i' ]

The value sent with next('hello from outside') reached inner, two levels down, without outer doing anything, and inner’s return 'inner result' became the value of the yield* expression in outer. Then yield* walked an array and a string like any for…of loop would.

Delegation makes recursive data easy to walk lazily, because a generator can delegate to another call of itself:

// A recursive generator: yield* walks each subfolder with the same function.
const project = {
  name: 'shop',
  children: [
    { name: 'src', children: [{ name: 'cart.mjs' }, { name: 'checkout.mjs' }] },
    { name: 'tests', children: [{ name: 'cart.test.mjs' }, { name: 'checkout.test.mjs' }] },
    { name: 'package.json' },
  ],
};

function* outline(node, depth = 0) {
  yield `${'  '.repeat(depth)}${node.children ? `${node.name}/` : node.name}`;
  for (const child of node.children ?? []) yield* outline(child, depth + 1);
}

function* files(node, path = '') {
  const here = `${path}${node.name}`;
  if (!node.children) yield here;
  else for (const child of node.children) yield* files(child, `${here}/`);
}

for (const line of outline(project)) console.log(line);

let visited = 0;
for (const file of files(project)) {
  visited += 1;
  if (file.endsWith('.test.mjs')) {
    console.log(`first test file: ${file}, found after ${visited} files`);
    break; // closes every generator in the chain of yield*
  }
}

shop/
  src/
    cart.mjs
    checkout.mjs
  tests/
    cart.test.mjs
    checkout.test.mjs
  package.json
first test file: shop/tests/cart.test.mjs, found after 3 files

outline yields one line per folder or file, indented by depth, and yield* outline(child, depth + 1) lets each subfolder produce its own lines. files walks the same tree for file paths only. The search for the first test file stopped after three files, and its break closed every generator in the chain, from the deepest yield* up to the outermost call, so nothing walked the rest of the tree.

Generators and async/await

Before JavaScript had async functions, libraries used generators for asynchronous code. The generator yielded a promise, and a small runner waited for it and resumed the generator with next(result), or with throw(error) when the promise was rejected. This runner is about 20 lines:

// How generators once stood in for async functions: a runner resumes the generator with each promise's result.
function run(generatorFunction) {
  return new Promise((resolve, reject) => {
    const gen = generatorFunction();
    function step(method, input) {
      let result;
      try {
        result = gen[method](input); // next(value) or throw(error)
      } catch (error) {
        reject(error);
        return;
      }
      if (result.done) resolve(result.value);
      else Promise.resolve(result.value).then((value) => step('next', value), (error) => step('throw', error));
    }
    step('next', undefined);
  });
}

const findUser = (id) => (id > 0 ? Promise.resolve({ id, name: `user ${id}` }) : Promise.reject(new Error(`no user ${id}`)));

const fromGenerator = await run(function* () {
  const user = yield findUser(7);
  console.log('generator got', user.name);
  try {
    yield findUser(-1);
  } catch (error) {
    console.log('generator caught:', error.message);
  }
  return 'generator done';
});
console.log(fromGenerator);

// The same steps with async and await, which the language has had since ES2017.
async function lookUp() {
  const user = await findUser(7);
  console.log('async got', user.name);
  try {
    await findUser(-1);
  } catch (error) {
    console.log('async caught:', error.message);
  }
  return 'async done';
}
console.log(await lookUp());

generator got user 7
generator caught: no user -1
generator done
async got user 7
async caught: no user -1
async done

The generator and the async function print the same steps: the yield and the await both pause until the promise settles, a rejection arrives as an exception at that point and can be caught, and the final return becomes the result. Async functions arrived in ES2017 and replaced this use of generators: they need no runner, await works in any expression, and engines and debuggers know what they are. Generators kept their main job, lazy sequences and iterables, and the two ideas meet again in async function*, the async generators of the module on asynchronous JavaScript.

Exercises

/**
 * @typedef {{ type: 'number', value: number } | { type: 'operator' | 'paren', value: string }} Token
 */

/**
 * The tokens of an arithmetic expression, from left to right. After next('back'), the same token comes again.
 * @param {string} source
 * @returns {Generator<Token, void, string | undefined>}
 */
export function* tokenize(source) {}

import { test, assert } from 'mysmartcopilot:test';
import { tokenize } from './tokens.mjs';

const num = (value) => ({ type: 'number', value });
const op = (value) => ({ type: 'operator', value });
const paren = (value) => ({ type: 'paren', value });

test('an expression becomes numbers, operators and parentheses', () => {
  assert.deepEqual([...tokenize('12 + 3*(4-1)')], [num(12), op('+'), num(3), op('*'), paren('('), num(4), op('-'), num(1), paren(')')]);
});

test('decimals, tabs and extra spaces', () => {
  assert.deepEqual([...tokenize('  0.5\t* 10 / 2.25 ')], [num(0.5), op('*'), num(10), op('/'), num(2.25)]);
});

test('an empty expression has no tokens', () => {
  assert.deepEqual([...tokenize('')], []);
  assert.deepEqual([...tokenize('   ')], []);
});

test("next('back') gives the same token again", () => {
  const tokens = tokenize('7 - 2');
  assert.deepEqual(tokens.next().value, num(7));
  assert.deepEqual(tokens.next('back').value, num(7));
  assert.deepEqual(tokens.next('back').value, num(7));
  assert.deepEqual(tokens.next().value, op('-'));
  assert.deepEqual(tokens.next().value, num(2));
  assert.equal(tokens.next().done, true);
});

test('an unknown character is a SyntaxError, after the tokens before it', () => {
  const tokens = tokenize('2 + x');
  assert.deepEqual(tokens.next().value, num(2));
  assert.deepEqual(tokens.next().value, op('+'));
  assert.throws(() => tokens.next(), SyntaxError);
});

test('a malformed number is a SyntaxError, without its first part', () => {
  const tokens = tokenize('4 * 1.2.3');
  assert.deepEqual(tokens.next().value, num(4));
  assert.deepEqual(tokens.next().value, op('*'));
  assert.throws(() => tokens.next(), SyntaxError);
  assert.throws(() => tokenize('5. + 1').next(), SyntaxError);
});

Interview questions

2 to 5 years · What does the value passed to next() do in a generator?

next(value) resumes a paused generator, and the yield expression where it paused evaluates to value. So in const amount = yield total, the generator hands out total and, when the caller next calls next(250), amount becomes 250 and the body continues until the next yield, whose value becomes the result of that same next(250) call. The first next() call only starts the body, so its argument is ignored: no yield is waiting yet. That is why code that feeds values into a generator usually calls next() once with no argument first. return(value) and throw(error) resume the same paused yield but make it behave like a return or a throw statement, which is how a caller finishes a generator or reports a failure into it.

Key takeaways

• next(value) resumes a generator, and the paused yield evaluates to value; the first next() only starts the body, so its argument is lost.
• return(value) finishes a paused generator after running its finally blocks; throw(error) raises the error at the paused yield, where the generator may catch it and carry on.
• for…of and destructuring call return() when they stop early, so cleanup in finally runs; a generator driven by hand and then abandoned is never cleaned up.
• Never yield inside finally.
• yield* forwards next, throw and return to the inner generator and evaluates to its return value; recursive yield* walks trees lazily.
• A generator makes a readable state machine whose state is the position in its code, but that state cannot be inspected or saved.
• async and await replaced generator runners for asynchronous code; generators remain the tool for lazy sequences.