Higher-order functions and callbacks

A higher-order function works with other functions: it accepts one as an argument, hands one back as its result, or does both. JavaScript allows this because functions are values: a function can be stored in a variable, kept in an array, passed into a call and returned from one, just like a number or a string. A function that you pass to another function, for it to call, is a callback. Most of the standard library works this way: array.map(fn), array.filter(fn), element.addEventListener('click', fn), setTimeout(fn, 1000) and promise.then(fn) are all higher-order functions, and each decides when to call your function and with which arguments. This lesson shows how to write your own, what the built-ins pass to their callbacks, which explains why ['1', '2', '3'].map(parseInt) gives [1, NaN, NaN], and how to join small functions into pipelines with pipe, compose and tap.

Functions are values

The program below checks passwords against a list of rules, and every rule is a function. Some rules are written directly; others are made by functions that return functions:

// Each rule is a function: it takes a password and returns a message, or null when the password passes.
const hasDigit = (text) => (/\d/.test(text) ? null : 'needs a digit');

// Functions that return functions: each call builds a rule with its own settings.
const minLength = (n) => (text) => (text.length >= n ? null : `needs at least ${n} characters`);
const notOneOf = (banned) => (text) => (banned.includes(text.toLowerCase()) ? 'is too common' : null);

// A function that takes functions: it runs every rule and keeps the messages.
function problemsWith(text, rules) {
  return rules.map((rule) => rule(text)).filter((message) => message !== null);
}

const rules = [minLength(10), hasDigit, notOneOf(['password123', 'qwerty12345'])];
for (const password of ['sunrise', 'Password123', 'mango-tree-42']) {
  const problems = problemsWith(password, rules);
  console.log(password.padEnd(14), problems.length ? problems.join('; ') : 'ok');
}

sunrise        needs at least 10 characters; needs a digit
Password123    is too common
mango-tree-42  ok

• hasDigit is a function stored in a constant, like any other value.
• minLength(10) returns a function. The returned rule keeps n in its closure, so minLength(10) and minLength(8) would be two different rules made by one factory. notOneOf() does the same with a list of banned passwords.
• problemsWith() takes functions: it receives the list of rules and calls each one with the text, through map(), which is itself a higher-order function, then keeps the messages with filter().

Adding a rule means adding a function to the array, without touching problemsWith(). That is the main reason higher-order functions exist: the code that decides when something happens is written once, and the code that decides what happens is passed in. MDN’s glossary calls a language with this property one with first-class functions.

What the built-ins pass to their callbacks

A higher-order function chooses the arguments of every call it makes, and they are often more than the one value you think of. This program passes the same kind of callback to six built-ins. spy(label) returns a function that prints every argument it was given:

// spy(label) returns a callback that prints every argument it receives, then returns the first one.
const spy = (label) => (...args) => {
  console.log(label.padEnd(10), args);
  return args[0];
};

const prices = [120, 80];
prices.map(spy('map'));
prices.filter(spy('filter'));
prices.reduce(spy('reduce'), 0);
Array.from('ab', spy('from'));
'2 kg, 5 kg'.replace(/(\d+) kg/g, spy('replace'));
setTimeout(spy('setTimeout'), 0, 'extra', 'arguments');

map        [ 120, 0, [ 120, 80 ] ]
map        [ 80, 1, [ 120, 80 ] ]
filter     [ 120, 0, [ 120, 80 ] ]
filter     [ 80, 1, [ 120, 80 ] ]
reduce     [ 0, 120, 0, [ 120, 80 ] ]
reduce     [ 0, 80, 1, [ 120, 80 ] ]
from       [ 'a', 0 ]
from       [ 'b', 1 ]
replace    [ '2 kg', '2', 0, '2 kg, 5 kg' ]
replace    [ '5 kg', '5', 6, '2 kg, 5 kg' ]
setTimeout [ 'extra', 'arguments' ]

• map() and filter() call the callback with the element, its index and the whole array, and so do forEach(), find(), some() and every().
• reduce() puts the running result first: the accumulator, then the element, its index and the array. Its first accumulator is the second argument of reduce(), here 0, and spy returns its first argument, so it stays 0.
• Array.from() passes the element and its index to its optional mapping function.
• replace() with a function calls it once for each match, with the matched text, each captured group, the position of the match and the whole string.
• setTimeout() passes the arguments that follow the delay, here 'extra' and 'arguments'. Its callback runs last, after the rest of the program has finished.

The documentation of each method lists these parameters, and a callback may ignore the ones it does not need. The trouble starts when the callback is a function that does not ignore them.

The map(parseInt) trap

parseInt() has two parameters: the text to read, and an optional radix, the base of the number system. Passed straight to map(), it receives the index as its radix:

// map() passes three arguments to its callback, and parseInt() reads the second one as the radix.
console.log(['1', '2', '3'].map(parseInt));
console.log(['10', '10', '10'].map(parseInt));

// What map() really called:
console.log(parseInt('10', 0), parseInt('10', 1), parseInt('10', 2));

// Fix 1: an arrow function that passes exactly the arguments parseInt() should get.
console.log(['1', '2', '3'].map((text) => parseInt(text, 10)));

// Fix 2: Number takes one argument, and ignores the others.
console.log(['1', '2', '3'].map(Number));

// Fix 3, itself a higher-order function: unary(fn) returns a function that passes on only its first argument.
const unary = (fn) => (first) => fn(first);
console.log(['1', '2', '3'].map(unary(parseInt)));

// The extra arguments are always there; console.log simply prints all of them.
['x', 'y'].forEach(console.log);

[ 1, NaN, NaN ]
[ 10, NaN, 2 ]
10 NaN 2
[ 1, 2, 3 ]
[ 1, 2, 3 ]
[ 1, 2, 3 ]
x 0 [ 'x', 'y' ]
y 1 [ 'x', 'y' ]

map() called parseInt('1', 0), parseInt('2', 1) and parseInt('3', 2). A radix of 0 means “work out the base from the text”, which gives base 10 and the value 1. A radix of 1 is outside the allowed range of 2 to 36, so the result is NaN whatever the text. In base 2 only the digits 0 and 1 exist, so '3' is NaN, while '10' in base 2 is 2, which is why the second line prints [10, NaN, 2].

Each fix makes sure parseInt() sees only what it should:

1. A small arrow function, (text) => parseInt(text, 10), names its one argument and passes the radix on purpose.
2. map(Number) works because Number() uses only its first argument. It is a different conversion, though: it needs the whole text to be a number, while parseInt() reads digits from the start and stops at the first other character. The lesson on type conversion compares the two on awkward input.
3. unary(parseInt) is a higher-order fix: unary() takes a function and returns a new one that passes on only its first argument, so the index never reaches the radix. Without a radix, parseInt() reads text that starts with 0x as hexadecimal, so the first fix stays the most explicit.

The last two lines show that nothing is wrong with map() itself. forEach(console.log) prints the element, the index and the array on every line, because console.log() prints every argument it gets. The same trap waits for any function with optional parameters passed to a built-in that sends extra arguments: when in doubt, wrap the call in an arrow function that names its one argument.

Callbacks that run now, and callbacks that run later

Some higher-order functions call your callback before they return; others store it and call it later, after the code that registered it has finished:

// Some built-ins call your function before they return; others keep it and call it later.
console.log('1. start');
[10, 20].forEach((n) => console.log(`2. forEach calls back at once with ${n}`));
setTimeout(() => console.log('5. setTimeout calls back after the script and its promise callbacks'), 0);
Promise.resolve('ok').then((value) => console.log(`4. then calls back once the script has finished: ${value}`));
console.log('3. end of the script');

1. start
2. forEach calls back at once with 10
2. forEach calls back at once with 20
3. end of the script
4. then calls back once the script has finished: ok
5. setTimeout calls back after the script and its promise callbacks

forEach() calls its callback for each element before its own line ends. then() and setTimeout() only store theirs: the promise callback runs as soon as the script has finished, and the timer callback after that, even with a delay of 0. A callback that runs later sees the variables of its closure as they are at that time, not as they were when it was registered. The module on the event loop explains this order in detail.

Methods lose their object when passed as callbacks

A method uses this to reach its object. Passed on its own, as basket.add, it is just a function: the built-in calls it without an object, and in a module this is then undefined:

// A method passed as a callback is called without its object, so `this` is undefined inside it.
const basket = {
  count: 0,
  add() {
    this.count += 1;
  },
};

try {
  ['soap', 'rice'].forEach(basket.add);
} catch (error) {
  console.log('forEach(basket.add):', error.name);
}

['soap', 'rice'].forEach(() => basket.add()); // 1. call the method on its object
['soap', 'rice'].forEach(basket.add.bind(basket)); // 2. bind() fixes this
['soap', 'rice'].forEach(basket.add, basket); // 3. forEach takes the value of this as a second argument
console.log('count after three fixed loops:', basket.count);

forEach(basket.add): TypeError
count after three fixed loops: 6

The first loop fails with a TypeError because this.count reads a property of undefined. The three fixes all make sure the method is called on its object: an arrow function that calls basket.add(), bind(), which returns a new function with this fixed, and the second argument of forEach(), which sets this for every call. Arrow functions are the usual choice, because they read like the call you mean. The lessons on methods and on call, apply and bind cover this in full.

Pipelines: pipe, compose and tap

A pipeline runs a value through a list of small functions, each working on what the previous one returned. A function that builds a pipeline is a higher-order function twice over: it takes the steps, and it returns the combined function.

// pipe(f, g, h) returns one function that runs f, then g on f's result, then h on g's result.
const pipe = (...steps) => (input) => steps.reduce((value, step) => step(value), input);

const trimAll = (names) => names.map((name) => name.trim());
const toTitleCase = (name) => name.toLowerCase().replace(/\b\w/g, (letter) => letter.toUpperCase());
const titleCaseAll = (names) => names.map(toTitleCase);
const unique = (names) => [...new Set(names)];

const typed = [' mumbai', 'PUNE ', 'Mumbai', 'new delhi', ' pune'];

const cleanCities = pipe(trimAll, titleCaseAll, unique);
console.log(cleanCities(typed));

// The same steps in another order: unique runs before the names look alike, so nothing is removed.
const tooEarly = pipe(unique, trimAll, titleCaseAll);
console.log(tooEarly(typed));

[ 'Mumbai', 'Pune', 'New Delhi' ]
[ 'Mumbai', 'Pune', 'Mumbai', 'New Delhi', 'Pune' ]

pipe() collects its steps with a rest parameter and returns a function that uses reduce() to carry one value through them: pipe(trimAll, titleCaseAll, unique)(typed) is unique(titleCaseAll(trimAll(typed))), written in the order it runs.

Order matters. In the second pipeline unique runs first, while ' mumbai' and 'Mumbai' still differ, so every name survives. Two relatives of pipe are worth knowing by name:

• compose(f, g, h) builds the same kind of function but runs the steps from right to left, f(g(h(x))), the order in which mathematics writes composition. compose(unique, titleCaseAll, trimAll) equals the first pipeline.
• tap(fn) makes a step that calls fn with the value, for a side effect such as logging, and passes the value on unchanged, so you can look inside a pipeline without changing what flows through it.

The exercise below asks you to write all three. JavaScript has no pipeline operator yet: a |> operator is a TC39 proposal at stage 2, so helpers like pipe() are how code builds pipelines today.

Keeping callbacks readable

Higher-order code is easy to read when each function does one thing and has a name that says what. A few habits help:

• Name a callback once it grows past a line or two, and pass the name: orders.filter(isOverdue) reads better than a five-line arrow function in the middle of the call, the function can be tested on its own, and a stack trace through it shows isOverdue instead of an anonymous frame.
• Use map() and filter() for their results. When the point of a loop is a side effect, such as saving or printing, forEach() or for…of says so.
• Keep the arguments in mind. Pass your own functions straight to built-ins only when they take one parameter, or when you know what the extra arguments will do.

Exercise

// Three higher-order functions: each takes functions and returns a new function.

/** pipe(f, g, h)(x) is h(g(f(x))); pipe()(x) is x. */
export function pipe(...fns) {
  // Runs only the first function: replace this with a version that runs all of them, left to right.
  return (input) => (fns.length ? fns[0](input) : input);
}

/** compose(f, g, h)(x) is f(g(h(x))); compose()(x) is x. */
export function compose(...fns) {
  return (input) => input;
}

/** tap(fn) calls fn with the value for its side effect, and passes the value on unchanged. */
export function tap(fn) {
  return (value) => fn(value);
}

import { test, assert } from 'mysmartcopilot:test';
import { compose, pipe, tap } from './steps.mjs';

const totals = [450, 1200, 80, 999];
const overHundred = (list) => list.filter((n) => n > 100);
const withTax = (list) => list.map((n) => Math.round(n * 1.18));
const sum = (list) => list.reduce((a, b) => a + b, 0);

test('pipe runs the steps from left to right', () => {
  assert.equal(pipe(overHundred, withTax, sum)(totals), 3126);
  assert.deepEqual(pipe(withTax, overHundred)([80, 90]), [106]);
  assert.deepEqual(pipe(overHundred, withTax)([80, 90]), []);
});

test('compose runs the steps from right to left', () => {
  assert.equal(compose(sum, withTax, overHundred)(totals), 3126);
  assert.deepEqual(compose(overHundred, withTax)([80, 90]), [106]);
  assert.equal(compose((n) => n - 1, (n) => n * 10)(5), 49);
});

test('no functions: the value comes back unchanged', () => {
  assert.equal(pipe()(42), 42);
  assert.equal(compose()('same'), 'same');
});

test('tap passes the value on, whatever its function returns', () => {
  const seen = [];
  const result = pipe(overHundred, tap((list) => seen.push(list.length)), sum)(totals);
  assert.equal(result, 2649);
  assert.deepEqual(seen, [3]);
  assert.equal(tap(() => 'something else')(7), 7);
});

test('a step that returns undefined passes undefined on', () => {
  const steps = [];
  const nothing = () => undefined;
  const describe = (value) => {
    steps.push(value);
    return typeof value;
  };
  assert.equal(pipe(nothing, describe)(10), 'undefined');
  assert.deepEqual(steps, [undefined]);
});

Interview questions

Fresher · What is a higher-order function? Give two examples from JavaScript’s standard library.

A higher-order function is one that receives a function as an argument, gives a function back as its result, or both. JavaScript supports them because functions are first-class values: they can be stored, passed and returned like any other value. Array.prototype.map() is one: it calls the function you pass for every element and builds a new array from the results. setTimeout() is another: it keeps your callback and calls it after the delay. A function that returns a function, such as a factory minLength(8) that returns a validation rule, is higher-order too. They let you write the control logic once, such as looping, waiting or validating, and pass in the part that varies. The thing to watch is which arguments the higher-order function passes to your callback, since a callback with optional parameters, such as parseInt, may misread them.

Key takeaways

• A higher-order function takes a function, returns one, or both; JavaScript allows it because functions are values.
• Built-ins choose the arguments of their callbacks: map(), filter() and forEach() pass the element, its index and the array, and reduce() puts the accumulator first.
• ['1', '2', '3'].map(parseInt) is [1, NaN, NaN] because the index becomes parseInt’s radix; pass (text) => parseInt(text, 10), or Number when the whole text must be a number.
• Some callbacks run before the call returns (forEach), others later (then, setTimeout).
• A method passed as a callback loses its object; call it from an arrow function, or use bind() or forEach’s second argument.
• pipe() runs steps left to right, compose() right to left, and tap() looks at a value without changing it.