Unit tests and expect tests

Split Rs 10 three ways with integer division and each share comes out as 333 paise: the shares add up to 999, one paisa has gone, and the program reports nothing. Unit tests and expect tests are two ways of making that missing paisa fail the build. A unit testunit test: A test that calls one function or a small piece of code directly and checks its result against a value the test names, so a wrong result fails the run. calls a function and checks its result against a value you name, here 1,000 paise; an expect testexpect test: A test that runs code which prints and compares the printout with a copy kept in the test file; a different printout is shown as a difference and fails the run. prints what the code produces and keeps that printout, in the test file, as the output to expect next time.

Below, an Alcotest suite catches the missing paisa in a recorded run, and a table compares the three kinds of test used here. Two later parts go further: expect tests with ppx_expect, where a hasty dune promote freezes a bug, and a cram test of the ledger’s command line, run with every other test by one dune test.

An Alcotest unit test names the value it expects

The ledger library of your first dune project gains a split function. Amounts are whole paise, 100 to the rupee, and split amount n returns n shares computed with /, which drops any remainder:

(* Amounts are whole paise: 100 paise make one rupee. *)

(* The sum of a list of amounts. *)
let total entries = List.fold_left ( + ) 0 entries

(* Splits a bill into n equal shares. *)
let split amount n = List.init n (fun _ -> amount / n)

The test file holds three cases in two groups; each calls the library and hands the result to a check. Alcotest.(check int) "Rs 10 three ways" 1000 actual takes four things in this order: a testable, here int, that says how to compare and print two values; a message for the report; the expected value; and the value the code computed. Alcotest.test_case gives each case a name and a speed, `Quick for one that runs every time, and Alcotest.run "ledger" runs every group:

(* Unit tests of the ledger library: each test case checks one result. *)

let test_total () =
  Alcotest.(check int) "Rs 16.25 in all" 1625 (Ledger.total [ 1200; 350; 75 ])

let test_split_even () =
  Alcotest.(check (list int)) "Rs 9 three ways" [ 300; 300; 300 ] (Ledger.split 900 3)

let test_split_keeps_every_paisa () =
  Alcotest.(check int) "Rs 10 three ways" 1000 (Ledger.total (Ledger.split 1000 3))

let () =
  Alcotest.run "ledger"
    [
      ("total", [ Alcotest.test_case "three entries" `Quick test_total ]);
      ( "split",
        [
          Alcotest.test_case "an even split" `Quick test_split_even;
          Alcotest.test_case "no paisa lost" `Quick test_split_keeps_every_paisa;
        ] );
    ]

Step through the run case by case, then read it as it was recorded:

from session import files, run

# The ledger project with a split that drops the paise left over, and three Alcotest cases.
files(
    "split/dune-project=split/dune-project",
    "split/lib/dune=split/lib/dune",
    "split/lib/ledger.ml=split/lib/ledger.ml",
    "split/test/dune=split/test/dune",
    "split/test/test_ledger.ml=split/test/test_ledger.ml",
)
run(
    """
cd split
!dune test
""",
    mask=[
        (r"This run has ID `[A-Z0-9]+'", "This run has ID `(new on every run)'"),
        (r"Logs saved to `[^']*'", "Logs saved to `(a file under _build)'"),
        (r"Full test results in `[^']*'", "Full test results in `(a folder under _build)'"),
        (r"in \d+\.\d+s", "in (time)s"),
    ],
)

$ cd split
$ dune test
File "test/dune", line 2, characters 7-18:
2 |  (name test_ledger)
           ^^^^^^^^^^^
Testing `ledger'.
This run has ID `(new on every run)'.

  [OK]          total          0   three entries.
  [OK]          split          0   an even split.
> [FAIL]        split          1   no paisa lost.

┌──────────────────────────────────────────────────────────────────────────────┐
│ [FAIL]        split          1   no paisa lost.                              │
└──────────────────────────────────────────────────────────────────────────────┘
ASSERT Rs 10 three ways
FAIL Rs 10 three ways

   Expected: `1000'
   Received: `999'

Raised at Alcotest_engine__Test.check in file "src/alcotest-engine/test.ml", lines 216-226, characters 4-19
Called from Alcotest_engine__Core.Make.protect_test.(fun) in file "src/alcotest-engine/core.ml", line 186, characters 17-23
Called from Alcotest_engine__Monad.Identity.catch in file "src/alcotest-engine/monad.ml", line 24, characters 31-35

Logs saved to `(a file under _build)'.
 ──────────────────────────────────────────────────────────────────────────────

Full test results in `(a folder under _build)'.
1 failure! in (time)s. 3 tests run.
[exit status 1]

Two cases passed and one failed. The box names the failed case by group, number and name, split 1 no paisa lost, then its message and both values: expected 1000, received 999. The Raised at lines are Alcotest’s own. dune’s first lines point at the stanza that declared the test, and dune test ended with status 1. The fix gives the remainder, 1000 mod 3, to the first shares, one paisa each, and the same suite passes:

(* Amounts are whole paise: 100 paise make one rupee. *)

(* The sum of a list of amounts. *)
let total entries = List.fold_left ( + ) 0 entries

(* Splits a bill into n shares that differ by at most one paisa: the first
   [amount mod n] shares take one paisa more, so the shares add up to [amount]. *)
let split amount n =
  let share = amount / n and extra = amount mod n in
  List.init n (fun i -> if i < extra then share + 1 else share)

from session import files, run

# The same project and tests, with the fixed split: the first (amount mod n) shares take one paisa more.
files(
    "split/dune-project=split/dune-project",
    "split/lib/dune=split/lib/dune",
    "split/fixed/ledger.ml=split/lib/ledger.ml",
    "split/test/dune=split/test/dune",
    "split/test/test_ledger.ml=split/test/test_ledger.ml",
)
run(
    """
cd split
dune test
""",
    mask=[
        (r"This run has ID `[A-Z0-9]+'", "This run has ID `(new on every run)'"),
        (r"Full test results in `[^']*'", "Full test results in `(a folder under _build)'"),
        (r"in \d+\.\d+s", "in (time)s"),
    ],
)

$ cd split
$ dune test
Testing `ledger'.
This run has ID `(new on every run)'.

  [OK]          total          0   three entries.
  [OK]          split          0   an even split.
  [OK]          split          1   no paisa lost.

Full test results in `(a folder under _build)'.
Test Successful in (time)s. 3 tests run.

Three kinds of test, and one command that runs them all

The suite above is an assertion test: you compute a value and state what it must equal, as most unit tests do. The three kinds of test in this lesson compare different things:

Expect tests are written by running them: they start with an empty expectation, dune test shows the printout as a difference, and dune promote copies it into the test file, a step dune calls promotionpromotion: Copying the output a test run produced over the expected output kept in the source tree, the step dune performs with dune promote once you accept the difference it showed.. A cram testcram test: A test that records a shell session, each command with the output and exit status it should produce, in a file whose name ends in .t; dune runs the commands and compares what they print with what the file holds. More in “Command-line tools with Arg and Cmdliner” runs the built program as a user does, so it sees what the other two cannot: the options, the messages and the exit status. The command-line tools lesson tests a word counter with one. All three kinds run with one command, dune test, which fails when any test of the project fails.

Practise: tests that catch a seeded bug

Turn the lesson round: take a date parser that is correct, copy it, seed one bug into the copy, and keep adding test cases until your suite fails on the copy and passes on the original. A suite that cannot tell the two apart is not testing the behaviour you care about.

The self-check below covers the kinds of test and how each fails; the weakest idea comes first.

Expect tests and accepting a diff with dune promote Pro: opens with a Pro, Premium or Ultimate pass

Cram tests for the CLI, and one dune test in CI Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

A warm-up for freshers and 2–5 years: what does Alcotest.check take, and what does a failed check print?

It takes four arguments, in this order: a testable such as int, string or list int, which says how to compare two values and how to print them; a message; the expected value; and the actual value the code computed. When the two differ, the check raises, Alcotest marks that case [FAIL] and goes on with the others, and at the end it prints the failed case by group, number and name, with the message and both values, expected and received. The suite then exits with status 1, which makes dune test fail.

Three harder questions follow, on a printer’s field-by-field checks, a teammate’s diff to promote, and a CI run to keep quick and stable. Pick your first move before opening each.

Key takeaways

• A unit test calls a function and checks its result against a value you name, and Alcotest.(check int) takes a testable, a message, the expected value and the computed one, in that order.
• A failed Alcotest case is reported by group, number and name with its expected and received values, and the suite exits with status 1, so dune test fails too.
• An expect test keeps the printout as its expectation and dune promote accepts a changed one, while a cram test records shell commands with their output and exit status.
• All three kinds of test run with one dune test, which fails when any test of the project fails.