Command-line tools with Arg and Cmdliner

./wordfreq --top 2 --ignore-case notes.txt reaches an OCaml program as five strings in Sys.argv and nothing else: no types, no help text, no sign of which word belongs to which option. Command-line tools with Arg and Cmdliner begin at that array and part company at once. Arg, in the standard library, walks the words against a list of keywords and fills in a few refs. Cmdliner takes typed declarations of the same options and builds the parser, the help page and the documented exit statuses from them.

You will read the argument vector and write the same word-frequency flags both ways. Two parts then go further: a documented wordfreq built from Cmdliner terms and run into each of its four exit statuses, and a dune cram test that records whole command lines with their statuses.

A command line reaches your program as plain strings

The manual describes Sys.argv as the command line arguments given to the process, whose first element is the command name used to invoke the program. Index 0 is therefore the name, a user’s arguments begin at index 1, and the array says nothing about what they mean. The shell has already split the line and taken the quotes off, so a program reading Sys.argv must decide which words are options, which belong to the option before them, and what to do with the rest:

from session import files, run

files("showargs.ml")
run(
    """
cat showargs.ml
ocamlfind ocamlopt -g showargs.ml -o showargs
./showargs
./showargs --top 3 --ignore-case notes.txt
./showargs ''
"""
)

$ cat showargs.ml
(* Prints the argument vector exactly as the process received it. *)
let () =
  Printf.printf "Sys.argv: %d entries\n" (Array.length Sys.argv);
  Array.iteri (fun i s -> Printf.printf "  %d: %S\n" i s) Sys.argv
$ ocamlfind ocamlopt -g showargs.ml -o showargs
$ ./showargs
Sys.argv: 1 entries
  0: "./showargs"
$ ./showargs --top 3 --ignore-case notes.txt
Sys.argv: 5 entries
  0: "./showargs"
  1: "--top"
  2: "3"
  3: "--ignore-case"
  4: "notes.txt"
$ ./showargs ''
Sys.argv: 2 entries
  0: "./showargs"
  1: ""

With no arguments the array held one string, the program’s own name. With --top 3 --ignore-case notes.txt it held five, and the empty argument '' arrived as an empty string, not as nothing.

The Arg module fills your refs from a list of keywords

Arg.parse specs anon_fun usage_msg walks the words for you. Each entry of specs is a triple of a keyword, a spec that says what to do when that keyword appears, and a one-line doc string. Arg.Set r writes true into a bool ref, and Arg.Set_int r reads the next word as an integer. A word that starts with no dash is a positional argumentpositional argument: An argument a program identifies by where it stands on the command line rather than by a name, such as the file in wordfreq notes.txt. The Arg module calls these anonymous arguments., which Arg.parse hands to anon_fun:

The same walk as a program: argparse.ml counts the words of its file, so the flags have something to act on, and its counts check against the word counter:

(* The same job as the Cmdliner tool, parsed with the Arg module of the standard library. *)
let top = ref 3
let ignore_case = ref false
let file = ref ""

let usage_msg = "argparse [-top N] [-ignore-case] FILE"

let set_file name =
  if !file = "" then file := name
  else raise (Arg.Bad (Printf.sprintf "one file at a time, so %S is too many" name))

let specs =
  [ ("-top", Arg.Set_int top, "N  print the N most frequent words (default 3)");
    ("-ignore-case", Arg.Set ignore_case, " count Chai and chai as one word") ]

let counts words =
  let table = Hashtbl.create 64 in
  List.iter
    (fun w -> Hashtbl.replace table w (1 + Option.value ~default:0 (Hashtbl.find_opt table w)))
    words;
  Hashtbl.fold (fun w n acc -> (w, n) :: acc) table []
  |> List.sort (fun (w1, n1) (w2, n2) -> if n1 = n2 then compare w1 w2 else compare n2 n1)

let () =
  Arg.parse specs set_file usage_msg;
  if !file = "" then (prerr_endline "argparse: a file is needed"; exit 2);
  let text = In_channel.with_open_text !file In_channel.input_all in
  let text = if !ignore_case then String.lowercase_ascii text else text in
  let words =
    String.split_on_char ' ' (String.map (fun c -> if c = '\n' || c = '.' || c = ',' then ' ' else c) text)
    |> List.filter (fun w -> w <> "")
  in
  List.iteri
    (fun i (w, n) -> if i < !top then Printf.printf "%4d  %s\n" n w)
    (counts words)

from session import files, run, write

files("argparse.ml")
write(
    "notes.txt",
    "chai and samosa at the stall.\nChai again, chai with ginger.\nsamosa, chutney and chai.\n",
)
run(
    """
ocamlfind ocamlopt -g argparse.ml -o argparse
./argparse -top 2 -ignore-case notes.txt
./argparse --help
!./argparse -top x notes.txt
!./argparse --top 2 notes.txt
"""
)

$ ocamlfind ocamlopt -g argparse.ml -o argparse
$ ./argparse -top 2 -ignore-case notes.txt
   4  chai
   2  and
$ ./argparse --help
argparse [-top N] [-ignore-case] FILE
  -top N  print the N most frequent words (default 3)
  -ignore-case  count Chai and chai as one word
  -help  Display this list of options
  --help  Display this list of options
$ ./argparse -top x notes.txt
./argparse: wrong argument 'x'; option '-top' expects an integer.
argparse [-top N] [-ignore-case] FILE
  -top N  print the N most frequent words (default 3)
  -ignore-case  count Chai and chai as one word
  -help  Display this list of options
  --help  Display this list of options
[exit status 2]
$ ./argparse --top 2 notes.txt
./argparse: unknown option '--top'.
argparse [-top N] [-ignore-case] FILE
  -top N  print the N most frequent words (default 3)
  -ignore-case  count Chai and chai as one word
  -help  Display this list of options
  --help  Display this list of options
[exit status 2]

Three things in that run matter. Arg.parse answers -help and --help without being asked. On an error it prints the reason, the usage message and the option list to standard error and ends the process with exit statusexit status: An integer a program returns to its parent process when it ends, where 0 means success and other values indicate failures. More in “Exceptions and resource cleanup” 2, as the manual’s own example also shows: wrong argument 'x'; option '-top' expects an integer, and unknown option '--top' for a command-line optioncommand-line option: A word of a command line that starts with a dash and names a setting rather than data, such as -i or --top; an option that takes a value reads it from the next word or after an equals sign. missing from the spec list. And nothing after the call runs, so a mistyped flag cannot half-finish a job.

What Cmdliner adds to the very same flags

Cmdliner is a library that turns typed declarations of your options into a parser, a man page and documented exit statuses; the OCaml.org tutorial names it as a modern interface that also generates UNIX man pages automatically, and the version installed here is 2.1.1. One option becomes one value: Arg.(value & opt int 3 & info ["n"; "top"] ~doc ~docv:"N") evaluates to an int, has the names -n and --top, defaults to 3, and carries its documentation in the labelled argumentslabelled argument: An argument passed by name, written ~name:value at the call. Labelled arguments of a complete call may come in any order. More in “Labelled and optional arguments” ~doc and ~docv. Command-line tools with Arg and Cmdliner ask the same questions of a line, and answer them with very different amounts of your code:

Two things follow. A documented Cmdliner command is three argument terms applied to one function, plus the information that becomes its help: the first part below builds wordfreq that way and runs it into each of its four exit statuses. A cram testcram test: A test that records a shell session, each command with the output and exit status it should produce, in a file whose name ends in .t; dune runs the commands and compares what they print with what the file holds. records commands, their output and their exit codes in a .t file that dunedune: OCaml's build system. It reads dune files that describe libraries, executables and tests, calls the compilers in the right order and keeps every result in _build. More in “Your first dune project and editor setup” replays: the second part tests the finished tool with one.

Practise: argv, specs and exit statuses

The questions use the two recorded runs above and the status each library ends with.

A documented wordfreq, from three Cmdliner terms to four exit statuses Pro: opens with a Pro, Premium or Ultimate pass

Testing the whole command line with a dune cram test Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Warm-up (fresher to 2–5 years): what does Arg.parse do when it meets an option that is not in the spec list?

It stops the program. Arg.parse prints three things to standard error, in this order: the reason, such as unknown option '--top'; the usage message you passed it; and the list of options with their doc strings, including the -help and --help it adds itself. Then it ends the process with exit status 2, which the OCaml.org tutorial’s own run shows as [2]. Nothing after the call runs, so a mistyped option cannot start a job and abandon it half done. An option with an empty doc string is left out of that list.

Three harder questions follow: a script that cannot tell a user’s mistake from a crash, a help text that has drifted from the behaviour, and a flag that must stay usable after a rename. Decide your first move, then open each question’s first step:

Key takeaways

• A command line reaches an OCaml program as Sys.argv, an array whose index 0 is the program’s own name and whose later entries mean nothing until your code gives them one.
• Arg.parse takes a spec list of keyword, spec and doc string plus a function for anonymous arguments, answers -help itself, and ends with status 2 after printing the reason and the usage to standard error.
• An Arg keyword matches exactly as written, so -top refuses --top, while one Cmdliner declaration carries every name of an option, its type, its default and its documentation.
• Cmdliner builds the parser, the man page and the documented exit statuses from those declarations, and a cram test replays whole command lines and compares their output and statuses.