Concurrent I/O with Eio and Lwt

A program that waits on two slow operations one after the other takes the sum of the two waits; a program that waits on them together takes the longer one. On a mock clock, the recorded run below took 3.5 seconds for two sleeps one after the other and 2.0 seconds for the same two sleeps under Fiber.both.

Structured concurrencystructured concurrency: A way of organising concurrent fibers so that a parent scope owns every child it starts: the scope waits for every child to finish or be cancelled before it returns. In Eio, Eio.Switch.run opens such a scope. is how Eio, the direct-style I/O library for OCaml 5, lets a program own that overlap without threads or callbacks: every fiberfiber: A lightweight thread of execution inside one domain: only one fiber of a domain runs at a time, until it waits for input, output or a timer, and then the next ready fiber runs. runs inside a scope that waits for it. You will run two waits together, attach fibers to a switch that waits for them, and read the results in direct style. A budget with cancellationcancellation: Stopping a fiber or a promise before it finishes, for example because a sibling failed or a time budget ran out; in Eio, Fiber.both and a switch cancel their other fibers when one of them raises. and a cap on buffered input build on that, and Lwt’s monadic concurrencymonadic concurrency: A style in which a function that waits returns a wrapped value such as 'a Lwt.t, and callers chain further work with bind (let*). Every waiting call shows in the types, and every caller of one returns a promise too. is set beside it.

Fibers wait together, and a switch waits for its fibers

Concurrent I/O with Eio starts at Eio_main.run, which picks an event loopevent loop: The scheduler at the heart of a concurrent-I/O library: it keeps the fibers or promises that are ready, runs one until it has to wait, and resumes it when the operating system reports that its input, output or timer is ready. Eio and Lwt each have one. back end for the platform (eio_posix was installed with it on macOS at version 1.6) and hands your function one env with a clock, a filesystem, a network and so on. A fiber is a light thread of execution that Eio switches away from whenever it waits. Eio.Fiber.both f g runs f and g as two fibers and returns when both have finished; if one raises, it cancels the other. Eio.Fiber.pair has the same shape and returns the two results as a pair.

The session below runs two “slow” operations one after the other, then with Fiber.both. It uses Eio_mock.Backend.run_full in place of Eio_main.run: its clock moves only when every fiber is asleep, so the printed times are the same on every run.

(* Two "slow" operations on a mock clock: one after the other, then with
   Eio.Fiber.both, then Eio.Fiber.pair, then two fibers forked onto a switch.
   Eio_mock.Backend.run_full stands in for Eio_main.run: it hands over an env
   whose clock advances only when every fiber is asleep, so the printed times
   are the same on every run. *)

let fetch label clock delay =
  let t0 = Eio.Time.now clock in
  Eio.Time.sleep clock delay;
  let t1 = Eio.Time.now clock in
  Printf.printf "%s: finished at t=%.1fs (waited %.1fs)\n%!" label t1 (t1 -. t0)

let one_after_the_other clock =
  let t0 = Eio.Time.now clock in
  fetch "serial a" clock 1.5;
  fetch "serial b" clock 2.0;
  let t1 = Eio.Time.now clock in
  Printf.printf "serial total: %.1fs\n%!" (t1 -. t0)

(* Fiber.both needs no switch: it returns when both fibers have finished. *)
let together clock =
  let t0 = Eio.Time.now clock in
  Eio.Fiber.both
    (fun () -> fetch "concurrent a" clock 1.5)
    (fun () -> fetch "concurrent b" clock 2.0);
  let t1 = Eio.Time.now clock in
  Printf.printf "concurrent total: %.1fs\n%!" (t1 -. t0)

let pair_result clock =
  let a, b =
    Eio.Fiber.pair
      (fun () -> Eio.Time.sleep clock 0.5; "answer A")
      (fun () -> Eio.Time.sleep clock 0.7; 42)
  in
  Printf.printf "pair: %s and %d\n%!" a b

(* Fiber.fork returns at once; the switch the fibers are attached to waits. *)
let forked clock =
  let t0 = Eio.Time.now clock in
  Eio.Switch.run (fun sw ->
    Eio.Fiber.fork ~sw (fun () -> fetch "forked a" clock 1.5);
    Eio.Fiber.fork ~sw (fun () -> fetch "forked b" clock 2.0);
    print_endline "both forked; Switch.run now waits for them");
  let t1 = Eio.Time.now clock in
  Printf.printf "switch total: %.1fs\n%!" (t1 -. t0)

let () =
  Eio_mock.Backend.run_full @@ fun env ->
  let clock = env#clock in
  one_after_the_other clock;
  print_endline "---";
  together clock;
  print_endline "---";
  pair_result clock;
  print_endline "---";
  forked clock

"""Two slow operations on a mock clock: serial waits add, Fiber.both and
Fiber.pair overlap them, and two fibers forked onto one switch keep
Switch.run waiting until both have finished. The mock clock advances only
when every fiber is asleep, so the printed times are the same on every run.
"""
from session import files, run, write

files("fibers.ml")
write("dune-project", "(lang dune 3.0)\n")
write("dune", "(executable (name fibers) (libraries eio.mock))\n")
run("""
dune exec ./fibers.exe
""")

$ dune exec ./fibers.exe
+mock time is now 1.5
serial a: finished at t=1.5s (waited 1.5s)
+mock time is now 3.5
serial b: finished at t=3.5s (waited 2.0s)
serial total: 3.5s
---
+mock time is now 5
concurrent a: finished at t=5.0s (waited 1.5s)
+mock time is now 5.5
concurrent b: finished at t=5.5s (waited 2.0s)
concurrent total: 2.0s
---
+mock time is now 6
+mock time is now 6.2
pair: answer A and 42
---
both forked; Switch.run now waits for them
+mock time is now 7.7
forked a: finished at t=7.7s (waited 1.5s)
+mock time is now 8.2
forked b: finished at t=8.2s (waited 2.0s)
switch total: 2.0s

Serial fetches of 1.5 s and 2.0 s finished at t=1.5 s and t=3.5 s, a total of 3.5 s. Under Fiber.both both fibers went to sleep at t=3.5 s; the backend printed +mock time is now 5 when the first wait ended and now 5.5 when the second did, a total of 2.0 s, the longer of the two waits. Step through that second run:

Fiber.pair paired a 0.5 s fiber returning "answer A" with a 0.7 s fiber returning 42 and returned both. The last run opened Eio.Switch.run and attached two fibers to its switch with Eio.Fiber.fork ~sw. fork returns at once, so both forked; Switch.run now waits for them printed first, and Switch.run returned only when both fibers had finished, again after 2.0 s. A switch waits for every fiber attached to it, releases the resources attached to it, and cancels its other fibers when one of them raises.

Why direct-style reads like ordinary code

Eio builds its scheduler on top of the OCaml 5 effect handlers of the two earlier lessons of this module. sleep, accept, read and write perform effects that the backend catches, so Buf_read.line r is an ordinary call that returns a string. The type of a function that waits is the type of its answer, as three waiting calls show:

Concurrent I/O with Eio therefore reads like the sequential code it replaces.

Three tools build on fibers. Eio.Time.with_timeout gives an operation a budget and cancels it when the budget runs out, returning Error with a `Timeout tag. Buf_read.of_flow ~max_size:N caps the bytes a reader buffers, so one client cannot decide how much memory the server uses. Lwt is the promise library a Dream server or an existing cohttp-lwt client is written with, so reading its let* and Lwt_main.run code is part of working in OCaml today.

Practise fibers, switches and direct style

Every question below is about the fibers above or the types of direct style: predict the wait or the result before you choose.

Cancellation, timeouts and an input limit a service lives behind Pro: opens with a Pro, Premium or Ultimate pass

Reading Lwt: promises, let-bind and where you still meet it Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Warm-up for freshers and 2–5 years: what is an Eio fiber, and how does Eio.Fiber.both differ from spawning two threads?

An Eio fiber is a lightweight unit of execution that Eio’s event loop schedules on top of OCaml 5 effect handlers: calling Eio.Time.sleep, Buf_read.line or Eio.Net.connect performs an effect that suspends the fiber until the backend has an answer, and other fibers run meanwhile. Eio.Fiber.both f g runs two fibers and returns when both have returned, so the wall time is the slower of the two waits, not the sum, and if one raises it cancels the other. Two systhreads of one domain also take turns, since only one of them at a time runs OCaml code, but each is an operating-system thread with its own stack. Fibers are switched by Eio, inside one thread, at the points where they wait.

Three harder questions follow: a service whose latency spikes come from one slow upstream, a line-based server that gets a request with no newline for a megabyte, and a team choosing between Eio and Lwt for a new HTTP service. Pick your first move before opening each:

Key takeaways

• Eio_main.run hands a program its env; Eio.Fiber.both and Fiber.pair run two fibers and return when both have finished, so the wall time is the slower wait, not the sum.
• Eio.Switch.run waits for every fiber forked onto its switch with Fiber.fork ~sw, releases what is attached to it, and cancels its other fibers when one of them raises.
• In direct style a function that waits returns its answer’s own type, string for Buf_read.line, where the Lwt version returns string Lwt.t.
• Eio.Time.with_timeout gives an operation a budget, Buf_read.of_flow ~max_size:N caps buffered input, and Lwt’s let* style is what a Dream server or a cohttp-lwt client is written in.