Files, channels and processes

A log file of six lines holds three INFO, one WARN and two ERROR entries. One function, In_channel.fold_lines, reads it a line at a time and counts (3, 1, 2) without ever holding the whole file in memory. That matters when the log is gigabytes and holding every line would exhaust memory, or when a second read would repeat work that a single pass could finish.

OCaml files are accessed through channels, abstract handles that the In_channel and Out_channel modules provide. A channel can read from or write to a file, a pipe or the standard streams. OCaml files opened in text mode translate line endings on some platforms, while binary mode reads and writes bytes exactly as they are. You will read and write files with channelschannel: An abstract type that reads from or writes to a stream of bytes, such as a file, a pipe or standard input/output., handle the Sys_error that a missing file raises, run external programs safely without letting user input inject commands, and work with paths, directories and environment variables.

Channels read and write streams of bytes or text

An In_channel.t reads from a file, a pipe or standard inputstandard streams: The three channels a process inherits: standard input (stdin) for reading, standard output (stdout) and standard error (stderr) for writing.; an Out_channel.t writes to them. The with_open_text and with_open_bin functions open a channel, pass it to a function and close it when that function returns, even if it raises an exception. Step through the lifecycle:

The session below creates a small log file, reads it three ways and writes a summary:

"""Channels, reading and writing files with In_channel and Out_channel."""
from session import toplevel, run, files, write

# Create a small log file
write("server.log", """[INFO] Starting server on port 8080
[ERROR] Connection refused: database
[INFO] Retry attempt 1
[WARN] Slow query: 2.3 seconds
[ERROR] Timeout waiting for response
[INFO] Shutting down gracefully
""")

run('''
cat server.log
''')

# Show In_channel.with_open_text, input_all, input_lines and fold_lines in one session
toplevel('''
(* Read the whole file at once *)
let contents = In_channel.with_open_text "server.log" In_channel.input_all;;

String.length contents;;

(* Read as a list of lines *)
let lines = In_channel.with_open_text "server.log" In_channel.input_lines;;

List.length lines;;

List.hd lines;;

(* Count lines by level, one at a time *)
let count_levels path =
  In_channel.with_open_text path (fun ic ->
    In_channel.fold_lines (fun (info, warn, err) line ->
      if String.length line >= 6 then
        match String.sub line 1 4 with
        | "INFO" -> (info + 1, warn, err)
        | "WARN" -> (info, warn + 1, err)
        | "ERRO" -> (info, warn, err + 1)
        | _ -> (info, warn, err)
      else (info, warn, err)
    ) (0, 0, 0) ic);;

count_levels "server.log";;

(* Write a summary to a file *)
Out_channel.with_open_text "summary.txt" (fun oc ->
  let (info, warn, err) = count_levels "server.log" in
  Printf.fprintf oc "INFO: %d\\nWARN: %d\\nERROR: %d\\n" info warn err);;
''')

run('''
cat summary.txt
''')

$ cat server.log
[INFO] Starting server on port 8080
[ERROR] Connection refused: database
[INFO] Retry attempt 1
[WARN] Slow query: 2.3 seconds
[ERROR] Timeout waiting for response
[INFO] Shutting down gracefully
$ ocaml
OCaml version 5.5.1
Enter #help;; for help.

# (* Read the whole file at once *)
  let contents = In_channel.with_open_text "server.log" In_channel.input_all;;
  val contents : string =
  "[INFO] Starting server on port 8080\n[ERROR] Connection refused: database\n[INFO] Retry attempt 1\n[WARN] Slow query: 2.3 seconds\n[ERROR] Timeout waiting for response\n[INFO] Shutting down gracefully\n"
# 
  String.length contents;;
  - : int = 196
# 
  (* Read as a list of lines *)
  let lines = In_channel.with_open_text "server.log" In_channel.input_lines;;
    val lines : string list =
  ["[INFO] Starting server on port 8080";
   "[ERROR] Connection refused: database"; "[INFO] Retry attempt 1";
   "[WARN] Slow query: 2.3 seconds"; "[ERROR] Timeout waiting for response";
   "[INFO] Shutting down gracefully"]
# 
  List.length lines;;
  - : int = 6
# 
  List.hd lines;;
  - : string = "[INFO] Starting server on port 8080"
# 
  (* Count lines by level, one at a time *)
  let count_levels path =
    In_channel.with_open_text path (fun ic ->
      In_channel.fold_lines (fun (info, warn, err) line ->
        if String.length line >= 6 then
          match String.sub line 1 4 with
          | "INFO" -> (info + 1, warn, err)
          | "WARN" -> (info, warn + 1, err)
          | "ERRO" -> (info, warn, err + 1)
          | _ -> (info, warn, err)
        else (info, warn, err)
      ) (0, 0, 0) ic);;
                        val count_levels : string -> int * int * int = <fun>
# 
  count_levels "server.log";;
  - : int * int * int = (3, 1, 2)
# 
  (* Write a summary to a file *)
  Out_channel.with_open_text "summary.txt" (fun oc ->
    let (info, warn, err) = count_levels "server.log" in
    Printf.fprintf oc "INFO: %d\nWARN: %d\nERROR: %d\n" info warn err);;
        - : unit = ()
$ cat summary.txt
INFO: 3
WARN: 1
ERROR: 2

• In_channel.with_open_text path f opens the file in text mode and calls f with the channel. When f returns, the channel is closed. Text mode translates line endings on Windows; use with_open_bin for binary files like images or archives.
• In_channel.input_all returns the whole file as one string. For small configuration files or templates this is the simplest choice.
• In_channel.input_lines (since 5.1) returns every line as a list. The session shows six lines and 196 characters. Lines do not include the newline character.
• In_channel.fold_lines f init ic (since 5.1) processes lines one at a time, calling f with the accumulator and each line. For a large file, this uses constant memory because only one line is in memory at a time. The accumulator works like List.fold_left: the function receives the current value and returns the next.
• Out_channel.with_open_text path f opens a file for writing and truncates any existing content. Use Printf.fprintf oc "..." or output_string oc s to write.

A missing file raises Sys-error with the path and the reason

When with_open_text cannot open a file, it raises Sys_error with a message that names the path and the reason. The program below tries to read a file that does not exist, catches the error and returns None instead of crashing:

"""Handling a missing file without crashing."""
from session import toplevel

# Show the Sys_error
toplevel('''
In_channel.with_open_text "missing.txt" In_channel.input_all;;
''')

# Show how to handle it
toplevel('''
let read_or_empty path =
  match In_channel.with_open_text path In_channel.input_all with
  | contents -> Some contents
  | exception Sys_error msg ->
      Printf.eprintf "Could not read %s: %s\\n%!" path msg;
      None;;

read_or_empty "missing.txt";;
''')

$ ocaml
OCaml version 5.5.1
Enter #help;; for help.

# In_channel.with_open_text "missing.txt" In_channel.input_all;;
Exception: Sys_error "missing.txt: No such file or directory".
$ ocaml
OCaml version 5.5.1
Enter #help;; for help.

# let read_or_empty path =
    match In_channel.with_open_text path In_channel.input_all with
    | contents -> Some contents
    | exception Sys_error msg ->
        Printf.eprintf "Could not read %s: %s\n%!" path msg;
        None;;
          val read_or_empty : string -> string option = <fun>
# 
  read_or_empty "missing.txt";;
  Could not read missing.txt: missing.txt: No such file or directory
- : string option = None

The message "missing.txt: No such file or directory" comes from the operating system. On different systems the exact wording may vary, but the path is always at the start. The pattern | exception Sys_error msg catches the error inside the match, which is cleaner than wrapping the whole expression in try ... with. The function read_or_empty returns Some contents when the file exists and None with a log line when it does not, so the caller can decide what to do without the program stopping.

Other file operations raise the same Sys_error: writing to a read-only file, opening a directory as a file, or removing a file that does not exist. Catching Sys_error in one place handles all of them. The locked parts below cover running external programs safely without a shell, working with paths and directories, and walking a directory tree while skipping symbolic links.

Practise reading and handling files

Running programs without a shell: the risk and the fix Pro: opens with a Pro, Premium or Ultimate pass

Paths, directories, the environment and symbolic links Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Key takeaways

• In_channel.with_open_text opens, passes to a function and closes the channel automatically, even if the function raises.
• fold_lines processes a large file a line at a time, using constant memory; input_lines loads every line into a list.
• A missing file raises Sys_error with the path and the reason; catching it lets the program report the problem instead of crashing.