Fonts, engines and Unicode

A seminar notice that names its speaker, Ирина Петрова, came out of LuaLaTeX with a gap where the 12 letters of the name should be, and the build ended with status 0. The only trace was 36 lines in the middle of the run’s output.

Below you will choose a LaTeX engine for the fonts a document needs, load OpenType fonts in LaTeX and in Typst, make a character the font lacks stop the build instead of vanishing, and check that the PDF’s text copies and searches the way it reads. Most results come from one seminar notice, built by every engine and read back by two text extractors.

pdfLaTeX, XeLaTeX and LuaLaTeX read fonts in two different ways

pdfLaTeX works with 8-bit fonts: each font holds at most 256 characters, arranged by a font encodingfont encoding: The table that says which character each slot of a font holds, such as OT1 or T1 for pdfLaTeX. OT1, the default, lacks some ASCII characters, so LaTeX draws them and they copy out wrongly. More in “Special characters, quotes, dashes and spaces” such as T1, and a package such as lmodern picks the font files. XeLaTeX and LuaLaTeX are Unicode engines: they read every character as its code pointcode point: The number Unicode gives a character, written U+ and four or more hexadecimal digits, such as U+2013 for the en dash. Two characters that look alike can have different code points. More in “Special characters, quotes, dashes and spaces” and set it with an OpenType fontOpenType font: A font file, ending in .otf or .ttf, that holds glyphs for Unicode characters together with rules for ligatures and other features. XeLaTeX and LuaLaTeX load such fonts through fontspec; Typst uses them directly., which the fontspec package loads. Its CTAN description calls fontspec a package for XeLaTeX and LuaLaTeX, so it is the first thing to fail when the engine is wrong. Here is the notice, once for the Unicode engines and once for pdfLaTeX:

% A seminar notice for LuaLaTeX or XeLaTeX. fontspec loads its default, Latin Modern, which has no Cyrillic letters.
\documentclass{article}
\usepackage{fontspec}
\begin{document}
Speaker: Ирина Петрова, from the field station. The office is efficient:
official badges are issued within a day.
\end{document}

% The same notice for pdfLaTeX, which cannot load fontspec: T1-encoded Latin Modern.
\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\begin{document}
Speaker: Ирина Петрова, from the field station. The office is efficient:
official badges are issued within a day.
\end{document}

This session builds them with all three engines and lists the fonts each PDF embeds with poppler’s pdffonts:

from session import files, read, run, write

# The notice through the three LaTeX engines. pdfLaTeX gets notice-pdf.tex, which loads no fontspec, and then a copy
# with the speaker's name in Latin letters; LuaLaTeX and XeLaTeX get notice.tex. pdffonts lists the fonts each PDF
# embeds.
files("notice/notice.tex", "notice/notice-pdf.tex")
write("notice-latin.tex", read("notice-pdf.tex").replace("Ирина Петрова", "Irina Petrova"))
run(r"""
!pdflatex -interaction=nonstopmode -halt-on-error notice.tex > /dev/null
grep -A1 "^! Fatal" notice.log
!pdflatex -interaction=nonstopmode -halt-on-error notice-pdf.tex > /dev/null
grep -A1 "^! LaTeX Error" notice-pdf.log
pdflatex -interaction=nonstopmode notice-latin.tex | grep Output
pdffonts notice-latin.pdf
lualatex -interaction=nonstopmode notice.tex | grep Output
pdffonts notice.pdf
xelatex -interaction=nonstopmode notice.tex | grep Output
pdffonts notice.pdf
""")

$ pdflatex -interaction=nonstopmode -halt-on-error notice.tex > /dev/null
[exit status 1]
$ grep -A1 "^! Fatal" notice.log
! Fatal Package fontspec Error: The fontspec package requires either XeTeX or
(fontspec)                      LuaTeX.
$ pdflatex -interaction=nonstopmode -halt-on-error notice-pdf.tex > /dev/null
[exit status 1]
$ grep -A1 "^! LaTeX Error" notice-pdf.log
! LaTeX Error: Unicode character И (U+0418)
               not set up for use with LaTeX.
$ pdflatex -interaction=nonstopmode notice-latin.tex | grep Output
Output written on notice-latin.pdf (1 page, 29696 bytes).
$ pdffonts notice-latin.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
LPQTCA+LMRoman10-Regular             Type 1            Custom           yes yes yes      4  0
$ lualatex -interaction=nonstopmode notice.tex | grep Output
Output written on notice.pdf (1 page, 5621 bytes).
$ pdffonts notice.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
VFSOVW+LMRoman10-Regular             CID Type 0C       Identity-H       yes yes yes      4  0
$ xelatex -interaction=nonstopmode notice.tex | grep Output
Output written on notice.pdf (1 page).
$ pdffonts notice.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
NUWFSN+LMRoman10-Regular-Identity-H  CID Type 0C       Identity-H       yes yes yes      4  0

pdfLaTeX stopped twice. With fontspec it gave the fatal error that fontspec requires XeTeX or LuaTeX; without it, it refused И, the first letter of the name, because no font it had loaded was set up for that character. With the name in Latin letters, it embedded LMRoman10-Regular as a Type 1 font. LuaLaTeX and XeLaTeX built notice.tex with no error and embedded the same design as a CID Type 0C font, which the pdffonts manual describes as a 16-bit PostScript CFF font: the OpenType version.

Which engine fits depends on the fonts and on where the source goes. Answer two questions about your document:

In short: pdfLaTeX when a template or venue asks for it and its fonts suffice, XeLaTeX for OpenType fonts that must also build on arXiv, whose list of processors includes xelatex but not LuaLaTeX, and LuaLaTeX or Typst otherwise.

fontspec loads a font by name in LuaLaTeX, by file in XeLaTeX

\setmainfont{Libertinus Serif} asks for a font by the name you would pick from a menu. Libertinus Serif ships with TeX Live, so its files sit in the TeX treeTeX tree: The folders where a TeX distribution keeps its classes, packages, fonts and settings, such as texmf-dist. Engines find each file in it by name, and kpsewhich prints where a file is. More in “Install TeX Live, TinyTeX or MiKTeX, and Typst”. A second copy of the notice names the same font by its files instead, with fontspec’s Extension and UprightFont keys, so that *-Regular becomes LibertinusSerif-Regular.otf:

% The notice in Libertinus Serif, a font of the TeX tree, asked for by its name.
\documentclass{article}
\usepackage{fontspec}
\setmainfont{Libertinus Serif}
\begin{document}
Speaker: Ирина Петрова, from the field station. The office is efficient:
official badges are issued within a day.
\end{document}

% The notice in Libertinus Serif, asked for by its file names, which LuaLaTeX and XeLaTeX both find.
\documentclass{article}
\usepackage{fontspec}
\setmainfont{LibertinusSerif}[
  Extension      = .otf,
  UprightFont    = *-Regular,
  BoldFont       = *-Bold,
  ItalicFont     = *-Italic,
  BoldItalicFont = *-BoldItalic]
\begin{document}
Speaker: Ирина Петрова, from the field station. The office is efficient:
official badges are issued within a day.
\end{document}

from session import files, read, run, write

# Libertinus Serif, which lives in the TeX tree, asked for by name and by file name. luaotfload-tool asks LuaLaTeX's
# font database where a name leads. XeLaTeX builds a copy of notice-name.tex under its own name, so that its PDF does
# not replace LuaLaTeX's.
files("notice/notice-name.tex", "notice/notice-file.tex")
write("notice-name-xe.tex", read("notice-name.tex"))
run(r"""
luaotfload-tool --find="Libertinus Serif"
lualatex -interaction=nonstopmode notice-name.tex | grep -E "rror|Output"
pdffonts notice-name.pdf
!xelatex -interaction=nonstopmode notice-name-xe.tex > /dev/null
grep -A2 "^! Package fontspec Error" notice-name-xe.log | head -3
grep -c "in font nullfont" notice-name-xe.log
pdftotext notice-name-xe.pdf - | wc -w
xelatex -interaction=nonstopmode notice-file.tex | grep -E "rror|Output"
pdffonts notice-file.pdf
""")

$ luaotfload-tool --find="Libertinus Serif"
luaotfload | resolve : Font "Libertinus Serif" found!
luaotfload | resolve : Resolved file name "…/texmf-dist/fonts/opentype/public/libertinus-fonts/LibertinusSerif-Regular.otf"
$ lualatex -interaction=nonstopmode notice-name.tex | grep -E "rror|Output"
Output written on notice-name.pdf (1 page, 7057 bytes).
$ pdffonts notice-name.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
KPJYNR+LibertinusSerif-Regular       CID Type 0C       Identity-H       yes yes yes      4  0
$ xelatex -interaction=nonstopmode notice-name-xe.tex > /dev/null
[exit status 1]
$ grep -A2 "^! Package fontspec Error" notice-name-xe.log | head -3
! Package fontspec Error: 
(fontspec)                The font "Libertinus Serif" cannot be found; this
(fontspec)                may be but usually is not a fontspec bug. Either
$ grep -c "in font nullfont" notice-name-xe.log
97
$ pdftotext notice-name-xe.pdf - | wc -w
       0
$ xelatex -interaction=nonstopmode notice-file.tex | grep -E "rror|Output"
Output written on notice-file.pdf (1 page).
$ pdffonts notice-file.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
ZFUMQQ+LibertinusSerif-Regular-Identity-H CID Type 0C       Identity-H       yes yes yes      4  0

LuaLaTeX’s font loader, luaotfload, keeps its own database of fonts, and luaotfload-tool --find showed that it includes the TeX tree: the name led to LibertinusSerif-Regular.otf. fontspec’s manual explains the difference: in LuaTeX, fonts of the TeX tree can be loaded by name, while XeTeX can do so on Windows and Linux but not on macOS, where this lesson ran; by file name, both always find them. So XeLaTeX failed with The font "Libertinus Serif" cannot be found, and in nonstopmode it went on to report 97 characters missing from nullfont, a font with no glyphs: the run ended with status 1 and a PDF with no text at all. By file name, XeLaTeX built the notice. Follow both lookups:

arXiv’s TeX Live page gives the same rule for XeLaTeX on its servers, where practically no fonts are registered with fontconfig: fonts of the TeX Live distribution need the file name lookup. A file name works in both engines, so use it for any font a document must find on other machines.

A character the font lacks is only a warning in the log

Back to the notice in Latin Modern, the font fontspec loads when you choose none. Latin Modern has no Cyrillic letters. LaTeX News 33 says what the Unicode engines then do: they typeset nothing and write a Missing character warning to the log. This session builds the notice, then a copy with \tracinglostchars=3, then the copy in Libertinus Serif, and lists which fonts of the TeX tree have И at all:

from session import files, run, shell

# The speaker's name in Latin Modern: how many Missing character lines LuaLaTeX and XeLaTeX print, the first of them,
# what the PDF's text then says, and what \tracinglostchars=3 changes, with -halt-on-error and without it. Then the
# notice in Libertinus Serif, the fee notice of the India note, and the fonts of the TeX tree that have И (U+0418) and
# ₹ (U+20B9), by fc-query's list of the characters each font covers.
files("notice/notice.tex", "notice/notice-strict.tex", "notice/notice-file.tex", "notice/rupee.tex")
run(r"""
lualatex -interaction=nonstopmode notice.tex | grep -c "Missing character"
grep -m1 "Missing character" notice.log
pdftotext notice.pdf - | head -1
xelatex -interaction=nonstopmode notice.tex | grep -E "Output"
grep -c "Missing character" notice.log
pdftotext notice.pdf - | head -1
!lualatex -interaction=nonstopmode -halt-on-error notice-strict.tex > /dev/null
grep -m1 -E "^Missing character" notice-strict.log
grep "Fatal error" notice-strict.log
!xelatex -interaction=nonstopmode -halt-on-error notice-strict.tex > /dev/null
grep -A1 "^! Missing character" notice-strict.log
!ls notice-strict.pdf
!lualatex -interaction=nonstopmode notice-strict.tex > /dev/null
pdftotext notice-strict.pdf - | head -1
lualatex -interaction=nonstopmode notice-file.tex | grep -E "Missing|Output"
pdftotext notice-file.pdf - | head -1
lualatex -interaction=nonstopmode rupee.tex | grep -c "Missing character"
pdftotext rupee.pdf - | head -1
""")


def covers(font, code):
    """Whether fc-query lists the character among the ones the font file has."""
    _, ranges = shell(f'fc-query -f "%{{charset}}" "$(kpsewhich {font})"')
    for part in ranges.split():
        low, _, high = part.partition("-")
        if int(low, 16) <= code <= int(high or low, 16):
            return True
    return False


print("# fonts of the TeX tree that have И (U+0418) and ₹ (U+20B9)")
for font in ["lmroman10-regular.otf", "texgyretermes-regular.otf", "XCharter-Roman.otf", "LibertinusSerif-Regular.otf",
             "NewCM10-Book.otf", "STIXTwoText-Regular.otf", "FiraSans-Regular.otf"]:
    marks = ["yes" if covers(font, code) else "no " for code in (0x418, 0x20B9)]
    print(f"{font:30} И {marks[0]}  ₹ {marks[1]}")

$ lualatex -interaction=nonstopmode notice.tex | grep -c "Missing character"
36
$ grep -m1 "Missing character" notice.log
Missing character: There is no И (U+0418) in font [lmroman10-regular]:+tlig;!
$ pdftotext notice.pdf - | head -1
Speaker: ����� �������, from the field station. The office is efficient: official
$ xelatex -interaction=nonstopmode notice.tex | grep -E "Output"
Output written on notice.pdf (1 page).
$ grep -c "Missing character" notice.log
12
$ pdftotext notice.pdf - | head -1
Speaker: ����� �������, from the field station. The office is efficient: official
$ lualatex -interaction=nonstopmode -halt-on-error notice-strict.tex > /dev/null
[exit status 1]
$ grep -m1 -E "^Missing character" notice-strict.log
Missing character: There is no И (U+0418) in font [lmroman10-regular]:+tlig;!
$ grep "Fatal error" notice-strict.log
!  ==> Fatal error occurred, no output PDF file produced!
$ xelatex -interaction=nonstopmode -halt-on-error notice-strict.tex > /dev/null
[exit status 1]
$ grep -A1 "^! Missing character" notice-strict.log
! Missing character: There is no И (U+0418) in font [lmroman10-regular]:mapping=tex-text;.
l.6 Speaker: Ирина 
$ ls notice-strict.pdf
ls: notice-strict.pdf: No such file or directory
[exit status 1]
$ lualatex -interaction=nonstopmode notice-strict.tex > /dev/null
[exit status 1]
$ pdftotext notice-strict.pdf - | head -1
Speaker: ����� �������, from the field station. The office is efficient: official
$ lualatex -interaction=nonstopmode notice-file.tex | grep -E "Missing|Output"
Output written on notice-file.pdf (1 page, 7057 bytes).
$ pdftotext notice-file.pdf - | head -1
Speaker: Ирина Петрова, from the field station. The office is efficient: official
$ lualatex -interaction=nonstopmode rupee.tex | grep -c "Missing character"
3
$ pdftotext rupee.pdf - | head -1
Hostel fee for the term: �4,500.
# fonts of the TeX tree that have И (U+0418) and ₹ (U+20B9)
lmroman10-regular.otf          И no   ₹ no 
texgyretermes-regular.otf      И no   ₹ no 
XCharter-Roman.otf             И yes  ₹ no 
LibertinusSerif-Regular.otf    И yes  ₹ yes
NewCM10-Book.otf               И yes  ₹ yes
STIXTwoText-Regular.otf        И yes  ₹ yes
FiraSans-Regular.otf           И yes  ₹ yes

Both engines ended with status 0 and wrote the PDF. LuaLaTeX wrote 36 Missing character lines for the 12 letters, the first There is no И (U+0418) in font [lmroman10-regular], XeLaTeX wrote 12, and nothing at the end of either run repeats them. The text of both PDFs read Speaker: ����� �������: a replacement character stands for each lost letter, so the loss also reaches search and copy. With \tracinglostchars=3 the first missing letter became an error. Run with -halt-on-error, both engines stopped there with status 1 and wrote no PDF; run without it, LuaLaTeX also ended with status 1 but still wrote the PDF, gap included, so a script has to stop on the status.

Step through what happened to the first letter of the name:

The fix is a font that has the characters. fc-query, which lists every character a font file covers, found И in XCharter, Libertinus Serif, New Computer Modern, STIX Two Text and Fira Sans, and not in Latin Modern or TeX Gyre Termes. The notice in Libertinus Serif built with no Missing character line and read the name back in full.

Typst falls back to any font it knows, then draws a box

Typst’s text reference gives its default font as Libertinus Serif, one of four fonts built into the command-line tool, and lists where it finds others: folders given with --font-path first, then the system’s fonts, then the built-in ones. typst fonts prints what it found. Typst tries the families of the font list in order for each character and, with fallback on by default, then any font it knows. To see what happens when none has the character, the session uses a Tamil word, with system fonts switched off:

// The notice in Typst, in its default font, Libertinus Serif.
Speaker: Ирина Петрова, from the field station. The office is efficient:
official badges are issued within a day.

// A Tamil word with no font for Tamil script asked for.
Thank you: நன்றி.

// The same word with the font list naming a Tamil font after the Latin one.
#set text(font: ("Libertinus Serif", "Noto Sans Tamil"))
Thank you: நன்றி.

from session import files, run

# Typst's fonts with system fonts ignored: the notice in the default font, a Tamil word with no Tamil font, the same
# word with a font list that names one, and both again with a folder that holds Noto Sans Tamil, which the second
# file names and the first finds by fallback.
files("notice/notice.typ", "tamil/thanks.typ", "tamil/thanks-named.typ",
      "fonts/NotoSansTamil-Regular.ttf=fonts/NotoSansTamil-Regular.ttf")
run(r"""
typst fonts
typst compile notice.typ
pdffonts notice.pdf
pdftotext notice.pdf - | head -1
typst compile thanks.typ
pdftotext thanks.pdf - | head -1
mutool draw -F txt thanks.pdf 2>/dev/null | head -1
typst compile thanks-named.typ
typst fonts --font-path fonts
typst compile --font-path fonts thanks-named.typ
pdffonts thanks-named.pdf
pdftotext thanks-named.pdf - | head -1
typst compile --font-path fonts thanks.typ thanks-fallback.pdf
pdftotext thanks-fallback.pdf - | head -1
""")

$ typst fonts
DejaVu Sans Mono
Libertinus Serif
New Computer Modern
New Computer Modern Math
$ typst compile notice.typ
$ pdffonts notice.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
WNBPYW+LibertinusSerif-Regular-Identity-H CID Type 0C       Identity-H       yes yes yes      8  0
$ pdftotext notice.pdf - | head -1
Speaker: Ирина Петрова, from the field station. The office is efficient: official badges are issued
$ typst compile thanks.typ
$ pdftotext thanks.pdf - | head -1
Thank you: ன்றி.
$ mutool draw -F txt thanks.pdf 2>/dev/null | head -1
Thank you: �ன்றி.
$ typst compile thanks-named.typ
warning: unknown font family: noto sans tamil
  ┌─ thanks-named.typ:2:16
  │
2 │ #set text(font: ("Libertinus Serif", "Noto Sans Tamil"))
  │                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

$ typst fonts --font-path fonts
DejaVu Sans Mono
Libertinus Serif
New Computer Modern
New Computer Modern Math
Noto Sans Tamil
$ typst compile --font-path fonts thanks-named.typ
$ pdffonts thanks-named.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
QBLHXJ+LibertinusSerif-Regular-Identity-H CID Type 0C       Identity-H       yes yes yes      8  0
BTLLDN+NotoSansTamil-Regular         CID TrueType      Identity-H       yes yes yes     11  0
$ pdftotext thanks-named.pdf - | head -1
Thank you: நன்றி.
$ typst compile --font-path fonts thanks.typ thanks-fallback.pdf
$ pdftotext thanks-fallback.pdf - | head -1
Thank you: நன்றி.

The notice needed nothing: Libertinus Serif has Cyrillic letters, and the text read the name back. With only the four built-in fonts, no font had Tamil letters, so Typst drew a box, a tofutofu: The empty box drawn where no font has a glyph for a character, named after its shape. Typst draws one when no font it knows has the character, and prints no warning., for each, printed nothing, and the text read back as ன்றி from pdftotext, one letter short. thanks-named.typ names Noto Sans Tamil in its font list, and that did draw a warning, unknown font family: noto sans tamil. With --font-path fonts, typst fonts listed Noto Sans Tamil, both files drew the word, and the text read நன்றி. Step through one letter:

Copying, search and screen readers get the PDF’s text, not its look

A PDF draws glyphs; the characters behind them come from a map that each font carries. The pdffonts manual describes its uni column as yes when the PDF has an explicit ToUnicode mapToUnicode map: The table in a PDF that says which characters each glyph of a font stands for, so that its text can be copied, searched and read aloud. pdffonts prints yes in its uni column when a font has one. for the font, and every PDF in this lesson said yes. Ligaturesligature: One glyph that stands for two or more letters, such as ffi or fl. The PDF must map it back to those letters, or copied and searched text comes out wrong. test that map: “efficient” and “official” are set with one ffi glyph, which must read back as three letters. LaTeX News 33 says pdfTeX’s PDFs now get that information automatically, so the most common ligatures copy as intended. This session builds the notice with all four engines and reads it back with poppler’s pdftotext and MuPDF’s mutool:

from session import files, read, run, shell, write

# The words "efficient" and "official" from the notice, set with their ffi ligatures by four engines, and read back
# by two text extractors: poppler's pdftotext and MuPDF's mutool. The LaTeX copies spell the speaker's name in Latin
# letters, which Latin Modern has. The table shows the start of the second sentence as each extractor reads it.
files("notice/notice-pdf.tex", "notice/notice.tex", "notice/notice.typ")
write("notice-latin.tex", read("notice-pdf.tex").replace("Ирина Петрова", "Irina Petrova"))
write("notice-lm.tex", read("notice.tex").replace("Ирина Петрова", "Irina Petrova"))
builds = {
    "pdfLaTeX": "pdflatex -interaction=nonstopmode notice-latin.tex > /dev/null && mv notice-latin.pdf pdflatex.pdf",
    "LuaLaTeX": "lualatex -interaction=nonstopmode notice-lm.tex > /dev/null && mv notice-lm.pdf lualatex.pdf",
    "XeLaTeX": "xelatex -interaction=nonstopmode notice-lm.tex > /dev/null && mv notice-lm.pdf xelatex.pdf",
    "Typst": "typst compile notice.typ typst.pdf",
}
for engine, command in builds.items():
    status, output = shell(command)
    if status:
        raise SystemExit(f"{engine} failed: {output}")
print("# the start of the second sentence, as each extractor reads it from each PDF")
for engine in builds:
    pdf = engine.lower() + ".pdf"
    for tool, command in [("poppler", f"pdftotext {pdf} -"), ("MuPDF", f"mutool draw -F txt {pdf} 2>/dev/null")]:
        _, text = shell(command)
        line = " ".join(text.split())
        sentence = line[line.index("The o"):]
        print(f"{engine:9} {tool:8} {sentence[:48]}")
run(r"""
pdffonts xelatex.pdf
""")

# the start of the second sentence, as each extractor reads it from each PDF
pdfLaTeX  poppler  The office is efficient: official badges are iss
pdfLaTeX  MuPDF    The office is efficient: official badges are iss
LuaLaTeX  poppler  The office is efficient: official badges are iss
LuaLaTeX  MuPDF    The office is efficient: official badges are iss
XeLaTeX   poppler  The office is efficient: official badges are issued
XeLaTeX   MuPDF    The office is efficient: official badges are iss
Typst     poppler  The office is efficient: official badges are iss
Typst     MuPDF    The office is efficient: official badges are iss
$ pdffonts xelatex.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
SLWVZF+LMRoman10-Regular-Identity-H  CID Type 0C       Identity-H       yes yes yes      4  0

Seven of the eight pairs read the sentence exactly. poppler read XeLaTeX’s PDF as “office”, “efficient” and “official”, with the single character ff, U+FB00, standing for two letters, while MuPDF read the same file correctly. A portal or search that does not normalise that character would not find the word “efficient” in the PDF.

Practise: a font check for every build

Two checks catch most of what this lesson showed before a reader does: a scan of the log for lost characters, and a read of pdffonts for fonts that are not embedded or have no Unicode map. The sample tests use the log lines and tables this page recorded, and the first hint of each step is open:

def missing_characters(log):
    """Return (code, character, font) for every character the log says a font had no glyph for, once each."""
    # Replace this comment: find each "There is no … in font …" report and collect what it names.
    return []

from glyphs import missing_characters

LUALATEX = """Missing character: There is no И (U+0418) in font [lmroman10-regular]:+tlig;!
Missing character: There is no И (U+0418) in font [lmroman10-regular]:+tlig;!
Missing character: There is no р (U+0440) in font [lmroman10-regular]:+tlig;!
Missing character: There is no И (U+0418) in font [lmroman10-regular]:+tlig;!
Output written on notice.pdf (1 page, 5621 bytes).
"""


def test_lualatex_repeats_a_character():
    """lists each lost letter once, in the order the log first reports it"""
    assert missing_characters(LUALATEX) == [("U+0418", "И", "lmroman10-regular"), ("U+0440", "р", "lmroman10-regular")]


def test_xelatex_and_the_strict_error():
    """reads XeLaTeX's warning and the error that \\tracinglostchars=3 turns it into"""
    log = ("Missing character: There is no И (U+0418) in font [lmroman10-regular]:mapping=tex-text;!\n"
           "! Missing character: There is no ✓ (U+2713) in font [lmroman10-regular]:mapping=tex-text;.\n")
    assert missing_characters(log) == [("U+0418", "И", "lmroman10-regular"), ("U+2713", "✓", "lmroman10-regular")]


def test_old_style_codes_in_nullfont():
    """turns "53 into U+0053 and keeps the order of the log"""
    log = ('Missing character: There is no S ("53) in font nullfont!\n'
           'Missing character: There is no p ("70) in font nullfont!\n'
           'Missing character: There is no S ("53) in font nullfont!\n')
    assert missing_characters(log) == [("U+0053", "S", "nullfont"), ("U+0070", "p", "nullfont")]


def test_the_same_character_in_two_fonts():
    """keeps one entry for each font that lacks the character"""
    log = ("Missing character: There is no И (U+0418) in font [lmroman10-regular]:+tlig;!\n"
           "Missing character: There is no И (U+0418) in font [lmroman10-bold]:+tlig;!\n")
    assert missing_characters(log) == [("U+0418", "И", "lmroman10-regular"), ("U+0418", "И", "lmroman10-bold")]


def test_a_clean_log():
    """returns an empty list when nothing was lost"""
    assert missing_characters("Output written on notice-file.pdf (1 page, 7057 bytes).\n") == []

def font_problems(table):
    """Return "<name>: not embedded" and "<name>: no Unicode map" for the fonts of a pdffonts table that need them."""
    # Replace this comment: read each font line after the two header lines, find its emb and uni values.
    return []

from fonts import font_problems

HEADER = ("name                                 type              encoding         emb sub uni object ID\n"
          "------------------------------------ ----------------- ---------------- --- --- --- ---------\n")


def test_the_lessons_notice():
    """finds nothing wrong with LuaLaTeX's notice.pdf"""
    table = HEADER + "VFSOVW+LMRoman10-Regular             CID Type 0C       Identity-H       yes yes yes      4  0\n"
    assert font_problems(table) == []


def test_a_long_name_shifts_the_columns():
    """reads both fonts of thanks-named.pdf, whose first name is longer than its column"""
    table = HEADER + (
        "QBLHXJ+LibertinusSerif-Regular-Identity-H CID Type 0C       Identity-H       yes yes yes      8  0\n"
        "BTLLDN+NotoSansTamil-Regular         CID TrueType      Identity-H       yes yes yes     11  0\n")
    assert font_problems(table) == []


def test_a_font_that_is_not_embedded():
    """reports a font whose emb column says no"""
    table = HEADER + "Arial-BoldMT                         TrueType          WinAnsi          no  no  yes      9  0\n"
    assert font_problems(table) == ["Arial-BoldMT: not embedded"]


def test_a_font_without_a_unicode_map():
    """reports a font whose uni column says no"""
    table = HEADER + "QWERTY+NotoSansTamil-Regular         CID TrueType      Identity-H       yes yes no      11  0\n"
    assert font_problems(table) == ["QWERTY+NotoSansTamil-Regular: no Unicode map"]


def test_both_problems_and_a_blank_line():
    """lists the embedding problem first and skips a blank line at the end"""
    table = HEADER + "Helvetica                            Type 1            Custom           no  no  no       5  0\n\n"
    assert font_problems(table) == ["Helvetica: not embedded", "Helvetica: no Unicode map"]

Afterwards, the self-check below covers engines, font loading, lost characters, Typst’s fallback and extracted text, beginning with whichever of them you are least sure of:

Interview questions

Warm-up (fresher to 2–5 years): when would you build a LaTeX document with XeLaTeX or LuaLaTeX instead of pdfLaTeX?

When it needs fonts or characters that pdfLaTeX cannot give it. pdfLaTeX uses 8-bit Type 1 fonts chosen through a font encoding and packages such as lmodern, and it stops on a character that its fonts are not set up for, as it did on the И of a Russian name. XeLaTeX and LuaLaTeX read Unicode and load any OpenType font through fontspec, which pdfLaTeX cannot load at all. Between the two: LuaLaTeX finds fonts of the TeX tree by name, while XeLaTeX needs their file names on macOS and on arXiv; and arXiv’s list of processors includes xelatex but not LuaLaTeX. Keep pdfLaTeX when a template or a venue asks for it and its fonts are enough. With either Unicode engine, add \tracinglostchars=3, because a character the font lacks is otherwise only a line in the log.

Two harder questions follow, about names that vanished from conference badges and about a font that a build server cannot find. Decide where you would start before you open each first step:

Key takeaways

• pdfLaTeX sets 8-bit Type 1 fonts through font encodings, while XeLaTeX and LuaLaTeX set OpenType fonts through fontspec, which stops pdfLaTeX with a fatal error.
• LuaLaTeX finds a TeX tree font such as Libertinus Serif by name, while XeLaTeX on macOS and on arXiv finds it only by file name, so file names keep a document building on every machine.
• A character the font lacks costs only a Missing character line and a replacement character in the text, until \tracinglostchars=3 makes it an error; choose a font that has the character.
• Typst falls back to any font it knows and then draws a tofu without a word, but warns about a named family it cannot find, so name your fonts and pass their folder with --font-path.
• Copying and search read the PDF’s text map, not the page: check it with pdffonts and two extractors, because poppler misread XeLaTeX’s ligatures where MuPDF did not.