YAML anchors, aliases and merge keys

A YAML file of 312 bytes holds eight levels: a list of four strings, then seven lists that each repeat the level before them four times through aliases. The last level expands to 65,536 strings, 502,441 bytes once it is written out as JSON. The feature behind it, writing a node once and reusing it, is what keeps CI pipelines and Compose files short, and it hides two more surprises: reused data is shared in memory, and merge keys follow rules that not every parser applies.

You will write anchors and aliases and see what a loader builds from them, change one job and watch two change, apply merge keys by the rules of the YAML 1.1 merge type, check which tools apply them at all, and put a limit on alias expansion before you load a file someone else wrote.

An anchor names a node, and an alias reuses it

An anchoranchor: A name given to a YAML node with &, such as &defaults, so that the node can be reused later in the same document by an alias., &name, marks a node, and an aliasalias: A YAML node written *name that stands for the most recent node anchored with that name earlier in the document. A loader shares that node rather than copying it., *name, stands for that node again later in the document. The YAML 1.2.2 specification ties each alias to “the most recent preceding node having the same anchor”, so an alias must come after its anchor, in the same document. Here are the jobs of the bookshop’s CI pipeline, whose defaults are written once:

# Jobs of the bookshop's CI pipeline: the defaults are written once
defaults: &defaults
  image: ruby:4.0
  retries: 2
  timeout_minutes: 10
jobs:
  test: *defaults
  lint: *defaults

Step through the file as a parser and then a loader see it, and count the mappings at each step:

The program loads the file with Psych, the YAML parser in Ruby’s standard library, changes one job, copies the jobs, tries an alias with no anchor, and writes the data back out:

# Anchors and aliases in Psych: refused by default, then shared, then copied.
require "yaml"

text = File.read("ci.yaml")
begin
  YAML.safe_load(text)
rescue Psych::AliasesNotEnabled => e
  puts "Default safe_load: #{e.class}"
end

ci = YAML.safe_load(text, aliases: true)
test, lint = ci["jobs"].values_at("test", "lint")
puts "test equals lint: #{test == lint}; the same object: #{test.equal?(lint)}"

test["retries"] = 5
puts "After test.retries = 5: lint #{lint['retries']}, defaults #{ci['defaults']['retries']}"

# The fix: give every job its own deep copy before changing anything.
ci = YAML.safe_load(text, aliases: true)
ci["jobs"].transform_values! { |job| Marshal.load(Marshal.dump(job)) }
ci["jobs"]["test"]["retries"] = 5
puts "With copies: test #{ci['jobs']['test']['retries']}, lint #{ci['jobs']['lint']['retries']}"

# An alias needs an anchor earlier in the same document.
begin
  YAML.safe_load("jobs:\n  deploy: *release\n", aliases: true)
rescue Psych::AnchorNotDefined => e
  puts "Alias without an anchor: #{e.message}"
end

# What Psych writes back for data that is shared.
shared = YAML.safe_load(text, aliases: true)
puts
puts YAML.dump(shared)

Default safe_load: Psych::AliasesNotEnabled
test equals lint: true; the same object: true
After test.retries = 5: lint 5, defaults 5
With copies: test 5, lint 2
Alias without an anchor: An alias referenced an unknown anchor: release

---
defaults: &1
  image: ruby:4.0
  retries: 2
  timeout_minutes: 10
jobs:
  test: *1
  lint: *1

The first line is Psych’s default: safe_load refuses every alias with Psych::AliasesNotEnabled until the program passes aliases: true, a deliberate choice that the last section of this lesson explains. With aliases allowed, test and lint are not only equal, they are the same object, exactly as the steps showed. An alias with no anchor before it is an error, Psych::AnchorNotDefined, as the specification requires.

GitHub’s documentation for Actions says that anchors and aliases “reduce repetition in your workflows”, and Docker’s Compose reference calls them a way to “create reusable blocks”. An alias always stands for a whole node; changing part of a reused node is the job of the merge keys below.

Aliases share one object, so one change shows up everywhere

An alias is not a copy. A loader builds the anchored node once and puts the same object wherever an alias stands, and PyYAML’s documentation says the same of Python: anchors and aliases are how a document refers “to the same object from different parts”. Here is the data that a loader builds for test: *defaults and lint: *defaults, written as Python so you can step through it:

# A loader's result for test: *d and
# lint: *d is one dict with two names.
defaults = {"retries": 2}
jobs = {"test": defaults, "lint": defaults}
jobs["test"]["retries"] = 5
print(jobs["lint"]["retries"])

# The fix: one copy for each job.
defaults = {"retries": 2}
jobs = {"test": dict(defaults), "lint": dict(defaults)}
jobs["test"]["retries"] = 5
print(jobs["lint"]["retries"])

5
2

Line 5 changed test, and line 6 found the change in lint, because both names lead to one dict. The second half gives each job its own copy first, and lint keeps 2. The Ruby run above showed the same in Psych: after test["retries"] = 5, both lint and defaults read 5, and with a deep copy per job, made with Marshal, lint kept 2.

The last lines of the Ruby output show what happens on the way back out. YAML.dump noticed the shared object and wrote it with an anchor of its own, &1 where the file had &defaults, and *1 for both jobs: the sharing survived, the names did not. A program that loads, changes and dumps a file with aliases rewrites its anchors.

Merge keys copy keys, and the mapping’s own keys win

An alias reuses a whole node. To reuse most of a mapping and change one key, YAML 1.1 has the merge keymerge key: The YAML 1.1 key <<, whose value is a mapping or a list of mappings whose keys are copied into the current mapping, except keys it already has. YAML 1.2 removed it., <<. The type page says that the keys of the merged mappings are inserted into the current mapping “unless the key already exists in it”, and that with a list of mappings “keys in mapping nodes earlier in the sequence override” the later ones. These four jobs use it:

# Merge keys: copy the keys of other mappings into this one
base: &base
  image: ruby:4.0
  retries: 2
  env: {CI: "true"}
slow: &slow
  timeout_minutes: 30
  retries: 5
test:
  <<: *base
  retries: 3              # a key of the mapping itself wins
deploy:
  <<: [*base, *slow]      # the earlier mapping wins: retries from base
lint:
  <<: *base
  env: {LINT: "on"}       # shallow: env is replaced, not combined
late:
  retries: 9
  <<: *base               # the merge key after the key

Step through three of them, one merged mapping at a time:

The program prints what Psych 5.3.1 built for each job:

# What Psych 5.3.1 builds from the merge keys of jobs.yaml.
require "yaml"

jobs = YAML.safe_load(File.read("jobs.yaml"), aliases: true)
%w[test deploy lint late].each do |name|
  puts "#{name.ljust(7)} #{jobs[name].inspect}"
end

# The merge type of YAML 1.1 keeps a key that the mapping already has,
# wherever the merge key stands. Here is what Psych did with late's retries:
puts
puts "late retries: #{jobs['late']['retries']} (the mapping says 9)"

test    {"image" => "ruby:4.0", "retries" => 3, "env" => {"CI" => "true"}}
deploy  {"timeout_minutes" => 30, "retries" => 2, "image" => "ruby:4.0", "env" => {"CI" => "true"}}
lint    {"image" => "ruby:4.0", "retries" => 2, "env" => {"LINT" => "on"}}
late    {"retries" => 2, "image" => "ruby:4.0", "env" => {"CI" => "true"}}

late retries: 2 (the mapping says 9)

test kept its own retries: 3 and took image and env from base. deploy merged two mappings and took retries: 2 from base, the earlier one, and only timeout_minutes from slow. lint shows that merging is shallow: its own env replaced the merged one whole, so the CI variable is gone rather than combined with LINT.

late is where Psych and the type page part. By the rule, retries: 9 belongs to the mapping, so it should win whatever its position. Psych applied the merge in the order of the lines and let base overwrite it with 2. Writing << first in its mapping, as Docker’s Compose reference does in its examples, gives the same result in Psych and by the rule.

Merge keys are YAML 1.1, so check every tool that reads the file

The merge key was never part of YAML 1.2. The specification changes page says that “the merge << and value = special mapping keys have been removed”, so a parser that follows only the core schema reads << as an ordinary key whose value is a mapping. What a given tool does is in its documentation, or in a probe:

When one reader of a file merges and another does not, they see different data: the second one finds a key called << and no image in the job. For files that several tools read, either check that every one of them merges, or avoid << and repeat the keys, or let the program apply merge keys itself after loading, which is the exercise of this lesson. In workflow files, use anchors and aliases only, the features the documentation describes.

Aliases can multiply data, so cap the expansion

Every alias in memory is a pointer, but whatever turns the data into text, JSON or a copy follows each one. A document in which every level repeats the level before it several times grows like a power. This one has eight levels of four:

# Each level holds four aliases of the level before it, eight levels in all
l1: &l1 [book, book, book, book]
l2: &l2 [*l1, *l1, *l1, *l1]
l3: &l3 [*l2, *l2, *l2, *l2]
l4: &l4 [*l3, *l3, *l3, *l3]
l5: &l5 [*l4, *l4, *l4, *l4]
l6: &l6 [*l5, *l5, *l5, *l5]
l7: &l7 [*l6, *l6, *l6, *l6]
l8: &l8 [*l7, *l7, *l7, *l7]

The program loads it, measures each level as JSON, and then loads it again through a guard that counts the nodes the document expands to on the parse tree, where every alias is still a single node, and refuses it above a limit:

# How far aliases multiply data, and a limit checked on the parse tree before loading.
require "yaml"
require "json"

text = File.read("nested.yaml")
puts "nested.yaml: #{text.bytesize} bytes"

# Loading builds each level once; the aliases share it. Writing it out does not.
data = YAML.safe_load(text, aliases: true)
puts format("%-6s %8s %11s", "level", "strings", "JSON bytes")
(1..8).each do |n|
  level = data["l#{n}"]
  puts format("%-6s %8d %11d", "l#{n}", level.flatten.size, JSON.generate(level).bytesize)
end

# The defence: count the nodes the document expands to on the parse tree,
# where every alias is still one node, and refuse it before any value is built.
def expanded(node, anchors, memo = {}.compare_by_identity)
  return memo[node] if memo.key?(node)

  size = case node
         when Psych::Nodes::Alias then expanded(anchors.fetch(node.anchor), anchors, memo)
         when Psych::Nodes::Scalar then 1
         else 1 + node.children.sum { |child| expanded(child, anchors, memo) }
         end
  anchors[node.anchor] = node if !node.is_a?(Psych::Nodes::Alias) && node.anchor
  memo[node] = size
end

def guarded_load(text, limit:)
  size = expanded(YAML.parse(text).root, {})
  raise ArgumentError, "expands to #{size} nodes, more than #{limit}" if size > limit

  YAML.safe_load(text, aliases: true)
end

puts
[["the first three levels", text.lines.first(4).join], ["all eight levels", text]].each do |what, yaml|
  loaded = guarded_load(yaml, limit: 10_000)
  puts "#{what}: loaded #{loaded.size} keys"
rescue ArgumentError => e
  puts "#{what}: refused, #{e.message}"
end

nested.yaml: 312 bytes
level   strings  JSON bytes
l1            4          29
l2           16         121
l3           64         489
l4          256        1961
l5         1024        7849
l6         4096       31401
l7        16384      125609
l8        65536      502441

the first three levels: loaded 3 keys
all eight levels: refused, expands to 116513 nodes, more than 10000

The 312 bytes loaded quickly, because Psych built each level once and shared it. Written out, they multiply by four per level: 4 strings at l1, 65,536 at l8, and 502,441 bytes of JSON for the last level alone, about 1,600 times the size of the file. With more levels or wider lists the same few lines can exhaust a program’s memory. The alias expansionalias expansion: The growth of YAML data when its aliases are followed and copied, as when it is written out as JSON: each level of aliases multiplies the size, so a short file can become very large. count of the guard is computed without building anything: 116,513 nodes for the whole document, so with a limit of 10,000 the guard loaded the first three levels and refused all eight.

Parsers ship such limits. Psych refuses every alias unless the program asks for them with aliases: true. The yaml package for JavaScript has maxAliasCount, 100 by default, to “prevent exponential entity expansion attacks by limiting data aliasing”.

Practise: apply merge keys after loading

A loader that follows YAML 1.2 returns << as an ordinary key, so a program that wants merges must apply them itself. The exercise asks you to do it by the rules of the merge type, own keys first, earlier mappings next, shallow, at any depth, with the four jobs of this lesson among its sample tests:

def apply_merge_keys(value):
    """value with every "<<" merge key applied, by the YAML 1.1 merge type, in mappings at any depth."""
    return value

from merge import apply_merge_keys

BASE = {"image": "ruby:4.0", "retries": 2, "env": {"CI": "true"}}
SLOW = {"timeout_minutes": 30, "retries": 5}


def test_one_mapping_and_an_own_key():
    """copies the keys of one mapping, and keeps the mapping's own key"""
    assert apply_merge_keys({"<<": BASE, "retries": 3}) == {"image": "ruby:4.0", "retries": 3, "env": {"CI": "true"}}


def test_a_list_of_mappings():
    """lets the earlier mapping of the list win"""
    assert apply_merge_keys({"<<": [BASE, SLOW]}) == {"image": "ruby:4.0", "retries": 2, "env": {"CI": "true"}, "timeout_minutes": 30}
    assert apply_merge_keys({"<<": [SLOW, BASE]})["retries"] == 5


def test_merging_is_shallow():
    """replaces a nested mapping instead of combining it"""
    assert apply_merge_keys({"<<": BASE, "env": {"LINT": "on"}})["env"] == {"LINT": "on"}


def test_the_order_of_lines_does_not_matter():
    """keeps the mapping's own key even when it was written before the merge key"""
    late = {"retries": 9, "<<": BASE}
    assert apply_merge_keys(late)["retries"] == 9


def test_mappings_at_any_depth():
    """applies merge keys inside nested mappings and lists, and drops the << key"""
    doc = {"jobs": [{"name": "test", "<<": BASE}], "base": BASE}
    result = apply_merge_keys(doc)
    assert result["jobs"][0] == {"name": "test", "image": "ruby:4.0", "retries": 2, "env": {"CI": "true"}}
    assert "<<" not in result["jobs"][0]
    assert result["base"] == BASE


def test_inputs_stay_unchanged():
    """builds new mappings and leaves the input as it was"""
    job = {"<<": BASE, "retries": 3}
    apply_merge_keys(job)
    assert job == {"<<": BASE, "retries": 3}
    assert BASE["retries"] == 2


def test_a_value_that_cannot_be_merged():
    """refuses a merge key whose value is not a mapping or a list of mappings"""
    try:
        apply_merge_keys({"<<": [1, 2]})
    except ValueError:
        return
    assert False, "a list of numbers is not a list of mappings: expected ValueError"

The session below mixes anchors, sharing, merge rules, tool support and expansion limits, and begins with the topic you have answered least well so far:

Interview questions

Warm-up (fresher to 2–5 years): anchor, alias and merge key: what does each one do?

An anchor, &name, marks a node so that it can be reused. An alias, *name, stands for the most recent node with that anchor earlier in the same document; it is not a copy, so a loader builds the node once and shares it, and a change through one alias shows up through all of them. A merge key, <<: *name or <<: [*a, *b], copies the keys of other mappings into the current one, except keys the mapping already has; with a list, earlier mappings win, and nested mappings are replaced whole. Anchors and aliases are part of YAML 1.2. The merge key is a YAML 1.1 type that YAML 1.2 removed, so some tools apply it and others keep << as an ordinary key.

Two questions follow that start from things that went wrong: a script that changed more jobs than it meant to, and a small file that took a tool’s memory. Think through the first step of each before you read it:

Key takeaways

• &name marks a node and *name reuses the most recent node with that anchor in the same document; Psych refuses aliases until the program allows them.
• An alias is the same object, not a copy: after one change both jobs read 5, and YAML.dump writes the sharing back with its own anchors such as &1.
• A merge key adds only the keys a mapping lacks, takes them from the earlier mapping of a list first, and is shallow; Psych applies it in line order, so put << first.
• The merge key is YAML 1.1 and was removed in YAML 1.2, so check every tool that reads a file before using it.
• Aliases multiply data: 312 bytes became 502,441 bytes of JSON; count the expansion on the parse tree and refuse files above a limit, or keep aliases off for files you did not write.