YAML essentials: mappings, sequences, scalars

The Pune branch of the bookshop keeps its settings in a YAML file of 13 lines. Put a tab in place of the four spaces before one key, and Psych, the YAML parser in Ruby’s standard library, refuses the whole file and points at line 7, one line above the tab. YAML reads a file’s structure from its indentation and the type of each value from its text, so a single character decides whether a file loads and what each value becomes.

You will build that file up step by step: nest mappings and sequences with spaces, write the same data in flow style and read JSON as YAML, choose between plain, single-quoted and double-quoted strings, and keep several documents in one file. Every example was read by Psych, so each rule comes with what a real parser did with it.

Indentation is the structure: mappings and sequences

A YAML document is built from three kinds of node. A mappingmapping: A YAML collection of keys with their values, written key: value one pair per line in block style, or {key: value} in flow style. The keys of one mapping must be unique. holds keys with their values, written key: value, one pair per line. A sequencesequence: A YAML collection of items in order, written one per line after a dash and a space in block style, or between square brackets in flow style. holds items in order, each written after a dash and a space. A scalarscalar: A single YAML value that is not a collection: text that the parser turns into a string, a number, a boolean or null. is one value: a string, a number, true or false, or null. Indentation, counted in spaces, decides which node a line belongs to. Here is the Pune branch’s file:

# Settings of the Pune branch of the bookshop
branch: Pune
opens: "09:30"
gift_wrap: true
counters: 3
printers:
  - name: front-desk
    paper: A4
  - name: stockroom
    paper: label
manager:
  name: Asha Kulkarni
  email: pune@bookshop.example

The root is a mapping whose keys have no indentation. printers has no value on its own line, so its value is the block indented under it: a sequence of two items. Each item is a mapping, and its second key, paper, sits exactly under name, the first key after the dash. manager holds a mapping of two keys. Step through how a parser decides, line by line:

The program below asks Psych for the same file and prints what it built as JSON, whose brackets and braces show the nesting. Then it makes two small changes and reads the file again each time:

# Read branch.yaml with Psych, the YAML parser in Ruby's standard library,
# and print the data it built as JSON, so the nesting is easy to see.
require "yaml"
require "json"

text = File.read("branch.yaml")
puts JSON.pretty_generate(YAML.safe_load(text))

# Two small changes, each read again.
changes = {
  "a tab before paper on line 8" => ["    paper: A4", "\tpaper: A4"],
  "paper on line 10 moved to column 0" => ["    paper: label", "paper: label"],
}
changes.each do |what, (from, to)|
  puts
  puts "With #{what}:"
  begin
    data = YAML.safe_load(text.sub(from, to))
    puts "  loaded. printers: #{data['printers'].inspect}"
    puts "  and a new root key paper: #{data['paper'].inspect}"
  rescue Psych::SyntaxError => e
    puts "  refused: #{e.message}"
  end
end

{
  "branch": "Pune",
  "opens": "09:30",
  "gift_wrap": true,
  "counters": 3,
  "printers": [
    {
      "name": "front-desk",
      "paper": "A4"
    },
    {
      "name": "stockroom",
      "paper": "label"
    }
  ],
  "manager": {
    "name": "Asha Kulkarni",
    "email": "pune@bookshop.example"
  }
}

With a tab before paper on line 8:
  refused: (<unknown>): found a tab character that violates indentation while scanning a plain scalar at line 7 column 11

With paper on line 10 moved to column 0:
  loaded. printers: [{"name" => "front-desk", "paper" => "A4"}, {"name" => "stockroom"}]
  and a new root key paper: "label"

The JSON is the tree the steps built: a mapping of six keys, a sequence of two mappings under printers and a mapping under manager. Psych also typed the plain values: 3 became an integer and true a boolean, while "09:30" stayed a string because of its quotes.

The first change, a tab before paper, stopped everything. The YAML 1.2.2 specification forbids tabs in indentation “to maintain portability”, because different systems treat tabs differently. Psych’s message names line 7, column 11: the start of front-desk, the value it was still reading when it met the tab, not the tab itself. When an error points at a line that looks right, look at the start of the next line.

The second change is worse, because nothing complains. With paper: label moved to the left margin, the line is no longer inside the second item: it is a new key of the root mapping, and the stockroom printer has no paper at all.

Three rules cover most block layouts. Any number of spaces nests a block, two in this file, and every key of one mapping must start at the same indentation. A key needs a colon followed by a space or a line break, so name:front-desk is one string, not a key and a value. And a mapping inside a sequence item lines its keys up with the first key after the dash, not with the dash.

Flow style looks like JSON, and JSON text is YAML 1.2

Block style is one way to write collections. Flow styleflow style: YAML written with brackets and braces, [a, b] for a sequence and {k: v} for a mapping, with commas between entries, as JSON is. Block style uses indentation and dashes instead. is the other: a sequence in square brackets, a mapping in braces, entries separated by commas, on one line or several. Here are the two printers both ways:

# The printers of the Pune branch in block style, as in branch.yaml
printers:
  - name: front-desk
    paper: A4
  - name: stockroom
    paper: label

# The same printers in flow style: brackets and braces, as in JSON
printers: [{name: front-desk, paper: A4}, {name: stockroom, paper: label}]

Flow style is compact, which suits short lists such as tags: [fiction, travel]; block style suits anything people edit line by line, because each line holds one entry. Inside brackets and braces a comma ends a plain value, so a value that holds a comma needs quotes.

Flow style is also where JSON meets YAML. A JSON object is a flow mapping whose keys and strings are double-quoted scalars, and a JSON array is a flow sequence. The YAML 1.2 specification says its “primary focus was making YAML a strict superset of JSON”, so a parser that follows YAML 1.2 reads JSON text and builds the same values. The exception is an object that repeats a name: RFC 8259 only says that names should be unique, while YAML requires the keys of a mapping to be unique. This program compares the two printer files, reads a flow list with a comma inside a value, reads the bookshop’s order.json with Ruby’s JSON parser and with Psych, and finally tries three numbers:

# Flow style, and JSON text read by a YAML parser (Psych) and by a JSON parser.
require "yaml"
require "json"

block = YAML.safe_load(File.read("printers-block.yaml"))
flow = YAML.safe_load(File.read("printers-flow.yaml"))
puts "Flow printers equal the block ones: #{flow == block}"

# Inside brackets a comma ends a plain scalar: quote a value that holds one.
puts YAML.safe_load('tags: [fiction, kids, teens, "kids, teens"]').inspect

text = File.read("order.json")
puts "order.json, YAML equals JSON: #{YAML.safe_load(text) == JSON.parse(text)}"
puts "receipt as YAML read it: #{YAML.safe_load(text)['receipt']}"

# Three numbers that JSON allows, with no space after the colons. Psych
# follows YAML 1.1, whose floats need a dot and a signed exponent.
numbers = '{"a":1.5e+3,"b":1e3,"c":2E-3}'
puts "JSON.parse:     #{JSON.parse(numbers)}"
puts "YAML.safe_load: #{YAML.safe_load(numbers)}"

Flow printers equal the block ones: true
{"tags" => ["fiction", "kids", "teens", "kids, teens"]}
order.json, YAML equals JSON: true
receipt as YAML read it: https://bookshop.example/r/BK-10042
JSON.parse:     {"a" => 1500.0, "b" => 1000.0, "c" => 0.002}
YAML.safe_load: {"a" => 1500.0, "b" => "1e3", "c" => "2E-3"}

The block and flow printers are the same data, and the list shows the comma rule: kids, teens written plain gave two items, while "kids, teens" in quotes gave one. The order read as YAML equals the order read as JSON, \/ escapes and all, and Psych also read the numbers written with no space after their colons: the specification changes list both details as new in YAML 1.2, the version that made YAML a superset of JSON.

The last two lines show the catch. Psych follows YAML 1.1, whose float type needs a dot and a sign before the exponent, so the JSON numbers 1e3 and 2E-3 came back as the strings "1e3" and "2E-3", while 1.5e+3 became 1500.0. A parser that follows the YAML 1.2 core schema reads all three as numbers, because its float pattern makes the dot and the sign optional. JSON text is YAML 1.2; a YAML 1.1 parser reads nearly all of it the same way, and the exceptions are values like these.

Plain, single-quoted or double-quoted: choose by what the text holds

Most strings in a YAML file need no quotes at all. A plain scalarplain scalar: A YAML scalar written without quotes. It cannot contain a colon and a space or a space and #, cannot start with most indicator characters, and the parser types it from its text. is text without quotes, and it may hold spaces, digits and most punctuation. Two pairs of characters are off limits inside it, because they mean something else there: a colon followed by a space would start a key, and a space followed by # starts a comment. A plain scalar also cannot start with most of the characters that YAML uses as indicators, such as [, {, &, *, !, |, >, ', ", %, @ and the backtick. When the text breaks one of these rules, quote it. This file holds fourteen values of the bookshop, each with a comment that says why it is written as it is:

# Strings of the bookshop, and how each one is written
title: The Little Prince          # plain: spaces inside are fine
sku: BK#101                       # plain: no space before #, so no comment
note: price #1 item               # plain: " #" starts a comment
command: 'echo Total: 5'          # single quotes: ": " is fine inside
apostrophe: 'it''s here'          # single quotes: '' stands for one '
win_path: 'C:\new\table'          # single quotes: backslashes stay as they are
win_path2: "C:\\new\\table"       # double quotes: \\ for each backslash
two_lines: "line one\nline two"   # double quotes: \n is a line break
city: "\u092A\u0941\u0923\u0947"  # double quotes: \u and four hex digits
padded: '  two spaces'            # quotes keep the leading spaces
qty: 2                            # plain digits: an integer
qty_text: "2"                     # quoted digits: a string
empty: ''                         # an empty string
nothing:                          # no value at all: null

The program prints the Ruby class and the value that Psych built from each line, then tries the command without quotes:

# What Psych builds from each line of quoting.yaml: the Ruby class and the value.
require "yaml"

YAML.safe_load(File.read("quoting.yaml")).each do |key, value|
  puts format("%-11s %-9s %s", key, value.class, value.inspect)
end

# The same command written plain: ": " inside a plain scalar is refused.
begin
  YAML.safe_load("command: echo Total: 5\n")
rescue Psych::SyntaxError => e
  puts
  puts "Plain command: #{e.message}"
end

title       String    "The Little Prince"
sku         String    "BK#101"
note        String    "price"
command     String    "echo Total: 5"
apostrophe  String    "it's here"
win_path    String    "C:\\new\\table"
win_path2   String    "C:\\new\\table"
two_lines   String    "line one\nline two"
city        String    "पुणे"
padded      String    "  two spaces"
qty         Integer   2
qty_text    String    "2"
empty       String    ""
nothing     NilClass  nil

Plain command: (<unknown>): mapping values are not allowed in this context at line 1 column 20

Read the output against the comments. BK#101 stayed whole, because no space came before its #, while price #1 item lost everything after price: the rest of the line was a comment. The single-quoted command kept its : , and 'it''s here' shows the only escape single quotes have: two quotes for one. A backslash means nothing in single quotes, so 'C:\new\table' kept both of them, while in double quotes each backslash had to be written \\ to give the same string.

Double quotes are the only style with escape sequencesescape sequence: A backslash and the characters after it, standing for one character inside a string, such as \n for a line feed or \u0041 for the letter A. More in “JSON syntax and data model (RFC 8259)”: \n became a line break, and the four \u escapes spelled पुणे, the name Pune in Devanagari. Quotes of either kind keep leading spaces, which a plain scalar loses.

The last four values are about types rather than quotes. Plain 2 became an integer, quoted "2" a string, '' an empty string, and a key with no value at all nil, Ruby’s null. A quoted scalar is always a string. A plain one is given its type by the parser, from its text, and that is where the traps of this module live. Last, the plain command echo Total: 5: its second : made Psych refuse the whole document.

All fourteen lines follow one rule of thumb:

One file can hold several documents

A YAML streamstream: A YAML text holding zero or more documents. A line --- starts a document and a line ... ends one, so one file can carry several documents. holds zero or more documents. A line --- starts a document, and a line ... ends one without starting the next, which lets a program that reads a stream as it arrives act on each document as soon as it is complete. Kubernetes uses streams to keep related resources together: its documentation groups them “in the same file (separated by --- in YAML)”. Here are the opening hours of three branches in one file:

# Opening hours of three branches, one document each
branch: Pune
opens: "09:30"
---
branch: Kochi
opens: "10:00"
...
---
# A document can start with a comment, too
branch: Indore
opens: "09:00"

The first document needs no ---, because a stream may begin with a bare document. After ..., YAML 1.2 would accept a bare document too, but YAML 1.1, whose rules Psych follows here, says that every later document begins with ---, so write it. A comment may stand at the start of a document, like anywhere else outside a scalar. The program reads the file twice, once as a stream and once as a single document, then tries two broken streams:

# One file, three documents: read them all, then see what reading one gives.
require "yaml"

text = File.read("branches.yaml")
docs = YAML.safe_load_stream(text)
puts "safe_load_stream: #{docs.size} documents"
docs.each { |doc| puts "  #{doc['branch']} opens at #{doc['opens']}" }

puts "safe_load:        #{YAML.safe_load(text).inspect}"

# A tab in the second document only: does the stream give back the other two?
broken = text.sub('opens: "10:00"', "\topens: \"10:00\"")
got = []
begin
  YAML.safe_load_stream(broken) { |doc| got << doc }
rescue Psych::SyntaxError => e
  puts "With a tab in document 2: #{got.size} documents, then #{e.class}"
end

# YAML 1.2 also accepts a bare document after "...". Psych, a YAML 1.1 parser, wants "---".
begin
  YAML.safe_load_stream("branch: Pune\n...\nbranch: Kochi\n")
rescue Psych::SyntaxError => e
  puts "A bare document after ...: #{e.problem}"
end

safe_load_stream: 3 documents
  Pune opens at 09:30
  Kochi opens at 10:00
  Indore opens at 09:00
safe_load:        {"branch" => "Pune", "opens" => "09:30"}
With a tab in document 2: 0 documents, then Psych::SyntaxError
A bare document after ...: did not find expected <document start>

safe_load_stream returned all three documents in order. safe_load, given the same text, returned the first one and said nothing about the other two, so a loader for one document is the wrong tool for a stream, and nothing warns you. The line about the tab shows that Psych reads a stream as a whole: with a tab in the second document, safe_load_stream handed over none of the three, not even the first, although the program gave it a block to receive each document. The last line is the --- rule at work: Psych refused a bare document after .... Psych also has a load_stream. Of the two, safe_load_stream is the one that takes the same safety options as safe_load, and the last lesson of this module shows why that matters for files you did not write.

Each document of a stream stands on its own. The specification changes list documents that “are now independent of each other”: a document does not inherit the directives of the one before, and an anchor, the reuse marker of the anchors lesson, works only inside its own document.

Practise: write strings that read back unchanged

A program that writes YAML has to decide, for every string, whether it can go out plain. The exercise asks you to make that decision with the rules of this lesson, and to write a double-quoted scalar, escapes included, whenever the plain form would change the string. Its sample tests use values from this lesson, such as echo Total: 5 and price #1 item:

def yaml_scalar(s):
    """The text to write after "key: " so that a YAML parser reads back exactly the string s."""
    return s

from scalar import yaml_scalar


def test_plain_text_stays_plain():
    """leaves ordinary text, a # without a space and a backslash plain"""
    assert yaml_scalar("The Little Prince") == "The Little Prince"
    assert yaml_scalar("BK#101") == "BK#101"
    assert yaml_scalar("C:\\new") == "C:\\new"
    assert yaml_scalar("Norway") == "Norway"


def test_colon_and_comment():
    """quotes text with ': ' or ' #' in it, or a colon at the end"""
    assert yaml_scalar("echo Total: 5") == '"echo Total: 5"'
    assert yaml_scalar("price #1 item") == '"price #1 item"'
    assert yaml_scalar("note:") == '"note:"'


def test_indicators_at_the_start():
    """quotes text that starts with an indicator character"""
    assert yaml_scalar("*bold*") == '"*bold*"'
    assert yaml_scalar("@pune") == '"@pune"'
    assert yaml_scalar("[draft]") == '"[draft]"'
    assert yaml_scalar("- item") == '"- item"'


def test_values_of_other_types():
    """quotes text that a parser would read as null, a boolean or a number"""
    assert yaml_scalar("2") == '"2"'
    assert yaml_scalar("3.10") == '"3.10"'
    assert yaml_scalar("NO") == '"NO"'
    assert yaml_scalar("null") == '"null"'
    assert yaml_scalar("~") == '"~"'
    assert yaml_scalar("-5") == '"-5"'
    assert yaml_scalar(".5") == '".5"'
    assert yaml_scalar(".inf") == '".inf"'
    assert yaml_scalar("+7") == '"+7"'


def test_empty_and_spaces():
    """quotes the empty string and text with a space at either end"""
    assert yaml_scalar("") == '""'
    assert yaml_scalar(" two") == '" two"'
    assert yaml_scalar("two ") == '"two "'


def test_escapes_inside_double_quotes():
    """escapes backslashes, quotes, line breaks and tabs inside the quotes"""
    assert yaml_scalar("line one\nline two") == '"line one\\nline two"'
    assert yaml_scalar("tab\there") == '"tab\\there"'
    assert yaml_scalar('"quoted"') == '"\\"quoted\\""'
    assert yaml_scalar("C:\\new: x") == '"C:\\\\new: x"'

Then check the four ideas, from indentation to streams, in a short session that starts with whichever idea you are least sure of and explains each choice as you make it:

Interview questions

Warm-up (fresher to 2–5 years): is every JSON document valid YAML?

For YAML 1.2, yes, with one exception: the specification made YAML a strict superset of JSON, so a JSON object is a flow mapping, an array a flow sequence, and a YAML 1.2 parser builds the same values from the same text. The exception is an object that repeats a name, which JSON only discourages and YAML forbids. Older YAML 1.1 parsers read almost all JSON the same way, but not every number: Psych, which follows YAML 1.1, read the JSON numbers 1e3 and 2E-3 as strings, because a YAML 1.1 float needs a dot and a signed exponent. The reverse is not true: most YAML is not JSON, since comments, plain strings, block style and anchors have no JSON form. In practice, read JSON with a JSON parser, and when YAML must accept JSON, test the parser you use with the number forms the other side writes.

Two harder questions follow: one about an error that points at the wrong line, one about keeping many documents in one file. Decide what you would check first, then open each one’s first step:

Key takeaways

• A YAML document is mappings, sequences and scalars, nested by indentation in spaces; a tab in the indentation stops the parser, and a line at the wrong level can load silently as different data.
• Flow style writes the same collections with brackets, braces and commas; JSON text is YAML 1.2, but a YAML 1.1 parser such as Psych read the JSON numbers 1e3 and 2E-3 as strings.
• Plain scalars cannot hold : or # or start with an indicator; single quotes escape only '', and double quotes are the only style with escape sequences such as \n and \u.
• A quoted scalar is always a string, while a plain one is typed by the parser, so 2 and "2" load differently.
• --- starts a document and ... ends one; safe_load_stream reads every document of a stream, while safe_load quietly returns only the first.