Schema evolution: change formats safely

Adding one optional field, delivery_slot, to an order schema made every new order fail the old validator, because the old schema said additionalProperties: false. The change looked harmless, and it would have stopped every service still running version 1 at the moment the first new order arrived.

Formats change because the data they carry changes, and the readers and writers of a format never all upgrade at the same instant. You will classify changes by the two directions of compatibility, design schemas that old readers can live with, apply the same rules to XML, CSV, configuration files and binary formats, and order a breaking change so that nothing stops.

Compatibility has two directions: old data and old readers

A change is backward compatiblebackward compatible: A schema change is backward compatible when readers using the new schema can read data written with the old one, so new code can meet old stored or queued data. when readers using the new schema can read data written with the old one, and forward compatibleforward compatible: A schema change is forward compatible when readers still using the old schema can read data written with the new one, so old code survives while producers move first. when readers still on the old schema can read data written with the new one. Fully compatible means both. The two directions matter at different moments: backward when new code meets stored or queued old data, forward when old code meets new data during a rollout.

This program checks four changes of an order schema both ways, with a small validator for the JSON Schema keywords the schemas use:

"""Six order schemas and a message written by each version's producer. For every change, two checks: the new schema on
old data (backward compatible: new readers read old data) and the old schema on new data (forward compatible: old
readers read new data)."""

CLOSED = {"additionalProperties": False}


def schema(fields, required, extra=None):
    return {"type": "object", "properties": fields, "required": required, **(extra or {})}


ID, AMOUNT, SLOT, CURRENCY = {"type": "string"}, {"type": "integer"}, {"type": "string"}, {"enum": ["INR", "USD"]}

VERSIONS = {
    "v1 closed": (schema({"id": ID, "amount_paise": AMOUNT}, ["id", "amount_paise"], CLOSED),
                  {"id": "A-1", "amount_paise": 125000}),
    "v2 closed": (schema({"id": ID, "amount_paise": AMOUNT, "delivery_slot": SLOT}, ["id", "amount_paise"], CLOSED),
                  {"id": "A-2", "amount_paise": 125000, "delivery_slot": "18:00-20:00"}),
    "v1 open": (schema({"id": ID, "amount_paise": AMOUNT}, ["id", "amount_paise"]),
                {"id": "A-1", "amount_paise": 125000}),
    "v2 open": (schema({"id": ID, "amount_paise": AMOUNT, "delivery_slot": SLOT}, ["id", "amount_paise"]),
                {"id": "A-2", "amount_paise": 125000, "delivery_slot": "18:00-20:00"}),
    "v3 open": (schema({"id": ID, "amount_paise": AMOUNT, "delivery_slot": SLOT, "currency": CURRENCY}, ["id", "amount_paise", "currency"]),
                {"id": "A-3", "amount_paise": 125000, "currency": "INR"}),
    "v4 open": (schema({"id": ID, "amount_paise": {"type": "string"}, "delivery_slot": SLOT, "currency": CURRENCY}, ["id", "amount_paise"]),
                {"id": "A-4", "amount_paise": "125000", "currency": "INR"}),
}

TYPES = {"string": str, "integer": int, "object": dict}


def errors(sch, value, path=""):
    """The JSON Schema 2020-12 keywords these schemas use: type, enum, properties, required, additionalProperties."""
    if "type" in sch and not (isinstance(value, TYPES[sch["type"]]) and not isinstance(value, bool)):
        return [f"{path}: not {'an' if sch['type'][0] in 'aeiou' else 'a'} {sch['type']}"]
    if "enum" in sch and value not in sch["enum"]:
        return [f"{path}: not one of {sch['enum']}"]
    found = []
    if isinstance(value, dict):
        found += [f"{path}/{name}: required" for name in sch.get("required", []) if name not in value]
        for name, item in value.items():
            if name in sch.get("properties", {}):
                found += errors(sch["properties"][name], item, f"{path}/{name}")
            elif sch.get("additionalProperties", True) is False:
                found.append(f"{path}/{name}: not allowed")
    return found


def verdict(backward, forward):
    return {(True, True): "full", (True, False): "backward", (False, True): "forward"}.get((backward, forward), "breaking")


CHANGES = [
    ("v1 closed", "v2 closed", "add optional delivery_slot, closed model"),
    ("v1 open", "v2 open", "add optional delivery_slot, open model"),
    ("v2 open", "v3 open", "add required currency"),
    ("v3 open", "v4 open", "make currency optional, amount_paise a string"),
]
for old, new, what in CHANGES:
    (old_schema, old_data), (new_schema, new_data) = VERSIONS[old], VERSIONS[new]
    back, fwd = errors(new_schema, old_data), errors(old_schema, new_data)
    print(f"{old} -> {new}: {what}")
    print(f"  new schema, old data: {'valid' if not back else '; '.join(back)}")
    print(f"  old schema, new data: {'valid' if not fwd else '; '.join(fwd)}")
    print(f"  compatible: {verdict(not back, not fwd)}")

v1 closed -> v2 closed: add optional delivery_slot, closed model
  new schema, old data: valid
  old schema, new data: /delivery_slot: not allowed
  compatible: backward
v1 open -> v2 open: add optional delivery_slot, open model
  new schema, old data: valid
  old schema, new data: valid
  compatible: full
v2 open -> v3 open: add required currency
  new schema, old data: /currency: required
  old schema, new data: valid
  compatible: forward
v3 open -> v4 open: make currency optional, amount_paise a string
  new schema, old data: /amount_paise: not a string
  old schema, new data: /amount_paise: not an integer
  compatible: breaking

Each verdict follows from which of the two checks failed. Adding an optional field to a closed schema is backward compatible only, because the old schema refuses the new member. The same change to an open schema is fully compatible. Adding a required field to an open schema is forward compatible only, because the new schema refuses every old order without it. Changing a type breaks both directions. Walk through the versions in order:

Open content models let old readers skip new data

The closed model is what made the first change half-safe. With additionalProperties: false, every new member is an error for every old reader, so the schema cannot grow without breaking someone. An open content modelopen content model: A schema that allows members or elements it does not name, such as a JSON Schema without additionalProperties false or an XSD sequence ending in xs:any, so a newer writer can add data old readers skip. allows members it does not name, and its readers follow the must-ignoremust-ignore policy: The rule of RFC 7493 section 4.2 that a receiver ignores a member whose name it does not recognise instead of treating it as an error, which lets a JSON protocol grow. policy of I-JSON, RFC 7493 section 4.2: a member whose name the receiver does not recognise must be ignored, not treated as an error. Follow a version 1 reader through a version 2 order:

The same reader, written strictly and then tolerantly:

"""A version 1 consumer meets a version 2 message: a strict reader fails, a tolerant reader ignores the unknown member
and keeps it, so that a service passing the message on does not drop it."""
import json
from dataclasses import dataclass, field

message = json.loads('{"id": "A-2", "amount_paise": 125000, "delivery_slot": "18:00-20:00"}')


@dataclass
class Order:  # what the version 1 code knows
    id: str
    amount_paise: int
    extra: dict = field(default_factory=dict)


try:
    Order(**message)
except TypeError as e:
    print("strict reader:", e)


def read_order(data):
    """Must-ignore: take the members this version knows, keep the rest untouched."""
    known = {"id", "amount_paise"}
    return Order(data["id"], data["amount_paise"], {k: v for k, v in data.items() if k not in known})


order = read_order(message)
print("tolerant reader:", order)

# Passing it on: a proxy that writes back only the fields it knows would delete delivery_slot for everyone after it.
lossy = {"id": order.id, "amount_paise": order.amount_paise}
kept = {"id": order.id, "amount_paise": order.amount_paise, **order.extra}
print("forwarded without extra:", json.dumps(lossy))
print("forwarded with extra:   ", json.dumps(kept))

strict reader: Order.__init__() got an unexpected keyword argument 'delivery_slot'
tolerant reader: Order(id='A-2', amount_paise=125000, extra={'delivery_slot': '18:00-20:00'})
forwarded without extra: {"id": "A-2", "amount_paise": 125000}
forwarded with extra:    {"id": "A-2", "amount_paise": 125000, "delivery_slot": "18:00-20:00"}

The strict reader failed with a TypeError on a member it had never heard of. The tolerant one kept the member aside, and the last two lines show why keeping matters: a service in the middle that writes back only the fields it knows deletes delivery_slot for every service after it.

XML Schema builds the same openness into the schema. A wildcard, xs:any, at the end of a sequence lets elements of other namespaces follow; with processContents="lax" the validator checks such an element when it knows a declaration for it and lets it pass when it does not. Here are a closed and an open version 1 schema and two version 2 orders, one that puts its new element in a namespace of its own and one that does not:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Version 1 of the order, closed: nothing may follow amountPaise. -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="urn:example:order"
           xmlns="urn:example:order" elementFormDefault="qualified">
  <xs:element name="order">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="id" type="xs:string"/>
        <xs:element name="amountPaise" type="xs:integer"/>
      </xs:sequence>
      <xs:attribute name="version" type="xs:string" use="required"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

<?xml version="1.0" encoding="UTF-8"?>
<!-- Version 1 of the order, open: elements and attributes of other namespaces may follow, checked if they are known. -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="urn:example:order"
           xmlns="urn:example:order" elementFormDefault="qualified">
  <xs:element name="order">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="id" type="xs:string"/>
        <xs:element name="amountPaise" type="xs:integer"/>
        <xs:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="version" type="xs:string" use="required"/>
      <xs:anyAttribute namespace="##other" processContents="lax"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

<order xmlns="urn:example:order" xmlns:v2="urn:example:order:v2" version="2">
  <id>A-2</id>
  <amountPaise>125000</amountPaise>
  <v2:deliverySlot>18:00-20:00</v2:deliverySlot>
</order>

<order xmlns="urn:example:order" version="2">
  <id>A-2</id>
  <amountPaise>125000</amountPaise>
  <deliverySlot>18:00-20:00</deliverySlot>
</order>

import java.io.File;
import java.util.List;
import javax.xml.XMLConstants;
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Validator;
import org.xml.sax.SAXParseException;

/** Two version 2 orders against the closed and the open version 1 schema, with the JDK's XSD validator. */
public class Evolve {
  public static void main(String[] args) throws Exception {
    SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
    factory.setProperty(XMLConstants.ACCESS_EXTERNAL_DTD, "");
    factory.setProperty(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
    for (String xsd : List.of("order-closed.xsd", "order-open.xsd")) {
      Validator validator = factory.newSchema(new File(xsd)).newValidator();
      validator.setProperty(XMLConstants.ACCESS_EXTERNAL_DTD, "");
      for (String xml : List.of("order-v2.xml", "order-v2-same-ns.xml")) {
        try {
          validator.validate(new StreamSource(new File(xml)));
          System.out.println(xsd + " + " + xml + ": valid");
        } catch (SAXParseException e) {
          System.out.println(xsd + " + " + xml + ": line " + e.getLineNumber() + ": " + e.getMessage());
        }
      }
    }
  }
}

order-closed.xsd + order-v2.xml: line 4: cvc-complex-type.2.4.d: Invalid content was found starting with element 'v2:deliverySlot'. No child element is expected at this point.
order-closed.xsd + order-v2-same-ns.xml: line 4: cvc-complex-type.2.4.d: Invalid content was found starting with element 'deliverySlot'. No child element is expected at this point.
order-open.xsd + order-v2.xml: valid
order-open.xsd + order-v2-same-ns.xml: line 4: cvc-complex-type.2.4.a: Invalid content was found starting with element '{"urn:example:order":deliverySlot}'. One of '{WC[##other:"urn:example:order"]}' is expected.

The closed schema refused both orders. The open one accepted the order whose new element sits in its own namespace, urn:example:order:v2, and refused the other, because ##other admits only elements in a namespace other than the schema’s target namespace. So an XML format that means to grow puts its extensions in new namespaces. XSD 1.1 adds xs:openContent, which can admit wildcard elements anywhere in a content model, but the JDK’s validator implements XSD 1.0 only, as the XML Schema lesson’s run showed.

Every format writes the same rules in its own terms

Binary formats state the rules most explicitly, because their data carries less of its own description. Protocol Buffers identify fields by number, not by name: changing a field’s number is not safe, adding fields is, and a removed field’s number must be reserved so it is never reused; proto3 parsers keep fields they do not know, and a missing field reads as its type’s default, zero, an empty string or false. Avro writes no field names or tags at all, so a reader needs the writer’s schema, and schema resolution matches the two schemas: what the writer wrote and the reader does not know is skipped, a field the reader expects but the writer never wrote gets the reader’s default, and with no default the read stops with an error. The system design track’s lesson on encoding and schema evolution works through Avro and Protocol Buffers in depth.

Text formats need the same discipline from their readers. A CSV file has no schema at all, only a header, and the classic break is a column inserted in the middle:

id,city,paise
A-1,Pune,125000
A-2,Kochi,4500

id,channel,city,paise
A-1,app,Pune,125000
A-2,web,Kochi,4500

"""Version 2 of an export inserts a column, channel, before city. A reader that uses positions reads the wrong
columns; a reader that uses the header's names reads both versions, and says which required column is missing."""
import csv

REQUIRED = ["id", "city", "paise"]


def by_position(path):
    with open(path, encoding="utf-8", newline="") as f:
        rows = csv.reader(f)
        next(rows)  # skip the header
        return [{"id": r[0], "city": r[1], "paise": r[2]} for r in rows]


def by_name(path):
    with open(path, encoding="utf-8", newline="") as f:
        rows = csv.DictReader(f)
        missing = [name for name in REQUIRED if name not in rows.fieldnames]
        if missing:
            raise ValueError(f"{path}: no column {', '.join(missing)} (columns: {', '.join(rows.fieldnames)})")
        return [{name: r[name] for name in REQUIRED} for r in rows]


for path in ["orders-v1.csv", "orders-v2.csv"]:
    print(f"{path} by position: {by_position(path)[0]}")
    print(f"{path} by name:     {by_name(path)[0]}")

with open("orders-v3.csv", "w", encoding="utf-8", newline="") as f:
    f.write("id,town,paise\r\nA-1,Pune,125000\r\n")  # a later export renamed city to town
try:
    by_name("orders-v3.csv")
except ValueError as e:
    print(e)

orders-v1.csv by position: {'id': 'A-1', 'city': 'Pune', 'paise': '125000'}
orders-v1.csv by name:     {'id': 'A-1', 'city': 'Pune', 'paise': '125000'}
orders-v2.csv by position: {'id': 'A-1', 'city': 'app', 'paise': 'Pune'}
orders-v2.csv by name:     {'id': 'A-1', 'city': 'Pune', 'paise': '125000'}
orders-v3.csv: no column city (columns: id, town, paise)

The positional reader took app as the city and Pune as the amount, without an error. The reader that uses the header’s names read both versions correctly, and when a later export renamed city to town, it said exactly which column was missing.

Configuration files evolve by version. A version key says which rules the file follows, and a migration for each step turns one version into the next. People edit these files, so a migration that rewrites the text from parsed values loses their comments; editing the text keeps them:

# Settings of the price checker for one shop
version = 1

# Seconds before a request to the shop is abandoned
timeout = 2.5
retries = 3   # tries after the first one

"""A settings file migrated from version 1 to version 3, one step at a time, by editing its text so that the comments
stay. tomllib reads every version back to check that the values are what the step promised."""
import re
import tomllib


def line_of(key):
    return re.compile(rf"^{key}\s*=\s*[^#\n]*?(\s*#.*)?\n", re.M)


def set_line(text, key, new_line):
    """Replace the line that assigns key, keeping a comment at its end."""
    if not line_of(key).search(text):
        raise ValueError(f"no line sets {key}")
    return line_of(key).sub(lambda m: new_line + (m.group(1) or "") + "\n", text, count=1)


def remove_line(text, key):
    """Remove the line that assigns key, and return its comment so that it can move with the value."""
    m = line_of(key).search(text)
    if not m:
        raise ValueError(f"no line sets {key}")
    return text[: m.start()] + text[m.end() :], (m.group(1) or "")


def v1_to_v2(text, data):
    """timeout in seconds becomes timeout_ms, a whole number of milliseconds; its comment must say so too."""
    text = set_line(text, "timeout", f"timeout_ms = {round(data['timeout'] * 1000)}")
    text = text.replace("# Seconds before a request", "# Milliseconds before a request")
    return set_line(text, "version", "version = 2")


def v2_to_v3(text, data):
    """retries moves into a [retry] table, with a backoff that the old file did not have."""
    text, comment = remove_line(text, "retries")
    text = set_line(text, "version", "version = 3")
    return text.rstrip("\n") + f"\n\n[retry]\nattempts = {data['retries']}{comment}\nbackoff_ms = 200\n"


MIGRATIONS = {1: v1_to_v2, 2: v2_to_v3}


def migrate(text, target=3):
    data = tomllib.loads(text)
    while data["version"] < target:
        text = MIGRATIONS[data["version"]](text, data)
        data = tomllib.loads(text)  # every step must leave a valid file of the next version
    return text, data


with open("checker-v1.toml", encoding="utf-8") as f:
    original = f.read()

text, data = migrate(original)
print(text, end="")
print("---")
print("read back:", data)
print("migrating again changes nothing:", migrate(text)[0] == text)

# Settings of the price checker for one shop
version = 3

# Milliseconds before a request to the shop is abandoned
timeout_ms = 2500

[retry]
attempts = 3   # tries after the first one
backoff_ms = 200
---
read back: {'version': 3, 'timeout_ms': 2500, 'retry': {'attempts': 3, 'backoff_ms': 200}}
migrating again changes nothing: True

Each step was checked by reading the result back, the comment that said seconds was changed to say milliseconds, and running the migration again changed nothing. For real files, libraries keep comments while editing parsed data: tomlkit for TOML, ruamel.yaml in Python and the Document API of the yaml package in JavaScript for YAML, whose documentation warns that its handling of trailing comments is not yet fully stable.

A breaking change ships in steps: readers first, then writers

The direction of a change decides who upgrades first. For a backward-compatible change, upgrade every consumer before any producer writes the new form, because only the new readers handle both. For a forward-compatible change, upgrade the producers first, and make sure the old data is gone before the consumers move. A fully compatible change can go in either order. A change that is compatible in neither direction is split until each step is, which is the pattern called expand and contractexpand and contract: Shipping a breaking change as compatible steps: readers first learn the new form beside the old one, writers then switch, and the old form is removed only when no data of it remains.. Here is the required currency of version 3 done that way:

Each step is safe on its own. Expanding the readers is fully compatible, because an optional field with a default accepts old and new orders. Sending the field changes nothing for readers that already know it. Contracting, making the field required, is safe only once no order without it remains anywhere: in queues, in storage, in backups that might be replayed. Two habits make this routine: a version in the data itself, a key or an attribute such as version="2", so that a reader can tell which rules a document follows, and schemas kept in one place with a compatibility check that runs before any change is merged, as the exercise below builds.

Practise: classify a schema change

The exercise turns the two checks of this lesson into a function: given an old and a new schema in a small schema language, decide whether the change is full, backward, forward or breaking. The sample tests cover added, removed and changed fields, and schemas opened and closed:

{"open": true, "fields": {"id": {"type": "string", "required": true}, "slot": {"type": "string"}}}

def compatibility(old, new):
    """"full", "backward", "forward" or "breaking". This first try only compares the names of the fields, so it calls
    every change breaking and every unchanged list of names full."""
    return "full" if set(old["fields"]) == set(new["fields"]) else "breaking"

from compat import compatibility


def schema(open_, **fields):
    return {"open": open_, "fields": fields}


S = {"type": "string"}
R = {"type": "string", "required": True}
INT = {"type": "integer", "required": True}


def test_same_schema():
    """an unchanged schema is fully compatible, open or closed"""
    assert compatibility(schema(True, id=R), schema(True, id=R)) == "full"
    assert compatibility(schema(False, id=R), schema(False, id=R)) == "full"


def test_add_optional_field():
    """adding an optional field: full when open, only backward when closed"""
    assert compatibility(schema(True, id=R), schema(True, id=R, slot=S)) == "full"
    assert compatibility(schema(False, id=R), schema(False, id=R, slot=S)) == "backward"


def test_add_required_field():
    """adding a required field breaks old data for new readers"""
    assert compatibility(schema(True, id=R), schema(True, id=R, currency=R)) == "forward"
    assert compatibility(schema(False, id=R), schema(False, id=R, currency=R)) == "breaking"


def test_remove_fields():
    """removing an optional field, or a required one"""
    assert compatibility(schema(True, id=R, slot=S), schema(True, id=R)) == "full"
    assert compatibility(schema(True, id=R, amount=INT), schema(True, id=R)) == "backward"


def test_change_requirement_or_type():
    """making a field required or optional, and changing its type"""
    assert compatibility(schema(True, id=R, slot=S), schema(True, id=R, slot=R)) == "forward"
    assert compatibility(schema(True, id=R, slot=R), schema(True, id=R, slot=S)) == "backward"
    assert compatibility(schema(True, id=R, amount=INT), schema(True, id=R, amount=R)) == "breaking"


def test_open_and_close():
    """opening a closed schema, and closing an open one"""
    assert compatibility(schema(False, id=R), schema(True, id=R)) == "backward"
    assert compatibility(schema(True, id=R), schema(False, id=R)) == "forward"

/** "full", "backward", "forward" or "breaking". This first try only compares the names of the fields, so it calls
 * every change breaking and every unchanged list of names full. */
export function compatibility(oldSchema, newSchema) {
  const a = Object.keys(oldSchema.fields).sort().join();
  const b = Object.keys(newSchema.fields).sort().join();
  return a === b ? 'full' : 'breaking';
}

import { test, assert } from 'mysmartcopilot:test';
import { compatibility } from './compat.mjs';

const schema = (open, fields) => ({ open, fields });
const S = { type: 'string' };
const R = { type: 'string', required: true };
const INT = { type: 'integer', required: true };

test('an unchanged schema is fully compatible, open or closed', () => {
  assert.equal(compatibility(schema(true, { id: R }), schema(true, { id: R })), 'full');
  assert.equal(compatibility(schema(false, { id: R }), schema(false, { id: R })), 'full');
});

test('adding an optional field: full when open, only backward when closed', () => {
  assert.equal(compatibility(schema(true, { id: R }), schema(true, { id: R, slot: S })), 'full');
  assert.equal(compatibility(schema(false, { id: R }), schema(false, { id: R, slot: S })), 'backward');
});

test('adding a required field breaks old data for new readers', () => {
  assert.equal(compatibility(schema(true, { id: R }), schema(true, { id: R, currency: R })), 'forward');
  assert.equal(compatibility(schema(false, { id: R }), schema(false, { id: R, currency: R })), 'breaking');
});

test('removing an optional field, or a required one', () => {
  assert.equal(compatibility(schema(true, { id: R, slot: S }), schema(true, { id: R })), 'full');
  assert.equal(compatibility(schema(true, { id: R, amount: INT }), schema(true, { id: R })), 'backward');
});

test('making a field required or optional, and changing its type', () => {
  assert.equal(compatibility(schema(true, { id: R, slot: S }), schema(true, { id: R, slot: R })), 'forward');
  assert.equal(compatibility(schema(true, { id: R, slot: R }), schema(true, { id: R, slot: S })), 'backward');
  assert.equal(compatibility(schema(true, { id: R, amount: INT }), schema(true, { id: R, amount: R })), 'breaking');
});

test('opening a closed schema, and closing an open one', () => {
  assert.equal(compatibility(schema(false, { id: R }), schema(true, { id: R })), 'backward');
  assert.equal(compatibility(schema(true, { id: R }), schema(false, { id: R })), 'forward');
});

Then try the four ideas on changes you have not seen yet: directions, open models, each format’s rules and rollouts. Every choice says why it is right or what it misses:

Interview questions

Warm-up (fresher to 5 years): what do backward and forward compatibility mean for a schema change?

A change is backward compatible when readers using the new schema can read data written with the old one, and forward compatible when readers still on the old schema can read data written with the new one; fully compatible means both. Adding an optional field to an open JSON schema is fully compatible. Adding a required one is only forward compatible: old readers ignore it, but new readers refuse old data that lacks it. Adding an optional field to a closed schema, one with additionalProperties: false, is only backward compatible, because old readers refuse the new member. Changing a field’s type breaks both directions. The direction also sets the rollout order: consumers first for a backward-compatible change, producers first for a forward-compatible one.

Two design questions follow, about a required field for twelve consumers and about a format that closed every schema. Classify each change yourself before you open the first steps:

Key takeaways

• A change is backward compatible when new readers read old data and forward compatible when old readers read new data; check both, as the matrix did for each of the four changes.
• Open content models and readers that follow I-JSON’s must-ignore policy let formats grow: additionalProperties: false turned an optional field into a break, and XML extensions belong in namespaces of their own.
• Every format has the same rules in its own terms: field numbers in Protocol Buffers, defaults in Avro, header names in CSV, version keys and migrations that keep comments in configuration files.
• A breaking change ships as expand and contract: readers learn the new form first, writers follow, and the old form is removed only when no data of it remains.