JSON interop: duplicates, key order, I-JSON

{"status": "paid", "status": "refunded"} is valid JSON, and both JSON.parse and Python’s json.loads read it as refunded without a word. A check that reads the first value sees a paid order, the service that reads the last one processes a refund, and nothing in either program fails.

You will detect duplicate names that standard parsers accept, with the JSON Pointer of each, see why JavaScript changes the order of some keys and when order matters, keep a parsed document from changing every object of a JavaScript program through its prototype, and apply the I-JSON rules of RFC 7493, which make a message read the same way by every receiver.

Duplicate names: both parsers keep the last value, without a word

RFC 8259 says that the names within an object SHOULD be unique, and calls what receivers do with duplicates “unpredictable”. It also says what parsers do: “Many implementations report the last name/value pair only”; others report an error, and some report every pair. Here is an order with three repeated names, one of them hidden behind an escape:

"""Find duplicate member names, with the JSON Pointer of the object that holds them."""
import json


class Pairs(list):
    """An object kept as its list of (name, value) pairs, so repeated names stay visible."""


def pointer(path):
    """An RFC 6901 JSON Pointer: each token with ~ written ~0 and / written ~1."""
    return "".join("/" + str(t).replace("~", "~0").replace("/", "~1") for t in path)


def duplicates(value, path=()):
    """Every repeated name, as (pointer of its object, name)."""
    found = []
    if isinstance(value, Pairs):
        seen = set()
        for name, item in value:
            if name in seen:
                found.append((pointer(path) or "(root)", name))
            seen.add(name)
            found += duplicates(item, path + (name,))
    elif isinstance(value, list):
        for i, item in enumerate(value):
            found += duplicates(item, path + (i,))
    return found


text = '{"id": "BK-1", "status": "paid", "items": [{"sku": "A", "qty": 1, "qty": 5}], "status": "refunded", "a/b": {"x": 1, "\\u0078": 2}}'
print("json.loads keeps:", json.loads(text)["status"], json.loads(text)["items"][0]["qty"])
for where, name in sorted(duplicates(json.loads(text, object_pairs_hook=Pairs)), key=lambda d: d[1]):
    print(f"duplicate {name!r} in {where}")

json.loads keeps: refunded 5
duplicate 'qty' in /items/0
duplicate 'status' in (root)
duplicate 'x' in /a~1b

// JSON.parse keeps the last of two equal names and offers no hook to notice, so this scanner reads the text itself
// and reports each repeated name with the JSON Pointer of its object. Names are compared after unescaping.
function duplicateNames(text) {
  const found = [];
  let i = 0;
  const skipSpace = () => { while (' \t\n\r'.includes(text[i]) && i < text.length) i++; };
  const readString = () => { // returns the unescaped string; JSON.parse does the unescaping
    const start = i++;
    while (text[i] !== '"') i += text[i] === '\\' ? 2 : 1;
    i++;
    return JSON.parse(text.slice(start, i));
  };
  const pointer = (path) => path.map((t) => '/' + String(t).replaceAll('~', '~0').replaceAll('/', '~1')).join('') || '(root)';
  const readValue = (path) => {
    skipSpace();
    if (text[i] === '{') {
      i++;
      const seen = new Set();
      for (skipSpace(); text[i] !== '}'; skipSpace()) {
        if (text[i] === ',') { i++; skipSpace(); }
        const name = readString();
        if (seen.has(name)) found.push(`duplicate ${JSON.stringify(name)} in ${pointer(path)}`);
        seen.add(name);
        skipSpace(); i++; // the colon
        readValue([...path, name]);
      }
      i++;
    } else if (text[i] === '[') {
      i++;
      for (let n = 0; (skipSpace(), text[i] !== ']'); n++) {
        if (text[i] === ',') i++;
        readValue([...path, n]);
        skipSpace();
      }
      i++;
    } else if (text[i] === '"') readString();
    else while (i < text.length && !',]} \t\n\r'.includes(text[i])) i++; // a number, true, false or null
  };
  JSON.parse(text); // refuse invalid JSON first, so the scanner only ever sees valid text
  readValue([]);
  return found.sort();
}

const text = '{"id": "BK-1", "status": "paid", "items": [{"sku": "A", "qty": 1, "qty": 5}], "status": "refunded", "a/b": {"x": 1, "\\u0078": 2}}';
console.log('JSON.parse keeps:', JSON.parse(text).status, JSON.parse(text).items[0].qty);
for (const line of duplicateNames(text)) console.log(line);

JSON.parse keeps: refunded 5
duplicate "qty" in /items/0
duplicate "status" in (root)
duplicate "x" in /a~1b

Both parsers kept refunded and 5, the last values, and neither said anything; Python’s documentation states that it “ignores all but the last name-value pair”, and ECMAScript’s says that earlier values for the same key “shall be overwritten”. The detectors found what the parsers hid. Python’s object_pairs_hook hands over each object as its list of pairs, before any name is dropped; JavaScript has no such hook, so the second program scans the text itself, after JSON.parse has confirmed that the text is valid. Both compare names after unescaping, which is why "x" and "\u0078" count as the same name, and both report the place as a JSON PointerJSON Pointer: A string that names one value inside a JSON document, such as /items/0/qty, defined by RFC 6901. Inside a name, ~ is written ~0 and / is written ~1.: /a~1b is the member a/b, with / written ~1, as RFC 6901 asks.

The danger is two readers that resolve a duplicate nameduplicate name: A name that appears more than once in the same JSON object. The grammar allows it, RFC 8259 says names SHOULD be unique, and parsers differ in which value they keep. differently. Step through one request read by a check that stops at the first value and by a service that keeps the last:

Member order: JavaScript lists integer-like keys first

In JSON’s data model an object is an unordered collection of members, and RFC 8259 notes that implementations whose behaviour does not depend on member order will be interoperable. Parsers still hand members over in some order, and the two languages differ:

// The order of members after JSON.parse: integer-like names first, in numeric order, then the rest as written.
const text = '{"b": 1, "2": 2, "1": 3, "a": 4, "10": 5}';
const obj = JSON.parse(text);
console.log('Object.keys:   ', Object.keys(obj).join(', '));
console.log('round trip:    ', JSON.stringify(obj));

// When the order is data, send it as an array: arrays keep their order in every parser.
const ordered = new Map(JSON.parse('[["b", 1], ["2", 2], ["1", 3], ["a", 4], ["10", 5]]'));
console.log('array of pairs:', [...ordered.keys()].join(', '));

Object.keys:    1, 2, 10, b, a
round trip:     {"1":3,"2":2,"10":5,"b":1,"a":4}
array of pairs: b, 2, 1, a, 10

"""The same document in Python: a dict keeps the order in which the names appear in the text."""
import json

text = '{"b": 1, "2": 2, "1": 3, "a": 4, "10": 5}'
obj = json.loads(text)
print("keys:          ", ", ".join(obj))
print("round trip:    ", json.dumps(obj, separators=(",", ":")))

keys:           b, 2, 1, a, 10
round trip:     {"b":1,"2":2,"1":3,"a":4,"10":5}

Python kept the order of the text, b, 2, 1, a, 10, because a dict remembers the order of insertion. JavaScript listed 1, 2, 10 first, in numeric order, and then b and a as written, and JSON.stringify wrote that new order back. The rule is in the ECMAScript specification: an object’s own keys come out with every array index first, in ascending numeric order, then the other strings in the order they were created. So names such as "2" and "10" move, and "b" and "a" do not.

Order matters in three places: when bytes are compared or signed, as in a hash or a signature over the text; when people read diffs; and when the order is itself data, such as the ranking of search results. For the first, canonical JSON (RFC 8785, a later lesson of this module) defines one order; for the last, send an array, which keeps its order in every parser, as the last line of the JavaScript output shows.

A prototype key is harmless in JSON.parse and dangerous in a merge

Every ordinary JavaScript object inherits from Object.prototype, and the name __proto__ has a setter there that changes an object’s prototype. JSON.parse does not run that setter: during parsing a "__proto__" member is created as an ordinary own property, as the specification requires. The danger starts when code copies or merges the parsed document. This program takes a settings document with a "__proto__" member and merges it four ways, then checks whether a brand-new empty object has gained a property:

// A settings document with a "__proto__" member, merged into defaults four ways. The check after each merge asks
// whether a brand-new, empty object has gained a property: if it has, Object.prototype itself was changed.
const body = '{"theme": "dark", "__proto__": {"isAdmin": true}}';
const parsed = JSON.parse(body);
const check = (label) => {
  console.log(`${label.padEnd(18)} ({}).isAdmin is ${({}).isAdmin}`);
  delete Object.prototype.isAdmin; // undo any change, so each merge starts clean
};

console.log('JSON.parse made own keys:', Object.keys(parsed).join(', '));
check('after JSON.parse');

const assigned = Object.assign({}, parsed); // [[Set]] of "__proto__" runs the setter: the copy's prototype changes
console.log('Object.assign copy: isAdmin', assigned.isAdmin, '| own keys:', Object.keys(assigned).join(', '));
check('after assign');

function naiveMerge(target, source) {
  for (const key in source) {
    if (typeof source[key] === 'object' && source[key] !== null) naiveMerge((target[key] ??= {}), source[key]);
    else target[key] = source[key];
  }
  return target;
}
naiveMerge({ theme: 'light' }, parsed); // target["__proto__"] is Object.prototype itself
check('after naive merge');

function safeMerge(target, source) {
  for (const key of Object.keys(source)) {
    if (key === '__proto__' || key === 'constructor' || key === 'prototype') continue; // never merge these names
    const value = source[key];
    if (typeof value === 'object' && value !== null && !Array.isArray(value)) {
      const inner = Object.hasOwn(target, key) ? target[key] : Object.create(null);
      target[key] = safeMerge(inner, value);
    } else target[key] = value;
  }
  return target;
}
const settings = safeMerge(Object.assign(Object.create(null), { theme: 'light' }), parsed);
console.log('safe merge result:', JSON.stringify(settings));
check('after safe merge');

JSON.parse made own keys: theme, __proto__
after JSON.parse   ({}).isAdmin is undefined
Object.assign copy: isAdmin true | own keys: theme
after assign       ({}).isAdmin is undefined
after naive merge  ({}).isAdmin is true
safe merge result: {"theme":"dark"}
after safe merge   ({}).isAdmin is undefined

JSON.parse kept __proto__ as an own key, and nothing else changed. Object.assign copies with ordinary assignment, which runs the setter, so the copy’s prototype became {"isAdmin": true}: the copy reports isAdmin without having it as its own key. The naive deep merge is the serious case. It reached target["__proto__"], which is Object.prototype itself, and merged isAdmin into it, so every object in the program, including an empty one, now reports isAdmin: true. This weakness is called prototype pollutionprototype pollution: A weakness of JavaScript programs in which input data adds or changes properties of a shared prototype such as Object.prototype, so every object seems to have them..

The safe merge stops all of it with three habits, the second of which OWASP’s prevention cheat sheet recommends:

A Map is a fourth way: its keys are data, never properties, so a key named __proto__ is just a string. And a schema check of the parsed settings, allowing only the names a settings object can have, refuses the document before any merge.

I-JSON: the rules that make a message safe for every receiver

RFC 7493, a Proposed Standard, defines I-JSONI-JSON: The Internet JSON profile of RFC 7493: JSON in UTF-8, without duplicate names or lone surrogates, with numbers within binary64 and big integers sent as strings., the Internet JSON profile: JSON with the choices made that RFC 8259 leaves open, so that every conforming receiver reads a message the same way. Its rules, with the lessons of this module that show why each one exists:

The rules that a program can check mechanically fit in one function. This one checks six messages:

"""Six messages checked against the I-JSON rules of RFC 7493 that a program can test."""
import json

MAX_EXACT = 2**53 - 1  # 9007199254740991, the largest integer every binary64 receiver holds exactly


def bad_text(s):
    """A surrogate (a lone half) or a noncharacter, which I-JSON strings must not contain."""
    for c in s:
        cp = ord(c)
        if 0xD800 <= cp <= 0xDFFF:
            return f"lone surrogate U+{cp:04X}"
        if 0xFDD0 <= cp <= 0xFDEF or cp & 0xFFFE == 0xFFFE:
            return f"noncharacter U+{cp:04X}"
    return None


def problems(data):
    """The I-JSON rules a message breaks, in the order they are checked."""
    try:
        text = data.decode("utf-8")
    except UnicodeDecodeError:
        return ["not UTF-8"]
    found = []

    def pairs(items):
        names = [name for name, _ in items]
        found.extend(f"duplicate name {n!r}" for n in sorted({n for n in names if names.count(n) > 1}))
        found.extend(p for p in map(bad_text, names) if p)
        return dict(items)

    value = json.loads(text, object_pairs_hook=pairs)
    stack = [value]
    while stack:
        v = stack.pop()
        if isinstance(v, dict):
            stack.extend(v.values())
        elif isinstance(v, list):
            stack.extend(v)
        elif isinstance(v, bool):
            continue
        elif isinstance(v, int) and abs(v) > MAX_EXACT:
            found.append(f"integer {v} beyond 2^53 - 1")
        elif isinstance(v, float) and abs(v) == float("inf"):
            found.append("number beyond binary64")
        elif isinstance(v, str) and bad_text(v):
            found.append(bad_text(v))
    if not isinstance(value, (dict, list)):
        found.append("top level is not an object or an array (a SHOULD)")
    return found


messages = {
    "id as a string": b'{"id": "12345678901234567890", "qty": 2}',
    "id as a number": b'{"id": 12345678901234567890, "qty": 2}',
    "qty twice": b'{"qty": 1, "qty": 2}',
    "half a surrogate pair": b'{"name": "\\ud800"}',
    "UTF-16 with a BOM": '{"name": "Pune"}'.encode("utf-16"),
    "a string on its own": b'"just a string"',
}
for label, data in messages.items():
    print(f"{label:<22} {'; '.join(problems(data)) or 'I-JSON'}")

id as a string         I-JSON
id as a number         integer 12345678901234567890 beyond 2^53 - 1
qty twice              duplicate name 'qty'
half a surrogate pair  lone surrogate U+D800
UTF-16 with a BOM      not UTF-8
a string on its own    top level is not an object or an array (a SHOULD)

Only the first message passes, the one with its 20-digit id as a string. The second carries the same id as a number, beyond 2^53 − 1; the third repeats qty; the fourth holds half a surrogate pair; the fifth is UTF-16; the sixth is a bare string at the top level, which I-JSON allows but advises against. Running such a check where messages enter a system, and refusing what fails, is a cheap way to make every program behind it agree.

Practise: an I-JSON checker with pointers

The exercise combines the duplicate finder and the I-JSON check: it reports every duplicate name, big integer and lone surrogate of a message with the JSON Pointer of the member it is about, including names that collide only after unescaping and pointers whose names hold ~ or /:

import json


class Pairs(list):
    """An object kept as its list of (name, value) pairs."""


def check_i_json(data):
    """The I-JSON problems of a message, sorted, each with its JSON Pointer."""
    problems = []
    return sorted(problems)

from ijson import check_i_json


def test_clean_message():
    """finds nothing in a message that follows the rules"""
    assert check_i_json(b'{"id": "12345678901234567890", "qty": 2, "flag": true, "big": 9007199254740991}') == []


def test_duplicates():
    """reports repeated names at the pointer of the member, nested ones too"""
    assert check_i_json(b'{"qty": 1, "qty": 2}') == ["duplicate /qty"]
    assert check_i_json(b'{"items": [{"sku": "A", "sku": "B"}]}') == ["duplicate /items/0/sku"]


def test_names_compared_after_unescaping():
    """treats a name and its escaped form as the same name"""
    assert check_i_json(b'{"a": 1, "\\u0061": 2}') == ["duplicate /a"]


def test_pointer_escapes():
    """writes ~ as ~0 and / as ~1 in pointers"""
    assert check_i_json(b'{"a/b": {"m~n": 1, "m~n": 2}}') == ["duplicate /a~1b/m~0n"]


def test_big_integers():
    """reports integers beyond 2^53 - 1 in either direction"""
    assert check_i_json(b'{"id": 9007199254740993, "n": [-9007199254740992, 7]}') == ["big integer /id", "big integer /n/0"]


def test_lone_surrogates():
    """reports a lone surrogate, but not a surrogate pair"""
    assert check_i_json(b'{"name": "\\ud800", "ok": "\\ud83d\\udcda"}') == ["lone surrogate /name"]


def test_not_utf8():
    """stops at bytes that are not UTF-8"""
    assert check_i_json(b'{"a": "\xff"}') == ["not UTF-8"]

Then take the questions on duplicates, member order, merges and I-JSON. Each one shows why an answer holds, and a slip sends you back to that idea first:

Interview questions

Warm-up (fresher to 2–5 years): what does a JSON parser do with an object that repeats a name?

The grammar allows it, so the text is valid JSON, but RFC 8259 says names SHOULD be unique and calls the behaviour of receivers unpredictable. In practice most parsers keep the last value: JSON.parse does by the ECMAScript specification, and Python’s json.loads does by its documentation, so {"qty": 1, "qty": 2} gives 2 in both, without a warning. Other parsers keep the first value, refuse the text or report every pair, which is why a check and a service can read different values from one message. To detect duplicates in Python, use object_pairs_hook, which sees every pair; in JavaScript, scan the text. I-JSON, RFC 7493, forbids duplicates, compared after unescaping.

Two harder questions follow, about a check and a service that read different values and about a settings merge that changed every user. Decide what you would check first, then open each first step:

Key takeaways

• A repeated name is valid JSON; JSON.parse and json.loads keep the last value silently, so detect duplicates with object_pairs_hook or a scanner, compare names after unescaping, and refuse them at the edge.
• JSON objects are unordered, but JavaScript lists integer-like keys first in numeric order, so never let meaning depend on member order: use arrays for ordered data and canonical forms for bytes that are compared.
• JSON.parse keeps "__proto__" as an own property, but Object.assign and naive deep merges can change prototypes; skip dangerous names, build objects with Object.create(null), copy own keys only, or use a Map.
• I-JSON requires UTF-8, no lone surrogates and no duplicate names, keeps numbers within binary64 with big integers as strings, and advises an object or an array at the top; check those rules where messages enter.