Data Formats (JSON, YAML, TOML, XML, CSV) Module 10 – Across formats: time, conversion, streaming, evolution, binary
Beyond text: CBOR, MessagePack, Protobuf, Parquet
Encode the same orders as JSON, CBOR, MessagePack, Protobuf and Avro, compare their sizes and round trips, and choose Parquet or Avro for analytics.
What you will learn
- Explain when a binary format beats JSON and what you give up
- Encode the same records as JSON, CBOR, MessagePack and Protobuf and compare sizes
- Choose Parquet or Avro for analytics and event streams
Before you start
What you’ll learn · Before you start
What you will learn
- Explain when a binary format beats JSON and what you give up
- Encode the same records as JSON, CBOR, MessagePack and Protobuf and compare sizes
- Choose Parquet or Avro for analytics and event streams
Before you start
On this page
1,000 orders took 112,364 bytes as compact JSON, 77,446 as CBOR and 27,630 as Avro. After gzip, the five formats tried here ended within 2,100 bytes of each other, from 11,200 to 13,270.
So a binary format is not simply smaller JSON: what it saves depends on the data, and each one gives something up, a readable text, a self-describing record, or freedom from a schema. You will see how CBOR and MessagePack keep JSON’s shape in fewer bytes, why Protocol Buffers and Avro need a schema, measure all of them on one dataset, and choose between Parquet and Avro for analytics and event streams.
CBOR and MessagePack keep JSON’s shape in fewer bytes
CBOR: Concise Binary Object Representation, RFC 8949: a binary format for JSON-like data in which each item starts with a byte that gives its major type and its length. It adds byte strings and tags, and needs no schema., RFC 8949 and an Internet Standard, and MessagePack: A schema-less binary format for JSON-like data, registered as application/vnd.msgpack, whose serializers use the format with the fewest bytes for each value, such as one byte for an integer from 0 to 127., whose
media type IANA registers as application/vnd.msgpack, encode the same values JSON does: maps, arrays, strings,
numbers, booleans and null. They write them as bytes instead of text. Each item starts with a byte that says what it is
and how long it is, so no quotes, commas or brackets are needed, and numbers take their binary width instead of one
byte per digit. Both carry the names of every member, so a reader needs no schema. Both also hold what JSON cannot:
byte strings, and in CBOR tags such as tag 1 for an epoch time, in MessagePack the timestamp extension type.
Step through the nine CBOR bytes of {"a": 1, "b": [2, 3]}, a row of the RFC’s table of examples:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The map. a2 is 101 00010 in bits: major type 5, a map, and an argument of 2, its number of pairs. Small arguments sit in the low five bits of the first byte.
- The first key. 61 is major type 3, a text string, with length 1, so one byte of UTF-8 follows: 61, the letter a. JSON spent three bytes on "a"; CBOR spends two.
- Its value. 01 is major type 0, an unsigned integer, with the value 1 itself in the low five bits. Integers up to 23 take one byte.
- The second key. 61 62 is another text string of one byte, the letter b. A map in CBOR, as in JSON, writes every key in full, in every record.
- An array. 82 is major type 4, an array, of 2 items. No brackets or commas are needed: the count says where the array ends.
- The last items. 02 and 03 are the integers 2 and 3. Nine bytes in all, against the 17 of the same value written as compact JSON.
This lesson’s encoders are small enough to read, written from the two specifications. CBOR follows the RFC’s preferred serialization, the shortest form for each integer and the shortest float that keeps the value; MessagePack follows its spec’s rule to use the format with the fewest bytes, a float 32 included when it holds the value:
"""Small encoders and decoders, written for this lesson from the specifications: CBOR (RFC 8949, preferred
serialization), MessagePack (its spec.md, smallest formats), and Protocol Buffers and Avro for one order schema."""
import struct
# ---- CBOR ---------------------------------------------------------------------------------------------------------
class Tag:
def __init__(self, number, value):
self.number, self.value = number, value
def __eq__(self, other):
return isinstance(other, Tag) and (self.number, self.value) == (other.number, other.value)
def _head(major, n):
"""The initial byte and the argument: the value itself up to 23, then 1, 2, 4 or 8 more bytes."""
if n < 24:
return bytes([major << 5 | n])
for ai, size in ((24, 1), (25, 2), (26, 4), (27, 8)):
if n < 1 << (8 * size):
return bytes([major << 5 | ai]) + n.to_bytes(size, "big")
raise ValueError("integer too large for CBOR without a bignum tag")
def _cbor_float(x):
"""The shortest of half, single and double precision that keeps the value, as preferred serialization asks."""
for ai, fmt in ((25, ">e"), (26, ">f")):
try:
packed = struct.pack(fmt, x)
except OverflowError:
continue
if struct.unpack(fmt, packed)[0] == x or x != x:
return bytes([0xE0 | ai]) + packed
return b"\xfb" + struct.pack(">d", x)
def cbor_encode(v):
if v is False:
return b"\xf4"
if v is True:
return b"\xf5"
if v is None:
return b"\xf6"
if isinstance(v, int):
return _head(0, v) if v >= 0 else _head(1, -1 - v)
if isinstance(v, float):
return _cbor_float(v)
if isinstance(v, bytes):
return _head(2, len(v)) + v
if isinstance(v, str):
data = v.encode("utf-8")
return _head(3, len(data)) + data
if isinstance(v, list):
return _head(4, len(v)) + b"".join(cbor_encode(x) for x in v)
if isinstance(v, dict):
return _head(5, len(v)) + b"".join(cbor_encode(k) + cbor_encode(x) for k, x in v.items())
if isinstance(v, Tag):
return _head(6, v.number) + cbor_encode(v.value)
raise TypeError(f"CBOR encoder: {type(v).__name__}")
def cbor_decode(data):
value, end = _cbor_item(data, 0)
if end != len(data):
raise ValueError("bytes after the CBOR item")
return value
def _cbor_item(b, i):
major, ai = b[i] >> 5, b[i] & 31
i += 1
if major == 7:
if ai in (20, 21, 22):
return (False, True, None)[ai - 20], i
size, fmt = {25: (2, ">e"), 26: (4, ">f"), 27: (8, ">d")}[ai]
return struct.unpack(fmt, b[i : i + size])[0], i + size
if ai < 24:
n = ai
else:
size = {24: 1, 25: 2, 26: 4, 27: 8}[ai]
n, i = int.from_bytes(b[i : i + size], "big"), i + size
if major == 0:
return n, i
if major == 1:
return -1 - n, i
if major in (2, 3):
raw = b[i : i + n]
return (raw if major == 2 else raw.decode("utf-8")), i + n
if major == 4:
out = []
for _ in range(n):
x, i = _cbor_item(b, i)
out.append(x)
return out, i
if major == 5:
out = {}
for _ in range(n):
k, i = _cbor_item(b, i)
out[k], i = _cbor_item(b, i)
return out, i
value, i = _cbor_item(b, i)
return Tag(n, value), i
# ---- MessagePack ---------------------------------------------------------------------------------------------------
def msgpack_encode(v):
if v is None:
return b"\xc0"
if v is False:
return b"\xc2"
if v is True:
return b"\xc3"
if isinstance(v, int):
if 0 <= v < 128:
return bytes([v]) # positive fixint
if -32 <= v < 0:
return struct.pack(">b", v) # negative fixint, 0xe0 to 0xff
if v >= 0:
for code, fmt, top in ((0xCC, ">B", 1 << 8), (0xCD, ">H", 1 << 16), (0xCE, ">I", 1 << 32), (0xCF, ">Q", 1 << 64)):
if v < top:
return bytes([code]) + struct.pack(fmt, v)
for code, fmt, low in ((0xD0, ">b", -(1 << 7)), (0xD1, ">h", -(1 << 15)), (0xD2, ">i", -(1 << 31)), (0xD3, ">q", -(1 << 63))):
if v >= low:
return bytes([code]) + struct.pack(fmt, v)
raise ValueError("integer too large for MessagePack")
if isinstance(v, float):
try:
packed = struct.pack(">f", v)
if struct.unpack(">f", packed)[0] == v:
return b"\xca" + packed # float 32, when it holds the value exactly
except OverflowError:
pass
return b"\xcb" + struct.pack(">d", v) # float 64 for every other value
if isinstance(v, (str, bytes)):
data = v.encode("utf-8") if isinstance(v, str) else v
n = len(data)
if isinstance(v, str) and n < 32:
return bytes([0xA0 | n]) + data # fixstr
codes = (0xD9, 0xDA, 0xDB) if isinstance(v, str) else (0xC4, 0xC5, 0xC6)
for code, fmt, top in zip(codes, (">B", ">H", ">I"), (1 << 8, 1 << 16, 1 << 32)):
if n < top:
return bytes([code]) + struct.pack(fmt, n) + data
if isinstance(v, list):
n = len(v)
head = bytes([0x90 | n]) if n < 16 else b"\xdc" + struct.pack(">H", n) if n < 1 << 16 else b"\xdd" + struct.pack(">I", n)
return head + b"".join(msgpack_encode(x) for x in v)
if isinstance(v, dict):
n = len(v)
head = bytes([0x80 | n]) if n < 16 else b"\xde" + struct.pack(">H", n) if n < 1 << 16 else b"\xdf" + struct.pack(">I", n)
return head + b"".join(msgpack_encode(k) + msgpack_encode(x) for k, x in v.items())
raise TypeError(f"MessagePack encoder: {type(v).__name__}")
def msgpack_decode(data):
value, end = _mp_item(data, 0)
if end != len(data):
raise ValueError("bytes after the MessagePack item")
return value
def _mp_item(b, i):
c = b[i]
i += 1
if c < 0x80:
return c, i
if c >= 0xE0:
return c - 256, i
if 0xA0 <= c <= 0xBF:
n = c & 31
return b[i : i + n].decode("utf-8"), i + n
if 0x90 <= c <= 0x9F or c in (0xDC, 0xDD):
n, i = (c & 15, i) if c <= 0x9F else _mp_len(b, i, 2 if c == 0xDC else 4)
out = []
for _ in range(n):
x, i = _mp_item(b, i)
out.append(x)
return out, i
if 0x80 <= c <= 0x8F or c in (0xDE, 0xDF):
n, i = (c & 15, i) if c <= 0x8F else _mp_len(b, i, 2 if c == 0xDE else 4)
out = {}
for _ in range(n):
k, i = _mp_item(b, i)
out[k], i = _mp_item(b, i)
return out, i
simple = {0xC0: None, 0xC2: False, 0xC3: True}
if c in simple:
return simple[c], i
if c == 0xCA:
return struct.unpack(">f", b[i : i + 4])[0], i + 4
if c == 0xCB:
return struct.unpack(">d", b[i : i + 8])[0], i + 8
fixed = {0xCC: ">B", 0xCD: ">H", 0xCE: ">I", 0xCF: ">Q", 0xD0: ">b", 0xD1: ">h", 0xD2: ">i", 0xD3: ">q"}
if c in fixed:
size = struct.calcsize(fixed[c])
return struct.unpack(fixed[c], b[i : i + size])[0], i + size
sizes = {0xD9: 1, 0xDA: 2, 0xDB: 4, 0xC4: 1, 0xC5: 2, 0xC6: 4}
n, i = _mp_len(b, i, sizes[c])
raw = b[i : i + n]
return (raw.decode("utf-8") if c >= 0xD9 else raw), i + n
def _mp_len(b, i, size):
return int.from_bytes(b[i : i + size], "big"), i + size
# ---- Protocol Buffers and Avro, for one order schema -----------------------------------------------------------------
def varint(n):
out = bytearray()
while True:
low, n = n & 0x7F, n >> 7
out.append(low | (0x80 if n else 0))
if not n:
return bytes(out)
def read_varint(b, i):
n = shift = 0
while True:
byte = b[i]
n |= (byte & 0x7F) << shift
i, shift = i + 1, shift + 7
if byte < 0x80:
return n, i
def zigzag(n):
return n << 1 if n >= 0 else (-n << 1) - 1
def unzigzag(n):
return n >> 1 if not n & 1 else -((n + 1) >> 1)Before measuring anything with them, check them against the examples that RFC 8949, the Protocol Buffers encoding guide and the Avro specification publish, and against the smallest formats of the MessagePack spec:
"""tiny_codecs.py against the published examples: rows of RFC 8949 Appendix A, the first-byte formats of the
MessagePack spec, the two encodings of the Protocol Buffers encoding guide, and zig-zag values of the Avro spec."""
from tiny_codecs import Tag, cbor_decode, cbor_encode, msgpack_decode, msgpack_encode, varint, zigzag
CBOR = [ # value, encoding listed in RFC 8949 Appendix A
(0, "00"), (23, "17"), (24, "1818"), (1000, "1903e8"), (1000000, "1a000f4240"), (-1, "20"), (-100, "3863"),
(1.1, "fb3ff199999999999a"), (1.5, "f93e00"), (100000.0, "fa47c35000"), (-0.0, "f98000"),
(False, "f4"), (None, "f6"), ("IETF", "6449455446"), ("ü", "62c3bc"), ([1, 2, 3], "83010203"),
({"a": 1, "b": [2, 3]}, "a26161016162820203"), (b"\x01\x02\x03\x04", "4401020304"), (Tag(1, 1363896240), "c11a514b67b0"),
]
MSGPACK = [ # value, its smallest format in the spec: fixint, int 8, uint 8, uint 16, nil, true, fixstr, fixarray, fixmap,
# float 32 for a value it holds exactly, float 64 otherwise
(0, "00"), (127, "7f"), (-1, "ff"), (-33, "d0df"), (128, "cc80"), (1000, "cd03e8"), (None, "c0"), (True, "c3"),
("a", "a161"), ([1, 2, 3], "93010203"), ({"a": 1, "b": [2, 3]}, "82a16101a162920203"), (1.5, "ca3fc00000"),
(1.1, "cb3ff199999999999a"),
]
failed = 0
for name, encode, decode, rows, source in [
("CBOR", cbor_encode, cbor_decode, CBOR, "as RFC 8949 Appendix A lists them"),
("MessagePack", msgpack_encode, msgpack_decode, MSGPACK, "in the smallest format the spec gives"),
]:
bad = [(v, want, encode(v).hex()) for v, want in rows if encode(v).hex() != want or decode(bytes.fromhex(want)) != v]
failed += len(bad)
print(f"{name}: {len(rows) - len(bad)} of {len(rows)} encodings {source}, each decoded back to its value")
for v, want, got in bad:
print(f" {v!r}: expected {want}, got {got}")
# Protocol Buffers encoding guide: field 1 = 150 as a varint, field 2 = "testing" as a length-delimited string.
test1 = varint(1 << 3 | 0) + varint(150)
test2 = varint(2 << 3 | 2) + varint(7) + b"testing"
print("Protobuf Test1:", test1.hex(" "), "| Test2:", test2.hex(" "))
failed += test1.hex() != "089601" or test2.hex() != "120774657374696e67"
# Avro: long and int values use zig-zag coding, so 1 is 02 and -1 is 01, as the table of the Avro spec shows.
avro = " ".join(varint(zigzag(n)).hex() for n in (0, -1, 1, -2, 2, -64, 64))
print("Avro zig-zag of 0, -1, 1, -2, 2, -64, 64:", avro)
failed += avro != "00 01 02 03 04 7f 8001"
print("every example matches" if not failed else f"{failed} mismatches")Output
CBOR: 19 of 19 encodings as RFC 8949 Appendix A lists them, each decoded back to its value MessagePack: 13 of 13 encodings in the smallest format the spec gives, each decoded back to its value Protobuf Test1: 08 96 01 | Test2: 12 07 74 65 73 74 69 6e 67 Avro zig-zag of 0, -1, 1, -2, 2, -64, 64: 00 01 02 03 04 7f 8001 every example matches
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_vectors.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Every row matched, and decoded back to its value. 1.5 became f93e00, a half-precision float of three bytes, while
1.1 needed all nine of a double, because preferred serialization picks the shortest float that keeps the value
exactly. MessagePack has no half-precision float, so its 1.5 took the five bytes of a float 32, ca3fc00000.
Treating binary as automatically smaller
Wrong: switching an API from JSON to MessagePack and expecting the size to drop by half. A map still writes every key in full, in every record, so a record of short values with long keys shrinks little. Right: measure your own data, raw and compressed, as the next sections do.
Checkpoint
Check: CBOR and MessagePack
Protocol Buffers and Avro leave the names in the schema
Protocol Buffers: A binary format whose .proto schema gives each field a type and a number. Only the numbers travel, as tags before each value, so a reader needs the schema to know a field's name and meaning. and Avro: A binary format that writes only the values, in the order of the schema, with no names or tags. A reader resolves its own schema against the writer's, which container files carry in their header. take the names out of the data.
In Protocol Buffers, a .proto file gives each field a type and a number, and only the number travels: each field
starts with a tag, the number shifted left by three bits with the wire type in the low three, so field 1 holding 150 is
08 96 01. A scalar field set to its default, such as an empty note, is not written at all. Avro goes further and writes
no tags: the fields follow one another in the order of the schema, integers in zig-zag
varint: A variable-length integer: seven bits of the number in each byte, low bits first, with the top bit set on every byte but the last. Protocol Buffers and Avro use it, so small numbers take one byte., so a reader needs the writer’s schema to know where one field ends and what it means.
Here are the two schemas of the order:
Protocol Buffers · codecs/order.proto
// The order, as Protocol Buffers sees it: each field has a type and a number, and only the number travels.
syntax = "proto3";
message Order {
int64 id = 1;
string sku = 2;
int32 qty = 3;
int64 paise = 4;
bool paid = 5;
int64 placed = 6; // Unix seconds
float weight_kg = 7; // 32-bit, to save four bytes an order
string note = 8;
}
message Orders {
repeated Order orders = 1;
}AVSC · codecs/order.avsc
{
"type": "array",
"items": {
"type": "record",
"name": "Order",
"fields": [
{"name": "id", "type": "long"},
{"name": "sku", "type": "string"},
{"name": "qty", "type": "int"},
{"name": "paise", "type": "long"},
{"name": "paid", "type": "boolean"},
{"name": "placed", "type": "long"},
{"name": "weight_kg", "type": "float"},
{"name": "note", "type": "string"}
]
}
}Step through what a reader can tell from each format’s bytes:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Protocol Buffers writes numbers. Each field starts with a tag, the field number shifted left by 3 with the wire type in the low three bits. id is field 1, a varint, so its tag is 08, then d0 86 03 for 50000.
- Read without the schema. A reader without order.proto still finds every field, because the wire type says how long each one is. It sees field 1 = 50000 and field 2 = 5 bytes, but not that they are id and sku.
- A float is just four bytes. Field 7 is four bytes, 9a 99 99 3f. Only the schema says it is a 32-bit float, the weight 1.2000000, and not an integer or part of a string.
- Avro writes values only. Avro writes the fields in the schema's order with no tags at all: 26 bytes for the order and its array block, against 30 for Protocol Buffers without the array framing.
- The wrong schema. A reader whose schema lists paise before id reads the same bytes without any error: id becomes 415002 and paise 50000. Avro needs the writer's schema, exactly.
This program reads the first order’s Protocol Buffers bytes without the .proto, then its Avro bytes with a schema
that lists two fields the other way round:
"""The first order's Protocol Buffers bytes read without order.proto, then its Avro bytes read with the wrong schema."""
import struct
from encode_compare import FIELDS, avro_decode, avro_encode, orders, proto_encode
from tiny_codecs import read_varint
data = proto_encode(orders[:1])
_, i = read_varint(data, 0)
size, i = read_varint(data, i)
print(f"Protobuf bytes of the order: {data[i:i + size].hex(' ')}")
print("read without the schema, each field is a number, a wire type and raw bytes:")
end = i + size
while i < end:
tag, i = read_varint(data, i)
number, wire = tag >> 3, tag & 7
if wire == 0:
value, i = read_varint(data, i)
shown = f"varint {value}"
elif wire == 2:
n, i = read_varint(data, i)
value, i = data[i : i + n], i + n
shown = f"{n} bytes {value!r}"
else:
value, i = data[i : i + 4], i + 4
shown = f"4 bytes {value.hex(' ')}, a float only if the schema says so: {struct.unpack('<f', value)[0]:.7f}"
print(f" field {number}, wire type {wire}: {shown}")
avro = avro_encode(orders[:1])
print(f"Avro bytes of the order: {avro.hex(' ')}")
swapped = FIELDS[:]
swapped[0], swapped[3] = swapped[3], swapped[0] # a reader whose schema lists paise before id
FIELDS[:] = swapped
try:
print("read with paise and id swapped:", avro_decode(avro))
except Exception as e:
print("read with paise and id swapped:", type(e).__name__, e)Output
Protobuf bytes of the order: 08 d0 86 03 12 05 4e 2d 32 30 34 18 02 20 9a aa 19 28 01 30 e6 84 88 d6 06 3d 9a 99 99 3f
read without the schema, each field is a number, a wire type and raw bytes:
field 1, wire type 0: varint 50000
field 2, wire type 2: 5 bytes b'N-204'
field 3, wire type 0: varint 2
field 4, wire type 0: varint 415002
field 5, wire type 0: varint 1
field 6, wire type 0: varint 1791099494
field 7, wire type 5: 4 bytes 9a 99 99 3f, a float only if the schema says so: 1.2000000
Avro bytes of the order: 02 a0 8d 06 0a 4e 2d 32 30 34 04 b4 d4 32 01 cc 89 90 ac 0d 9a 99 99 3f 00 00
read with paise and id swapped: [{'paise': 50000, 'sku': 'N-204', 'qty': 2, 'id': 415002, 'paid': True, 'placed': 1791099494, 'weight_kg': 1.2000000476837158, 'note': ''}]
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 without_schema.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Without its schema, the Protocol Buffers message still parsed into fields, because each wire type says how long its
value is, but it showed field 1, wire type 0: varint 50000 where the schema says id, and four bytes for field 7
that are a float only because the schema says so. The Avro bytes, read with paise and id swapped, gave id 415002
and paise 50000, and nothing raised an error. That is why Avro files carry the writer’s schema in their header, and
why messages between services need a registry or a version in each message.
Checkpoint
Check: Protocol Buffers and Avro
Sizes depend on the data, so measure your own
The orders here have eight short fields: an id, a product code, a quantity, an amount in paise, a paid flag, a timestamp in Unix seconds, a weight and a note that is usually empty. This program encodes 1,000 of them five ways, compresses each result with gzip, and reads each one back:
"""1,000 synthetic orders in five formats: their sizes, their sizes after gzip, and what came back from each round trip.
Protocol Buffers and Avro follow order.proto and order.avsc; their encoders below are written from the two
specifications for this one schema."""
import gzip
import json
import random
import struct
from tiny_codecs import cbor_decode, cbor_encode, msgpack_decode, msgpack_encode, read_varint, unzigzag, varint, zigzag
rng = random.Random(7)
SKUS = ["B12", "C7", "N-204", "PEN-9", "LAMP-31"]
orders = [
{"id": 50_000 + i, "sku": rng.choice(SKUS), "qty": rng.randint(1, 4), "paise": rng.randrange(1_000, 900_000),
"paid": rng.random() < 0.8, "placed": 1_791_090_000 + rng.randrange(86_400), "weight_kg": rng.choice([0.25, 0.35, 1.2]),
"note": rng.choice(["", "", "", "gift wrap", "call on arrival"])}
for i in range(1_000)
]
FIELDS = [("id", 1, "varint"), ("sku", 2, "string"), ("qty", 3, "varint"), ("paise", 4, "varint"), ("paid", 5, "varint"),
("placed", 6, "varint"), ("weight_kg", 7, "float"), ("note", 8, "string")]
def proto_encode(batch):
"""message Orders { repeated Order orders = 1; } with proto3's rule: a field at its default is not written."""
out = bytearray()
for o in batch:
msg = bytearray()
for name, number, kind in FIELDS:
v = o[name]
if v == ("" if kind == "string" else 0):
continue
if kind == "varint":
msg += varint(number << 3 | 0) + varint(int(v))
elif kind == "string":
data = v.encode("utf-8")
msg += varint(number << 3 | 2) + varint(len(data)) + data
else:
msg += varint(number << 3 | 5) + struct.pack("<f", v)
out += varint(1 << 3 | 2) + varint(len(msg)) + msg
return bytes(out)
def proto_decode(data):
names = {number: (name, kind) for name, number, kind in FIELDS}
batch, i = [], 0
while i < len(data):
_, i = read_varint(data, i) # the tag of field 1 of Orders
size, i = read_varint(data, i)
end, o = i + size, {"id": 0, "sku": "", "qty": 0, "paise": 0, "paid": False, "placed": 0, "weight_kg": 0.0, "note": ""}
while i < end:
tag, i = read_varint(data, i)
name, kind = names[tag >> 3]
if kind == "varint":
o[name], i = read_varint(data, i)
o[name] = bool(o[name]) if name == "paid" else o[name]
elif kind == "string":
n, i = read_varint(data, i)
o[name], i = data[i : i + n].decode("utf-8"), i + n
else:
o[name], i = struct.unpack("<f", data[i : i + 4])[0], i + 4
batch.append(o)
return batch
def avro_encode(batch):
"""An Avro array of records: one block with its count, the fields in schema order, nothing else, then 0."""
out = bytearray(varint(zigzag(len(batch))))
for o in batch:
for name, _, kind in FIELDS:
v = o[name]
if kind == "string":
data = v.encode("utf-8")
out += varint(zigzag(len(data))) + data
elif name == "paid":
out += b"\x01" if v else b"\x00"
elif kind == "float":
out += struct.pack("<f", v)
else:
out += varint(zigzag(v))
return bytes(out + b"\x00")
def avro_decode(data):
count, i = read_varint(data, 0)
batch = []
for _ in range(unzigzag(count)):
o = {}
for name, _, kind in FIELDS:
if kind == "string":
n, i = read_varint(data, i)
o[name], i = data[i : i + unzigzag(n)].decode("utf-8"), i + unzigzag(n)
elif name == "paid":
o[name], i = data[i] == 1, i + 1
elif kind == "float":
o[name], i = struct.unpack("<f", data[i : i + 4])[0], i + 4
else:
n, i = read_varint(data, i)
o[name] = unzigzag(n)
batch.append(o)
return batch
if __name__ == "__main__":
FORMATS = [
("JSON", lambda b: json.dumps(b, separators=(",", ":")).encode(), lambda d: json.loads(d)),
("CBOR", cbor_encode, cbor_decode),
("MessagePack", msgpack_encode, msgpack_decode),
("Protocol Buffers", proto_encode, proto_decode),
("Avro", avro_encode, avro_decode),
]
print(f"{'format':<18}{'bytes':>8}{'gzip':>8} round trip")
for name, encode, decode in FORMATS:
data = encode(orders)
back = decode(data)
changed = sorted({k for o, b in zip(orders, back) for k in o if o[k] != b[k] or type(o[k]) is not type(b[k])})
note = "every value the same" if not changed else "changed: " + ", ".join(changed)
print(f"{name:<18}{len(data):>8}{len(gzip.compress(data, mtime=0)):>8} {note}")
first = orders[0]
print("first order:", json.dumps(first))
print("its weight after a 32-bit float:", struct.unpack("<f", struct.pack("<f", first["weight_kg"]))[0])
print("its CBOR bytes:", cbor_encode(first).hex())Output
format bytes gzip round trip
JSON 112364 13270 every value the same
CBOR 77446 12610 every value the same
MessagePack 78128 12700 every value the same
Protocol Buffers 35953 12422 changed: weight_kg
Avro 27630 11200 changed: weight_kg
first order: {"id": 50000, "sku": "N-204", "qty": 2, "paise": 415002, "paid": true, "placed": 1791099494, "weight_kg": 1.2, "note": ""}
its weight after a 32-bit float: 1.2000000476837158
its CBOR bytes: a862696419c35063736b75654e2d32303463717479026570616973651a0006551a6470616964f566706c616365641a6ac20266697765696768745f6b67fb3ff3333333333333646e6f746560
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 encode_compare.py
Raw, the formats fall into two groups. CBOR and MessagePack saved about 30 % on JSON, mostly the quotes, commas and the digits of the numbers, because every record still spells out its eight keys. Protocol Buffers and Avro, which leave the keys in the schema, took a third and a quarter of JSON’s size. After gzip the groups almost meet, from 11,200 to 13,270 bytes, because repeated keys are exactly what compression removes. Where traffic is compressed anyway, what is left to win is mostly encoding and parsing time, which this program does not measure: time your own code before you switch.
One field did not survive two of the round trips. The schemas declare weight_kg as a 32-bit float, to save four
bytes an order, and 1.2 came back as 1.2000000476837158. JSON, CBOR and MessagePack kept the 64-bit value. A
smaller type is a choice about precision, and the round trip says what it costs.
The bytes are a property of the format, not of a language. The same first order, encoded by a CBOR encoder written in JavaScript, gives the same 76 bytes as the Python one:
// The first order of encode_compare.py, encoded with a CBOR encoder written in JavaScript from RFC 8949. The bytes
// must equal the Python encoder's: CBOR is defined by its bytes, not by a language.
const order = { id: 50000, sku: "N-204", qty: 2, paise: 415002, paid: true, placed: 1791099494, weight_kg: 1.2, note: "" };
function head(major, n) {
if (n < 24) return [(major << 5) | n];
for (const [ai, size] of [[24, 1], [25, 2], [26, 4], [27, 8]]) {
if (n < 2 ** (8 * size)) {
const bytes = [];
for (let i = size - 1; i >= 0; i--) bytes.push(Math.floor(n / 2 ** (8 * i)) % 256);
return [(major << 5) | ai, ...bytes];
}
}
throw new RangeError("too large");
}
function float(x) {
const view = new DataView(new ArrayBuffer(8));
if (typeof view.setFloat16 === "function" && Math.f16round(x) === x) {
view.setFloat16(0, x);
return [0xf9, ...new Uint8Array(view.buffer, 0, 2)];
}
if (Math.fround(x) === x) {
view.setFloat32(0, x);
return [0xfa, ...new Uint8Array(view.buffer, 0, 4)];
}
view.setFloat64(0, x);
return [0xfb, ...new Uint8Array(view.buffer, 0, 8)];
}
function encode(v) {
if (v === false) return [0xf4];
if (v === true) return [0xf5];
if (v === null) return [0xf6];
if (typeof v === "number") return Number.isInteger(v) ? (v >= 0 ? head(0, v) : head(1, -1 - v)) : float(v);
if (typeof v === "string") {
const bytes = new TextEncoder().encode(v);
return [...head(3, bytes.length), ...bytes];
}
if (Array.isArray(v)) return [...head(4, v.length), ...v.flatMap(encode)];
const entries = Object.entries(v);
return [...head(5, entries.length), ...entries.flatMap(([k, x]) => [...encode(k), ...encode(x)])];
}
const hex = Buffer.from(encode(order)).toString("hex");
console.log(`${hex.length / 2} bytes:`, hex);
console.log("1.5 as CBOR:", Buffer.from(encode(1.5)).toString("hex"));
// Python's 100000.0 is a float and became fa47c35000; JavaScript has one number type, so 100000.0 is the integer 100000.
console.log("100000.0 as CBOR:", Buffer.from(encode(100000.0)).toString("hex"));Output
76 bytes: a862696419c35063736b75654e2d32303463717479026570616973651a0006551a6470616964f566706c616365641a6ac20266697765696768745f6b67fb3ff3333333333333646e6f746560 1.5 as CBOR: f93e00 100000.0 as CBOR: 1a000186a0
Recorded with Node.js 24.21.0 on macOS 26 arm64. To run it yourself: mise exec node@24.21.0 -- node cbor_one.mjs
The last line shows a difference between the languages, not the format: JavaScript has one number type, so 100000.0
is the integer 100000 there and is written as one, while Python’s float became fa47c35000.
Key idea
Measure the formats on your own records, raw and compressed: keys repeated in every record are what Protocol Buffers and Avro save and what gzip removes from the others, and a smaller number type changes values.
Checkpoint
Check: Measuring sizes and round trips
Columnar files read only the columns a query needs
The formats so far write records one after another. A question such as “the total of paise over all orders” then
reads every byte of every record to use one field. Parquet: A columnar file format for tables: rows are split into row groups, each with one compressed chunk per column, and metadata at the end of the file says where every chunk starts, so a query reads only its columns. stores a table by columns instead:
rows are split into row groups, each row group holds one contiguous column chunk per column, chunks are made of pages
that are encoded and compressed, and file metadata at the end says where every chunk starts. Follow a reader through
a file:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Start at the end. A reader first reads the last 8 bytes: a 4-byte little-endian length of the file metadata, then PAR1 again. The metadata is written last so that a writer can stream rows in one pass.
- The metadata. The file metadata gives the schema, the row groups and the start of every column chunk. The reader now knows where the paise of every row group are without reading any row.
- Row groups. Rows are split into row groups, horizontal slices of the table. A job can give each row group to a different worker.
- One chunk per column. Inside a row group each column is a contiguous column chunk. A query that sums paise reads the paise chunks and skips every other column.
- Pages. Each chunk is made of pages, the unit that is encoded and compressed. Similar values sit together in a column, so they compress better than mixed rows.
- The magic number. PAR1 at the start and at the end marks the file as Parquet. A reader never reads the other columns at all, which is where the savings come from.
Python’s standard library has no Parquet writer, so this program builds a toy file with the same idea, one compressed chunk per column and a footer with their places, and sums one column from it and from JSON Lines:
"""Why columnar files answer analytics questions cheaply: 100,000 orders as JSON Lines, one row after another, and in a
toy columnar file laid out the way Parquet lays out a row group (one chunk per column, the chunks' places in a footer
read first). Summing one column reads one chunk. A model of the idea, not the Parquet format itself."""
import gzip
import json
import random
rng = random.Random(11)
CITIES = ["Pune", "Kochi", "Jaipur", "Indore", "Surat", "Mysuru"]
COLUMNS = ["id", "city", "sku", "qty", "paise", "paid", "placed"]
rows = [{"id": 70_000 + i, "city": rng.choice(CITIES), "sku": f"SKU-{rng.randrange(400):03d}", "qty": rng.randint(1, 4),
"paise": rng.randrange(1_000, 900_000), "paid": rng.random() < 0.8, "placed": 1_791_090_000 + rng.randrange(86_400)}
for i in range(100_000)]
with open("orders.jsonl", "w", encoding="utf-8") as f:
for r in rows:
f.write(json.dumps(r) + "\n")
# The toy columnar file: each column's values together, gzip-compressed, then a footer that says where each chunk is.
with open("orders.cols", "wb") as f:
f.write(b"COL1")
places = {}
for c in COLUMNS:
chunk = gzip.compress(json.dumps([r[c] for r in rows]).encode(), mtime=0)
places[c] = [f.tell(), len(chunk)]
f.write(chunk)
footer = json.dumps(places).encode()
f.write(footer + len(footer).to_bytes(4, "little") + b"COL1")
def sum_from_rows(path):
total, read = 0, 0
with open(path, "rb") as f:
for line in f: # every byte of every row is read, to use one field
read += len(line)
total += json.loads(line)["paise"]
return total, read
def sum_from_columns(path, column):
with open(path, "rb") as f:
f.seek(-8, 2)
tail = f.read(8)
size = int.from_bytes(tail[:4], "little")
f.seek(-8 - size, 2)
places = json.loads(f.read(size)) # the footer first, as Parquet readers read the file metadata first
start, length = places[column]
f.seek(start)
values = json.loads(gzip.decompress(f.read(length)))
return sum(values), 8 + size + length
rows_total, rows_read = sum_from_rows("orders.jsonl")
cols_total, cols_read = sum_from_columns("orders.cols", "paise")
size_rows = sum(len(json.dumps(r)) + 1 for r in rows)
with open("orders.cols", "rb") as f:
size_cols = len(f.read())
print(f"JSON Lines: {size_rows:>9,} bytes; summing paise read {rows_read:>9,} bytes")
print(f"columnar file: {size_cols:>9,} bytes; summing paise read {cols_read:>9,} bytes")
print(f"same total: {rows_total == cols_total} ({cols_total:,} paise)")
with open("orders.jsonl", "rb") as f:
print(f"JSON Lines with gzip: {len(gzip.compress(f.read(), mtime=0)):,} bytes")Output
JSON Lines: 11,311,202 bytes; summing paise read 11,311,202 bytes columnar file: 1,135,933 bytes; summing paise read 338,986 bytes same total: True (45,060,563,284 paise) JSON Lines with gzip: 1,465,862 bytes
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 columns.py
Summing paise read all 11.3 MB of the JSON Lines and 339 KB of the columnar file, about 3 %, and the columnar file as
a whole was smaller than the JSON Lines compressed with gzip, because similar values stored together compress better.
Real Parquet adds typed encodings, statistics per chunk that let a reader skip whole row groups, and nested columns;
the data analysis track’s lesson on Parquet and Arrow writes and
reads real files with pyarrow.
Avro is the other choice for large data, for a different job: it writes rows, so appending one event is cheap, and an Avro container file carries the writer’s schema in its header, so any reader can resolve it against its own. Events that arrive one at a time suit Avro or Protocol Buffers; a finished table that people query by column suits Parquet. Answer four questions about your own data:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Format | People can read it | Readable without a schema | Size of 1,000 orders | Reads one column cheaply |
|---|---|---|---|---|
| JSON Needs: nothing but a parser | Good plain text, in any editor | Good every name and value is in the text | Poor 112,364 bytes, 13,270 after gzip | Poor every row is read to use one field |
| CBOR Needs: a CBOR library | Poor bytes; a tool shows its diagnostic notation | Good names and types travel with the data | Mixed 77,446 bytes, 12,610 after gzip | Poor every row is read to use one field |
| MessagePack Needs: a MessagePack library | Poor bytes | Good names and types travel with the data | Mixed 78,128 bytes, 12,700 after gzip | Poor every row is read to use one field |
| Protocol Buffers Needs: the .proto, and generated code | Poor bytes with field numbers instead of names | Mixed numbers and wire types only, no names | Good 35,953 bytes, 12,422 after gzip | Poor every message is read to use one field |
| Avro Needs: the writer's schema | Poor bytes | Poor nothing without the writer's schema | Good 27,630 bytes, 11,200 after gzip | Poor rows one after another, as an event stream needs |
| Parquet Needs: a Parquet reader, files only | Poor a file of column chunks; a viewer shows it | Good the schema is in the file's metadata | Good compressed column by column | Good reads the metadata, then only that column's chunks |
How the answers decide
- “Do people read or edit the data by hand?” Yes: rules out CBOR, MessagePack, Protocol Buffers, Avro and Parquet. Only JSON is text a person can read and edit; every binary format needs a tool to show it.
- “How is the data used?” Queries over a few columns of many rows: rules out JSON, CBOR, MessagePack and Protocol Buffers and adds a caveat to Avro. A query over a few columns should read only those columns; row formats read every byte of every record.
- “How is the data used?” A stream of events, appended: rules out Parquet. Parquet writes its metadata at the end of a file, so it suits finished batches, not records appended one at a time.
- “Do all readers share a schema you control?” No, readers get only the data: rules out Avro and Protocol Buffers. Without the schema an Avro reader cannot read anything, and a Protocol Buffers reader sees numbers instead of names.
- “How is the data used?” Messages between services: rules out Parquet. Parquet is a file format for tables, not a message format.
Checkpoint
Check: Columnar files and choosing a format
Practise: encode JSON values as CBOR
The exercise is the heart of a CBOR encoder: the first byte of each item, the shortest argument, negative numbers, UTF-8 lengths, arrays and maps. The sample tests are rows of RFC 8949’s table of examples, so a passing encoder writes the same bytes as any encoder that uses preferred serialization:
Exercise · Medium · Python, JavaScript
Encode JSON values as CBOR, byte for byte
Write cbor_encode(value) in Python (cborEncode(value) in JavaScript) that encodes a JSON value without numbers that have fractions: None/null, False/false, True/true, whole numbers, strings, arrays and objects. Python returns bytes; JavaScript returns an array of byte values from 0 to 255.
CBOR (RFC 8949) starts every item with one byte: the major type in the top three bits and additional information in the low five.
| Major type | Holds | The argument is | | --- | --- | --- | | 0 | an integer 0 or more | the integer | | 1 | a negative integer n | −1 − n | | 3 | a text string | its length in UTF-8 bytes, then the bytes | | 4 | an array | its number of items, then each item | | 5 | a map | its number of pairs, then each key and value |
An argument below 24 sits in the low five bits. Otherwise the low five bits are 24, 25, 26 or 27, and the argument follows in 1, 2, 4 or 8 bytes, big-endian: always the shortest that fits, which is what the RFC calls preferred serialization. false, true and null are the single bytes f4, f5 and f6.
The sample tests are rows of the RFC's table of examples, Appendix A. JavaScript's numbers are exact only up to 2^53, so its tests check 4294967296, the smallest integer that needs 8 bytes, where Python's check the RFC's 18446744073709551615. In JavaScript, utf8() in the starter gives the UTF-8 bytes of a string.
Python · Starter code · cbor.py
def cbor_encode(value):
"""The CBOR bytes of a JSON value without fractions. This first try writes every argument in one byte and
treats negative numbers like positive ones: the tests show where that breaks."""
if value is None:
return b"\xf6"
if isinstance(value, bool):
return b"\xf5" if value else b"\xf4"
if isinstance(value, int):
return bytes([0x18, abs(value)])
if isinstance(value, str):
data = value.encode("utf-8")
return bytes([0x60 | len(data)]) + data
if isinstance(value, list):
return bytes([0x80 | len(value)]) + b"".join(cbor_encode(x) for x in value)
return bytes([0xA0 | len(value)]) + b"".join(cbor_encode(k) + cbor_encode(v) for k, v in value.items())The sample tests · test_cbor.py
from cbor import cbor_encode
def hexes(*values):
return [cbor_encode(v).hex() for v in values]
def test_small_integers():
"""integers below 24 fit in the first byte"""
assert hexes(0, 1, 10, 23) == ["00", "01", "0a", "17"]
def test_larger_integers():
"""1, 2, 4 and 8 more bytes, always the shortest that fits"""
assert hexes(24, 100, 1000, 1000000, 1000000000000) == ["1818", "1864", "1903e8", "1a000f4240", "1b000000e8d4a51000"]
assert cbor_encode(18446744073709551615).hex() == "1bffffffffffffffff"
def test_negative_integers():
"""major type 1 holds -1 - n"""
assert hexes(-1, -10, -100, -1000) == ["20", "29", "3863", "3903e7"]
def test_simple_values():
"""false, true and null"""
assert hexes(False, True, None) == ["f4", "f5", "f6"]
def test_text_strings():
"""the length counts UTF-8 bytes, not characters"""
assert hexes("", "a", "IETF", '"\\', "ü", "水") == ["60", "6161", "6449455446", "62225c", "62c3bc", "63e6b0b4"]
def test_arrays_and_maps():
"""nested arrays and maps, keys in the order given"""
assert hexes([], [1, 2, 3], [1, [2, 3], [4, 5]]) == ["80", "83010203", "8301820203820405"]
assert hexes({}, {"a": 1, "b": [2, 3]}, ["a", {"b": "c"}]) == ["a0", "a26161016162820203", "826161a161626163"]
big = list(range(1, 26))
assert cbor_encode(big).hex() == "98190102030405060708090a0b0c0d0e0f101112131415161718181819"JavaScript · Starter code · cbor.mjs
/** The UTF-8 bytes of a string, as an array of numbers (given: the sandbox has no TextEncoder). */
export function utf8(text) {
return Array.from(unescape(encodeURIComponent(text)), (c) => c.charCodeAt(0));
}
/** The CBOR bytes of a JSON value without fractions. This first try writes every argument in one byte and treats
* negative numbers like positive ones: the tests show where that breaks. */
export function cborEncode(value) {
if (value === null) return [0xf6];
if (value === true) return [0xf5];
if (value === false) return [0xf4];
if (typeof value === 'number') return [0x18, Math.abs(value)];
if (typeof value === 'string') {
const bytes = utf8(value);
return [0x60 | bytes.length, ...bytes];
}
if (Array.isArray(value)) return [0x80 | value.length, ...value.flatMap(cborEncode)];
const entries = Object.entries(value);
return [0xa0 | entries.length, ...entries.flatMap(([k, v]) => [...cborEncode(k), ...cborEncode(v)])];
}The sample tests · cbor.test.mjs
import { test, assert } from 'mysmartcopilot:test';
import { cborEncode } from './cbor.mjs';
const hex = (value) => cborEncode(value).map((b) => b.toString(16).padStart(2, '0')).join('');
const hexes = (...values) => values.map(hex);
test('integers below 24 fit in the first byte', () => {
assert.deepEqual(hexes(0, 1, 10, 23), ['00', '01', '0a', '17']);
});
test('1, 2, 4 and 8 more bytes, always the shortest that fits', () => {
assert.deepEqual(hexes(24, 100, 1000, 1000000, 1000000000000), ['1818', '1864', '1903e8', '1a000f4240', '1b000000e8d4a51000']);
assert.equal(hex(4294967296), '1b0000000100000000');
});
test('major type 1 holds -1 - n', () => {
assert.deepEqual(hexes(-1, -10, -100, -1000), ['20', '29', '3863', '3903e7']);
});
test('false, true and null', () => {
assert.deepEqual(hexes(false, true, null), ['f4', 'f5', 'f6']);
});
test('the length counts UTF-8 bytes, not characters', () => {
assert.deepEqual(hexes('', 'a', 'IETF', '"\\', 'ü', '水'), ['60', '6161', '6449455446', '62225c', '62c3bc', '63e6b0b4']);
});
test('nested arrays and maps, keys in the order given', () => {
assert.deepEqual(hexes([], [1, 2, 3], [1, [2, 3], [4, 5]]), ['80', '83010203', '8301820203820405']);
assert.deepEqual(hexes({}, { a: 1, b: [2, 3] }, ['a', { b: 'c' }]), ['a0', 'a26161016162820203', '826161a161626163']);
const big = Array.from({ length: 25 }, (_, i) => i + 1);
assert.equal(hex(big), '98190102030405060708090a0b0c0d0e0f101112131415161718181819');
});A hint
Write one helper first, head(major, n), that returns the first byte and the argument: bytes([major << 5 | n]) when n < 24, otherwise the byte with 24, 25, 26 or 27 and then n in 1, 2, 4 or 8 big-endian bytes. Every case of the encoder is then one call to head followed by the item's content.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide, QuickJS): nothing is sent to mysmartcopilot.com. The first run of each language downloads it: Python (about 13.5 MB) or JavaScript (about 0.6 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then test yourself on bytes, schemas, sizes and columns. The questions adapt to your answers and give the reason behind every choice:
Practice that adapts to you
Practise this lesson
12 questions about 4 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher to 5 years): when does a binary format beat JSON?
When size or parsing time matter more than reading the data by eye, and when you can measure the gain. Schema-based formats such as Protocol Buffers and Avro leave field names out of every record, so 1,000 small orders took a third to a quarter of their JSON size. Schema-less binary formats such as CBOR and MessagePack keep the names and save less, about 30 % here, but they also carry byte strings and tagged values that JSON cannot. After gzip the differences shrank to 16 % at most, so for compressed traffic what is left to win is mostly parsing time, which you measure too. For analytics over a few columns, a columnar file such as Parquet beats every row format by reading only those columns. You give up text that people can read, and with Protobuf and Avro the ability to read data without its schema.
Two design questions follow, about telemetry from a mobile app and about storage for analytics. Write down the numbers you would need, then open each first step:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
A mobile app uploads telemetry as JSON batches of about 40 KB, compressed with gzip, over slow connections. The team proposes Protocol Buffers to cut data costs. How do you decide, and what numbers do you need first?
-
Senior to staff
An analytics team stores a day's events as JSON Lines, about 200 GB a day with 60 fields, and most queries use 3 of them. Recommend a storage format and layout, and explain the trade-offs.
Flashcards · 6 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
How do CBOR and MessagePack differ from JSON?
They encode JSON's values as bytes, member names included, need no schema, and add byte strings and tagged values, such as CBOR's tags and MessagePack's extension types. For the lesson's orders they saved about 30 %.
-
Why can't Protocol Buffers or Avro data be read without its schema?
The names live in the schema. Protobuf writes field numbers and wire types, Avro writes values only, so the wrong schema reads the wrong values without an error.
-
How did gzip change the size comparison of the five formats?
Raw, Protobuf and Avro were a third and a quarter of JSON's size; after gzip all five sat within 2,100 bytes, because compression removes the repeated keys.
-
Why did the value 1.2 change in a binary encoding?
It was stored as a 32-bit float, which cannot hold 1.2 exactly. A smaller number type changes values.
-
When does a columnar file such as Parquet beat row formats?
For analytics over a few columns: it reads only those columns, 3 % of the bytes in the toy file.
-
When do Avro or Protobuf fit better than Parquet?
For records that are appended or sent one at a time, such as events and messages.
Key takeaways
- CBOR and MessagePack encode JSON’s values as bytes with every member’s name, need no schema, add byte strings and tags, and saved about 30 % on JSON for these orders.
- Protocol Buffers and Avro leave the names in the schema: Protobuf writes field numbers and wire types, Avro writes values only, so the wrong schema reads the wrong values without an error.
- Measure your own data raw and compressed: Protobuf and Avro were a third and a quarter of JSON’s size, but gzip
brought all five within 2,100 bytes, and a 32-bit float changed
1.2. - Columnar files such as Parquet read only the columns a query needs, 3 % of the bytes in the toy file, while Avro and Protobuf suit records that are appended or sent one at a time.
Terms in this lesson
- Avro
- A binary format that writes only the values, in the order of the schema, with no names or tags. A reader resolves its own schema against the writer's, which container files carry in their header.
- CBOR
- Concise Binary Object Representation, RFC 8949: a binary format for JSON-like data in which each item starts with a byte that gives its major type and its length. It adds byte strings and tags, and needs no schema.
- MessagePack
- A schema-less binary format for JSON-like data, registered as application/vnd.msgpack, whose serializers use the format with the fewest bytes for each value, such as one byte for an integer from 0 to 127.
- Parquet
- A columnar file format for tables: rows are split into row groups, each with one compressed chunk per column, and metadata at the end of the file says where every chunk starts, so a query reads only its columns.
- Protocol Buffers
- A binary format whose .proto schema gives each field a type and a number. Only the numbers travel, as tags before each value, so a reader needs the schema to know a field's name and meaning.
- varint
- A variable-length integer: seven bits of the number in each byte, low bits first, with the top bit set on every byte but the last. Protocol Buffers and Avro use it, so small numbers take one byte.
References
- RFC 8949: Concise Binary Object Representation (CBOR) (IETF)
- MessagePack specification (MessagePack project)
- Media type application/vnd.msgpack (IANA)
- Protocol Buffers: encoding (Google)
- Language Guide (proto 3): default field values (Google)
- Apache Avro 1.12.0 Specification (Apache Software Foundation)
- Parquet file format (Apache Software Foundation)
- Parquet concepts: row groups, column chunks and pages (Apache Software Foundation)
- struct: interpret bytes as packed binary data (Python Software Foundation)
- gzip: support for gzip files (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress