Numbers, bytes and text on small chips

Put 16777217.0 into a float on a Pico and the board keeps 16777216.0: it stores floats in 32 bits, and that number needs 25 bits of precision. The same code on your laptop, on the unix port or in the browser build keeps it exactly, so sensor maths that passes its tests on a computer can be off on the board by more than the sensor’s own error.

This lesson shows what single-precision floats do to real calculations, how to keep sensor maths in whole numbers, how to pack and unpack binary frames with struct, int.to_bytes() and int.from_bytes(), and how MicroPython handles text as UTF-8 bytes under only three encoding names.

The Pico and the ESP32 keep floats in 32 bits

The rp2 and esp32 ports build MicroPython with single-precisionsingle precision: The 32-bit floating-point format of IEEE 754, with a 24-bit significand, about 7 significant decimal digits. The rp2 and esp32 ports store MicroPython floats this way; CPython uses the 64-bit double format. floats, as their configuration files say with MICROPY_FLOAT_IMPL_FLOAT. The unix port’s standard variant and the browser build use double precision, the same 64-bit floats as CPython. A single-precision float keeps a 24-bit significand, about 7 significant decimal digits, and every operation on such a board rounds its result to the nearest value that fits.

The next program is a model of that rounding: f32() rounds a number to the nearest single-precision value with struct, whose f format is IEEE 754’s 32-bit format, and every operation goes through it, so it gives the result a board stores, not an estimate. It prints each result as a double-precision build keeps it, and as a single-precision board stores it:

"""Four calculations as a double-precision and a single-precision build store their results.

The rp2 and esp32 ports build MicroPython with single-precision floats (MICROPY_FLOAT_IMPL_FLOAT
in their mpconfigport.h); the unix port and the browser build use doubles, as CPython does.
f32() rounds a number to the nearest IEEE 754 single-precision value, as every operation on
such a board does, so this model gives the board's stored result exactly.
"""
import struct


def f32(x):
    return struct.unpack("<f", struct.pack("<f", x))[0]


def short(x):
    """The shortest decimal that reads back as the same single-precision value."""
    for digits in range(1, 10):
        text = "%.*g" % (digits, x)
        if f32(float(text)) == x:
            return text
    return repr(x)


def tenth_added(r, times):
    total = r(0.0)
    for _ in range(times):
        total = r(total + r(0.1))
    return total


CASES = [
    ("0.1 + 0.2", lambda r: r(r(0.1) + r(0.2))),
    ("1 / 3", lambda r: r(r(1.0) / r(3.0))),
    ("16777217.0", lambda r: r(16777217.0)),
    ("0.1 added 10000 times", lambda r: tenth_added(r, 10000)),
]

for name, calc in CASES:
    double = calc(float)
    single = calc(f32)
    print(name)
    print("  double: %r" % double)
    print("  single: %s  (stored as %r)" % (short(single), single))


def gap(x):
    """The distance from x to the next larger single-precision value."""
    bits = struct.unpack("<I", struct.pack("<f", x))[0]
    return struct.unpack("<f", struct.pack("<I", bits + 1))[0] - x


for x in (1.0, 1000.0, 1000000.0, 16777216.0):
    print("next single-precision value after %r is %r away" % (x, gap(x)))

0.1 + 0.2
  double: 0.30000000000000004
  single: 0.3  (stored as 0.30000001192092896)
1 / 3
  double: 0.3333333333333333
  single: 0.33333334  (stored as 0.3333333432674408)
16777217.0
  double: 16777217.0
  single: 16777216  (stored as 16777216.0)
0.1 added 10000 times
  double: 1000.0000000001588
  single: 999.9029  (stored as 999.9028930664062)
next single-precision value after 1.0 is 1.1920928955078125e-07 away
next single-precision value after 1000.0 is 6.103515625e-05 away
next single-precision value after 1000000.0 is 0.0625 away
next single-precision value after 16777216.0 is 2.0 away

Three things stand out. 0.1 + 0.2 lands exactly on the single-precision value nearest to 0.3, so the board’s result prints as 0.3, while CPython’s famous 0.30000000000000004 appears only in double precision. 16777217.0 cannot be stored at all: above 2²⁴ = 16,777,216 the next value is 2.0 away. And adding 0.1 ten thousand times gives 999.9029 instead of 1000, because each of the 10,000 additions is rounded to the nearest single-precision value and the roundings do not cancel out.

The program prints the shortest decimal that reads back as the stored value. Since 1.26, MicroPython’s own float printer is built for that property too; its release notes measure 98.5 % of single-precision values printing in a form that reads back exactly, so a board shows these short forms in most cases.

Keep sensor maths in whole numbers

MicroPython’s speed guide gives the advice for every port: some ports keep floats on the heap and some do their float arithmetic in software, so take readings as integers, keep them as integers, and convert to floats only where the rest of the code needs them. On a single-precision board there is a second reason: integers do not round. A sensor that reports whole hundredths of a degree can stay in hundredths from the bus to the log, which is fixed-pointfixed-point: Keeping a fractional quantity as a whole number of small units, such as a temperature in hundredths of a degree, so that the arithmetic is exact integer arithmetic and the point is placed only when the number is shown. arithmetic.

The next program puts both reasons into numbers. First it converts readings in degrees to hundredths with int(t * 100); then it adds up a week of readings taken once a second, 604,800 of them, three ways:

"""Sensor maths in floats and in whole numbers of hundredths, on double and single-precision builds.

f32() rounds to the nearest single-precision value, as each operation on a Pico or an
ESP32 board does (the model of float_width.py). The sensor reports whole hundredths of
a degree, so the integer path never rounds at all.
"""
import struct

F = struct.Struct("<f")


def f32(x):
    return F.unpack(F.pack(x))[0]


# 1. Converting a reading in degrees to hundredths with int(t * 100) truncates.
for t in (19.99, 36.6):
    double = int(t * 100)
    single = int(f32(f32(t) * f32(100.0)))
    print("int(%s * 100): double %d, single %d" % (t, double, single))

# 2. A week of readings, one a second, added up for the mean: 21.00 to 21.74 degrees.
n = 7 * 24 * 3600
degrees = [f32(f32(h) / f32(100.0)) for h in range(2100, 2175)]
exact = 0
double = 0.0
single = 0.0
for i in range(n):
    hundredths = 2100 + (i * 37) % 75
    exact += hundredths
    double += hundredths / 100
    single = f32(single + degrees[hundredths - 2100])
print("%d readings; sum of hundredths %d, %s 2**30" % (n, exact, "above" if exact >= 2**30 else "below"))
mean = exact // n
print("mean from integers:      %d.%02d" % (mean // 100, mean % 100))
print("mean from double floats: %.4f" % (double / n))
print("mean from single floats: %.4f" % f32(single / f32(n)))

int(19.99 * 100): double 1998, single 1999
int(36.6 * 100): double 3660, single 3659
604800 readings; sum of hundredths 1292457600, above 2**30
mean from integers:      21.37
mean from double floats: 21.3700
mean from single floats: 21.3452

int(t * 100) truncates in both precisions, on different values: 19.99 becomes 1998 on a double-precision build, and 36.6 becomes 3659 on a single-precision one. The week’s mean is 21.37 degrees from the integers and from double floats, but 21.3452 from single floats: the running total grew so large that each new reading was rounded on the way in, and the mean lost 0.0248 of a degree. The integer sum is exact by construction, not by luck.

Integers on MicroPython grow without limit, as in CPython, but they come in two sizes. A small integersmall integer: An integer that MicroPython keeps inside the object word itself, with no heap allocation: on most 32-bit ports one from -2**30 to just below 2**30. Larger integers stay exact but are stored on the heap., from −2³⁰ to just below 2³⁰ on most 32-bit ports, lives inside the object word and needs no heap; a larger one is stored on the heap, which costs time and memory. The week’s sum, 1,292,457,600 hundredths, is above 2³⁰ = 1,073,741,824, so on a board it is a heap integer: still exact, but allocated. A day’s sum, a seventh of it, stays small.

struct packs numbers into frames of bytes

Sensors, radios and other boards speak in bytes, so numbers have to be packed into frames and unpacked again. MicroPython’s struct module does it the CPython way, with a format string: > for big-endianbig-endian: The byte order that puts the most significant byte of a number first, as network protocols do; > in a struct format. Little-endian, <, puts the least significant byte first. order, B for an unsigned byte, h for a signed 16-bit number, H for an unsigned one, and so on, in the standard sizes of its documentation. That page lists one difference, no spaces in a format, and the differences pages add that MicroPython does not check how many values you pass. struct.pack_into() writes into a buffer you made once, so a loop that sends a frame a second does not make a new bytes object each time. Step through one six-byte frame:

The temperature’s two bytes, ff e2, are -30 in two’s complementtwo's complement: The usual way of storing negative whole numbers in a fixed number of bits: in 16 bits, -30 is stored as 65536 - 30 = 65506, so the same bytes read as -30 when signed and 65506 when unsigned.: read as a plain unsigned number they are 65506. The program below packs the same frame, checks it with a CRC-8CRC: Cyclic redundancy check: a short value computed from a block of bytes by polynomial division, sent along with them, so that a receiver that computes it again can tell when bits were changed on the way., reads it back, and shows both readings of those two bytes:

"""A six-byte sensor frame packed and checked with struct, the same way on CPython and MicroPython.

Byte 0 is the sensor, bytes 1-2 the temperature in hundredths of a degree (signed, big-endian),
bytes 3-4 the humidity in tenths of a percent (unsigned) and byte 5 a CRC-8 of the first five
(polynomial 0x07, as SMBus uses). struct needs no spaces in its formats on MicroPython.
"""
import struct

FORMAT = ">BhH"  # 5 bytes: sensor, temperature, humidity


def crc8(data):
    crc = 0
    for byte in data:
        crc ^= byte
        for _ in range(8):
            crc = ((crc << 1) ^ 0x07) & 0xFF if crc & 0x80 else (crc << 1) & 0xFF
    return crc


def sign16(value):
    """Two's complement of a 16-bit unsigned value: what int.from_bytes(..., signed=True) gives in CPython."""
    return value - 0x10000 if value & 0x8000 else value


frame = bytearray(6)  # made once; pack_into fills it without a new allocation
struct.pack_into(FORMAT, frame, 0, 3, -30, 455)  # sensor 3, -0.30 degrees, 45.5 %
frame[5] = crc8(frame[:5])
print("frame:", frame.hex())

sensor, temp, hum = struct.unpack_from(FORMAT, frame, 0)
print("crc ok:", crc8(frame[:5]) == frame[5])
print("sensor %d, temperature %d hundredths, humidity %d tenths" % (sensor, temp, hum))
raw = int.from_bytes(frame[1:3], "big")
print("bytes 1-2 unsigned: %d, sign-extended: %d" % (raw, sign16(raw)))
print("check value of CRC-8 over b'123456789': 0x%02x" % crc8(b"123456789"))
print("(-30).to_bytes(2, 'big', signed=True): %s" % (-30).to_bytes(2, "big", signed=True).hex())

frame: 03ffe201c721
crc ok: True
sensor 3, temperature -30 hundredths, humidity 455 tenths
bytes 1-2 unsigned: 65506, sign-extended: -30
check value of CRC-8 over b'123456789': 0xf4
(-30).to_bytes(2, 'big', signed=True): ffe2

The frame is 03ffe201c721, the same bytes as the figure, and the CRC function gives 0xf4 for the bytes of “123456789”, the standard check value of this CRC, so it can be tested against any other implementation.

For a single number, int.to_bytes() and int.from_bytes() are shorter, with one trap. Since 1.29, to_bytes(length, byteorder, signed=True) works on MicroPython as in CPython, and the program’s last line used it. from_bytes() does not support signed numbers at all: the 1.29.0 source takes its arguments by position only, so signed=True raises TypeError, and a fourth argument is accepted but ignored, so negative values come out as large positive ones. For signed fields, use a struct format such as h, or the sign step of sign16() in the program.

Text is UTF-8, under three encoding names

MicroPython stores every string as UTF-8UTF-8: The encoding that stores each Unicode code point in one to four bytes, ASCII characters in one. MicroPython keeps every str as UTF-8, and accepts only the names utf-8, utf8 and ascii when it converts text., and on all ports of support tiers 1 to 3 its strings are made of Unicode code points: len(), indexing and slicing count code points, not bytes. That is CPython’s model, and the difference shows only at the edges, where text turns into bytes and back:

• Three encoding names. encode(), decode(), str() and bytes() accept only "utf-8", "utf8" and "ascii", spelled exactly so. Since 1.29 any other name raises LookupError, "latin-1" and even "UTF-8" included, where CPython also accepts other spellings and many more encodings.
• Errors when decoding. Since 1.29, decode() takes "strict", "ignore" or "replace", given by position: keyword arguments raise TypeError. A strict decode of bad bytes raises UnicodeError, and except UnicodeError also catches CPython’s UnicodeDecodeError, a subclass of it. Encoding ignores the errors argument altogether.
• No \N{...} escapes. Writing a character by its Unicode name raises NotImplementedError.

The next program measures three words in characters and in bytes, cuts a display line to fit 16 and 20 bytes, and decodes a buffer whose last character was cut in half:

"""Text as characters and as UTF-8 bytes, in forms that MicroPython 1.29.0 accepts too.

MicroPython keeps str as UTF-8 and accepts only the encoding names "utf-8", "utf8" and
"ascii", in lower case; bytes.decode() takes its arguments by position.
"""


def fit_utf8(text, max_bytes):
    """The longest start of text whose UTF-8 form fits in max_bytes, never cut inside a character."""
    out = ""
    used = 0
    for ch in text:
        size = len(ch.encode("utf-8"))
        if used + size > max_bytes:
            break
        out += ch
        used += size
    return out


for word in ("hello", "héllo", "नमस्ते"):
    print("%s: %d characters, %d bytes in UTF-8" % (word, len(word), len(word.encode("utf-8"))))

line = "Temp 21.5 °C नमस्ते"
for width in (16, 20):
    part = fit_utf8(line, width)
    print("fits in %d bytes: %r (%d bytes)" % (width, part, len(part.encode("utf-8"))))

broken = "café".encode("utf-8")[:4]  # the last character cut in half
print("cut bytes:", broken)
print("decoded with 'replace':", repr(broken.decode("utf-8", "replace")))
print("decoded with 'ignore':", repr(broken.decode("utf-8", "ignore")))

hello: 5 characters, 5 bytes in UTF-8
héllo: 5 characters, 6 bytes in UTF-8
नमस्ते: 6 characters, 18 bytes in UTF-8
fits in 16 bytes: 'Temp 21.5 °C ' (14 bytes)
fits in 20 bytes: 'Temp 21.5 °C नम' (20 bytes)
cut bytes: b'caf\xc3'
decoded with 'replace': 'caf�'
decoded with 'ignore': 'caf'

“नमस्ते” is 6 code points and 18 bytes, three bytes each, so a 16-byte display line holds Temp 21.5 °C and a space, 14 bytes, and not one Devanagari letter more. The cutting function walks the text one character at a time; step through its loop on a shorter line, with a limit of 8 bytes:

text = "°C नमस्ते"
limit = 8
out = ""
used = 0
for ch in text:
    size = len(ch.encode())
    if used + size > limit:
        break
    out += ch
    used += size
print(out, used)

°C न 7

Practise: decode a buffer of frames

Write the receiving end of the frame: a function that splits a buffer into six-byte frames, checks each CRC and returns the readings in whole numbers. The sample tests include cold readings, a broken byte and a buffer cut short:

import struct


def crc8(data):
    """CRC-8 with the polynomial 0x07 (as SMBus uses), starting from 0."""
    crc = 0
    for byte in data:
        crc ^= byte
        for _ in range(8):
            crc = ((crc << 1) ^ 0x07) & 0xFF if crc & 0x80 else (crc << 1) & 0xFF
    return crc


def decode_frames(buf):
    """Return ([(sensor, temperature, humidity), ...], skipped) for the six-byte frames in buf."""
    return [], 0

import struct

from frames import crc8, decode_frames


def frame(sensor, temperature, humidity):
    head = struct.pack(">BhH", sensor, temperature, humidity)
    return head + bytes([crc8(head)])


def test_one_frame():
    """decodes the lesson's frame: sensor 3, -0.30 degrees, 45.5 %"""
    assert decode_frames(bytes.fromhex("03ffe201c721")) == ([(3, -30, 455)], 0)


def test_several_and_negative():
    """decodes several frames in order, cold ones included"""
    buf = frame(1, 2137, 400) + frame(2, -1250, 999) + frame(3, -1, 0)
    readings, skipped = decode_frames(bytearray(buf))
    assert readings == [(1, 2137, 400), (2, -1250, 999), (3, -1, 0)]
    assert skipped == 0


def test_bad_crc_skipped():
    """skips a frame whose CRC does not match, and counts it"""
    good = frame(1, 2000, 500)
    bad = bytearray(frame(2, 2100, 510))
    bad[2] ^= 0x10
    readings, skipped = decode_frames(good + bytes(bad) + good)
    assert readings == [(1, 2000, 500), (1, 2000, 500)]
    assert skipped == 1


def test_cut_short():
    """ignores bytes that do not make a whole frame"""
    buf = frame(4, 150, 600) + frame(5, 160, 610)[:4]
    assert decode_frames(buf) == ([(4, 150, 600)], 0)


def test_whole_numbers():
    """returns whole numbers, never floats"""
    readings, _ = decode_frames(frame(6, -32768, 65535))
    assert readings == [(6, -32768, 65535)]
    assert all(type(value) is int for value in readings[0])


def test_empty():
    """gives nothing for an empty buffer"""
    assert decode_frames(b"") == ([], 0)

Then take the self-check. It covers single-precision floats, whole-number maths, frames and text, starts with the part you have answered worst so far, and explains each answer you choose:

Interview questions

Warm-up (fresher): Why does 16777217.0 come back as 16777216.0 on a Pico?

The rp2 port builds MicroPython with single-precision floats, the IEEE 754 32-bit format, which keeps a 24-bit significand. Every whole number up to 2 to the 24th power, 16,777,216, fits exactly, but above it the gap between neighbouring values is 2, so 16,777,217 rounds to the nearest value that exists, 16,777,216. The unix port, the browser build and CPython use 64-bit doubles, which hold every whole number up to 2 to the 53rd, so the same line keeps its value there. On a board, keep counts and sensor readings as integers, which have no such limit.

This lesson leads to two longer questions in the module’s bank:

Key takeaways

• The rp2 and esp32 ports store floats in 32 bits, about 7 significant digits, while the unix port, the browser build and CPython use doubles, so the same calculation can give different results on a board.
• Readings kept in whole units, such as hundredths, never round; int(t * 100) truncates in both precisions, and a long single-precision sum loses accuracy as it grows.
• Integers below 2³⁰ are small and need no heap on 32-bit ports; larger ones stay exact but are allocated on the heap.
• struct packs and unpacks frames as in CPython, without spaces in formats, and int.from_bytes() ignores signedness on MicroPython, so signed fields use a format such as h or a sign step of your own.
• Strings are UTF-8 code points; only "utf-8", "utf8" and "ascii" are accepted as encoding names, and decoding takes its error mode by position.