Filesystems: LittleFS, FAT and vfs.mount

A Pico keeps every file you copy to it in the last 1,408 KiB of its 2 MiB flash chip, and that chip cannot change a byte the way RAM does: a bit that has been cleared to 0 becomes a 1 again only when the whole 4,096-byte sector around it is erased. What open() shows you as files is the work of two layers in between, a filesystem that decides where data goes and a block device that reads, programs and erases the flash.

You will follow a file down those layers, compare the two filesystems MicroPython puts on flash, LittleFS and FAT, read vfs.mount() and os.statvfs() the way a board reports them, and build a block device in RAM, so that code which writes files can be tested on your computer.

Flash is erased in sectors, so files sit on a block device

Say a file holds the text 21.5 and the new value is 19.0. The Pico’s flash has two operations that change data. Erasing sets every bit of a sectorsector (of flash): The smallest part of a flash chip that can be erased at once, 4,096 bytes on the Pico. An erase sets all its bits to 1; programming can then clear bits to 0, and only another erase sets them back. to 1, so all its bytes read 0xff; the pico-sdk’s flash functions erase in sectors of 4,096 bytes, and an ESP32 board’s partitions erase in blocks of the same size. Programming then turns bits from 1 to 0, and nothing turns a 0 back into a 1 except another erase of the whole sector. The next program models one sector by those two rules and writes the new value straight over the old one:

"""One 4,096-byte sector of a Pico's flash, by the rules of the pico-sdk's flash functions.

Erasing a sector sets all its bits to 1, so every byte reads 0xff. Programming can only turn
bits from 1 to 0, so a byte programmed over old data becomes the AND of the old and the new.
"""
SECTOR = 4096


class Sector:
    def __init__(self):
        self.data = bytearray(b"\xff" * SECTOR)  # a new chip comes erased
        self.erases = 0

    def erase(self):
        self.data[:] = b"\xff" * SECTOR
        self.erases += 1

    def program(self, offset, buf):
        for i, b in enumerate(buf):
            self.data[offset + i] &= b  # 1 can become 0, never 0 back to 1

    def read(self, offset, n):
        return bytes(self.data[offset:offset + n])


def show(label, data):
    print("%-22s %-8r %s" % (label, data, data.hex(" ")))


s = Sector()
show("erased:", s.read(0, 4))
s.program(0, b"21.5")
show("programmed 21.5:", s.read(0, 4))
s.program(0, b"19.0")  # the new value over the old, no erase
show("19.0 over it:", s.read(0, 4))
s.program(4, b"22.0")  # into erased bytes after it
show("22.0 after it:", s.read(4, 4))
s.erase()
s.program(0, b"19.0")
show("erase, then 19.0:", s.read(0, 4))
print("erases: %d, each one of %d bytes" % (s.erases, SECTOR))

erased:                b'\xff\xff\xff\xff' ff ff ff ff
programmed 21.5:       b'21.5'  32 31 2e 35
19.0 over it:          b'01.0'  30 31 2e 30
22.0 after it:         b'22.0'  32 32 2e 30
erase, then 19.0:      b'19.0'  31 39 2e 30
erases: 1, each one of 4096 bytes

Programming 19.0 over 21.5 left 01.0: every bit that was already 0 stayed 0, and the result is a different number that still looks like a number. The bytes after it were still erased, so 22.0 went in cleanly: data can always be added in erased space. Changing data in place, though, means erasing and rewriting all 4,096 bytes of the sector for a change of 4.

So a filesystem never writes to the chip directly. It works through a block deviceblock device: An object that stores data in numbered blocks of one size and offers readblocks(), writeblocks() and ioctl() for the block count, the block size and erasing. MicroPython's filesystems sit on one, such as rp2.Flash or an SD card., an object that splits the storage into numbered blocks of one size. On a Pico, rp2.Flash() is the block device for the part of the flash kept for files: 352 blocks of 4,096 bytes, the 1,441,792 bytes the RPI_PICO firmware sets aside. On an ESP32 board it is the partition named vfs, an esp32.Partition. Every block device has the same three methods:

The forms with an offset make up the extended interface, the forms without one the simple interface. LittleFS needs the extended one: it erases a block with ioctl(6, n) and then programs parts of it, the add-in-erased-space pattern the program showed to be safe.

LittleFS survives power cuts; FAT is what a PC can read

Two filesystems matter on a board’s block devices, and both the rp2 and the esp32 port start with the same one. Their boot scripts decide what happens before your boot.py runs:

The two ports treat a broken filesystem in opposite ways: a Pico starts again empty, while an ESP32 board waits, with your data still on the flash for a recovery attempt.

LittleFSLittleFS: A small filesystem designed for flash: it writes changes to free blocks, commits them in small checked logs and spreads erases over the blocks. The rp2 and esp32 ports of MicroPython format their flash with LittleFS v2. was designed for flash. Its documentation lists what that buys: it is copy-on-writecopy-on-write: Writing changed data to free blocks and switching to it in one small commit, instead of writing over the old data, so that a power cut leaves the last committed version intact., writing changed data to free blocks instead of over the old data; it keeps its metadata in small logs, two blocks each, whose commits carry a 32-bit CRC; a file’s changes count only once the file is closed or synced; and rename and remove are atomic even if power is lost. A power cut leaves the filesystem in its last committed state. LittleFS also spreads its erases over the blocks it allocates.

FATFAT: The File Allocation Table filesystem of SD cards and USB drives, which every computer can read. MicroPython's documentation warns that a power failure during a write can corrupt it., the filesystem of SD cards and USB drives, has the opposite strength: any computer reads it. MicroPython’s filesystem documentation warns that FAT is not tolerant to power failure during writes, which can corrupt the filesystem, and recommends LittleFS wherever a PC does not need to read the storage. The littlefs design notes count FAT among the block-based filesystems that bind data to fixed places on the storage, which leaves them no way to manage wear. Answer the three questions below for a device of your own:

In short: LittleFS for the board’s own flash, which is what both ports format anyway; FAT for an SD card that people carry to a PC, with the files that change often kept on LittleFS; and LittleFS on a card only through a driver with the extended interface.

vfs.mount() shows what is mounted, os.statvfs() how full it is

Every path goes through a table of mounts. A filesystem is mounted at a mount pointmount point: The folder at which a mounted filesystem appears, / or a name such as /sd. MicroPython sends every path that starts with it, followed by a slash or nothing, to that filesystem., / or a name such as /sd, with vfs.mount(fs, "/sd"), and vfs.umount("/sd") takes it away again. vfs.mount() with no arguments returns the table itself: a list of (filesystem, mount point) pairs, one pair on a fresh Pico, its LittleFS object and /. Follow two paths through a Pico with an SD card mounted at /sd:

The rule from the last steps is the one to remember: a mount point catches a path only when the path continues with a slash or ends there, and the root mount takes every other path.

How full a filesystem is comes from os.statvfs(path), a tuple of ten numbers, of which the first four matter: the block size twice, the number of blocks and the free blocks. Free bytes are st[0] * st[3]. MicroPython’s LittleFS driver counts the free blocks by walking every block the filesystem uses, sets the inode fields to 0, and puts the longest name, 255 bytes, last. On FAT the block is a cluster. mpremote df prints the same figures from your computer. The next program works out each board’s filesystem from its configuration, then reads a statvfs result:

"""How big each board's filesystem is, and what an os.statvfs() tuple says in bytes.

Sizes follow MicroPython 1.29.0: RPI_PICO keeps 1,441,792 bytes of its flash for the
filesystem; an ESP32 board gives it the flash after the last partition of its table.
The LittleFS driver reports block size, block count and free blocks in statvfs.
"""
BLOCK = 4096
MiB = 1024 * 1024


def esp32_vfs(table_end, flash):
    return flash - table_end


boards = [
    ("RPI_PICO", 1441792),
    ("ESP32_GENERIC, 4 MB", esp32_vfs(0x200000, 4 * MiB)),
    ("ESP32_GENERIC, 8 MB", esp32_vfs(0x200000, 8 * MiB)),
    ("ESP32_GENERIC-OTA", esp32_vfs(0x310000, 4 * MiB)),
]
print("%-20s %10s %7s" % ("board", "bytes", "blocks"))
for name, size in boards:
    print("%-20s %10s %7s" % (name, "{:,}".format(size), "{:,}".format(size // BLOCK)))

# Suppose os.statvfs("/") on a Pico returns this:
st = (4096, 4096, 352, 160, 160, 0, 0, 0, 0, 255)
bsize, blocks, bfree = st[0], st[2], st[3]
print()
print("os.statvfs('/') =", st)
print("size {:,} B, used {:,} B, free {:,} B ({} %)".format(
    bsize * blocks, bsize * (blocks - bfree), bsize * bfree, 100 * bfree // blocks))


def blocks_for(nbytes):
    return -(-nbytes // bsize)  # rounded up


# LittleFS writes a new version of a file to free blocks and frees the old
# blocks only after it commits, so a rewrite needs room for the whole new copy,
# and a little more: each block of a file also holds a few pointers.
for size in (614400, 700000):
    need = blocks_for(size)
    print("new copy of {:,} B: needs at least {} blocks, {} free: {}".format(
        size, need, bfree, "fits" if need <= bfree else "no room"))

board                     bytes  blocks
RPI_PICO              1,441,792     352
ESP32_GENERIC, 4 MB   2,097,152     512
ESP32_GENERIC, 8 MB   6,291,456   1,536
ESP32_GENERIC-OTA       983,040     240

os.statvfs('/') = (4096, 4096, 352, 160, 160, 0, 0, 0, 0, 255)
size 1,441,792 B, used 786,432 B, free 655,360 B (45 %)
new copy of 614,400 B: needs at least 150 blocks, 160 free: fits
new copy of 700,000 B: needs at least 171 blocks, 160 free: no room

An ESP32_GENERIC board gives its filesystem the flash after the end of its partition table: the firmware ends at 2 MiB, so a 4 MB module gets 2 MiB and an 8 MB module 6 MiB. The OTA firmware’s two application slots end at 0x310000, which leaves 983,040 bytes of a 4 MB module. The last two lines show a cost of copy-on-write. LittleFS writes the new version of a file to free blocks and frees the old blocks only after the commit, so a 700,000-byte rewrite needs at least 171 free blocks at once, however many blocks the version it replaces holds. The exact count is a little higher, because each block of a file also holds a few 4-byte pointers to earlier blocks.

A block device in RAM gives tests a filesystem of their own

Code that writes files needs tests a laptop can run: a full filesystem, a missing file, a save cut short. The unix port of MicroPython, the one that runs on your computer, is built with the same LittleFS and FAT drivers as the boards, so a test can make a small filesystem of its own in RAM. It needs a block device class with the three methods, vfs.VfsLfs2.mkfs(dev) to format it, and vfs.mount(vfs.VfsLfs2(dev), "/ram") to mount it. A filesystem of 64 blocks of 512 bytes holds less than 32 KiB, so a test fills it within a few writes and then meets the same OSError with errno 28 that a full board raises.

The program below is such a class, written so that it also runs on CPython, with a wrapper that counts the calls a filesystem makes:

"""A block device in RAM, with the extended interface of vfs.AbstractBlockDev.

On the unix port or a board, a filesystem goes onto it with
    vfs.VfsLfs2.mkfs(dev)
    vfs.mount(vfs.VfsLfs2(dev), "/ram")
Here, on CPython, the calls a filesystem makes show what each method does.
"""


class RAMBlockDev:
    def __init__(self, block_size, block_count):
        self.block_size = block_size
        self.data = bytearray(block_size * block_count)

    def readblocks(self, block, buf, offset=0):
        start = block * self.block_size + offset
        buf[:] = self.data[start:start + len(buf)]

    def writeblocks(self, block, buf, offset=None):
        # Without an offset: whole blocks, erased by this method if needed (RAM needs none).
        # With an offset: only these bytes change, after the caller's ioctl(6, block).
        start = block * self.block_size + (offset or 0)
        self.data[start:start + len(buf)] = buf

    def ioctl(self, op, arg):
        if op == 4:  # number of blocks
            return len(self.data) // self.block_size
        if op == 5:  # bytes in a block
            return self.block_size
        if op in (1, 2, 3, 6):  # init, shutdown, sync, erase: nothing to do in RAM
            return 0
        return None


class Counting:
    """Wraps a block device and counts the calls a filesystem makes."""

    def __init__(self, dev):
        self.dev = dev
        self.reads = self.writes = self.erases = 0

    def readblocks(self, block, buf, offset=0):
        self.reads += 1
        return self.dev.readblocks(block, buf, offset)

    def writeblocks(self, block, buf, offset=None):
        self.writes += 1
        return self.dev.writeblocks(block, buf, offset)

    def ioctl(self, op, arg):
        if op == 6:
            self.erases += 1
        return self.dev.ioctl(op, arg)


dev = Counting(RAMBlockDev(512, 64))
count, size = dev.ioctl(4, 0), dev.ioctl(5, 0)
print("device: %d blocks of %d bytes = %d bytes" % (count, size, count * size))

dev.ioctl(6, 3)  # LittleFS erases a block before it programs it
dev.writeblocks(3, b"hello", 16)  # 5 bytes at offset 16 of block 3
buf = bytearray(5)
dev.readblocks(3, buf, 16)
print("block 3, offset 16:", bytes(buf), "at byte", 3 * size + 16)

dev.writeblocks(10, bytes(range(256)) * 2)  # FAT's way: one whole block
block = bytearray(size)
dev.readblocks(10, block)
print("block 10 ends with", block[-3:].hex(" "))
print("calls: %d reads, %d writes, %d erase" % (dev.reads, dev.writes, dev.erases))

device: 64 blocks of 512 bytes = 32768 bytes
block 3, offset 16: b'hello' at byte 1552
block 10 ends with fd fe ff
calls: 2 reads, 2 writes, 1 erase

Block 3 at offset 16 is byte 1,552 of the device, 3 × 512 + 16: every address on a block device is that one sum. The erase in this class does nothing, which the documentation allows for storage that needs no erase, but it also means the class accepts the write without an erase that the flash program showed going wrong.

Practise: a RAM block device that behaves like flash

Write RAMFlash, the stricter device: it reads as erased flash when it is new, programs by AND, erases on ioctl(6, n), refuses addresses past its end, and counts every erase of every block, a count the next lesson uses to measure wear. The sample tests run in your browser:

EINVAL = 22


class RAMFlash:
    """A block device in RAM that follows the rules of NOR flash."""

    def __init__(self, block_size, block_count):
        self.block_size = block_size
        self.block_count = block_count
        self.data = bytearray(block_size * block_count)
        self.erase_counts = [0] * block_count

    def readblocks(self, block, buf, offset=0):
        start = block * self.block_size + offset
        buf[:] = self.data[start:start + len(buf)]
        return 0

    def writeblocks(self, block, buf, offset=None):
        start = block * self.block_size + (offset or 0)
        self.data[start:start + len(buf)] = buf
        return 0

    def ioctl(self, op, arg):
        if op == 4:
            return self.block_count
        if op == 5:
            return self.block_size
        return None

from ramflash import RAMFlash


def test_new_device_is_erased():
    """a new device reads 0xff in every byte"""
    dev = RAMFlash(16, 4)
    buf = bytearray(64)
    assert dev.readblocks(0, buf) == 0
    assert buf == b"\xff" * 64


def test_sizes():
    """ioctl 4 and 5 give the block count and the block size"""
    dev = RAMFlash(512, 8)
    assert dev.ioctl(4, 0) == 8
    assert dev.ioctl(5, 0) == 512
    assert dev.ioctl(1, 0) == 0 and dev.ioctl(3, 0) == 0


def test_offset_write_lands_inside_its_block():
    """a write at an offset changes exactly those bytes of that block"""
    dev = RAMFlash(16, 4)
    assert dev.writeblocks(2, b"\x12\x34", 5) == 0
    whole = bytearray(64)
    dev.readblocks(0, whole)
    assert whole[37:39] == b"\x12\x34"
    assert whole.count(0xFF) == 62


def test_programming_only_clears_bits():
    """programming over old data gives the AND of the old and the new bytes"""
    dev = RAMFlash(16, 2)
    dev.writeblocks(0, b"21.5", 0)
    dev.writeblocks(0, b"19.0", 0)
    buf = bytearray(4)
    dev.readblocks(0, buf, 0)
    assert bytes(buf) == b"01.0"


def test_erase_sets_ones_and_counts():
    """ioctl 6 erases one block to 0xff and counts the erase"""
    dev = RAMFlash(16, 3)
    dev.writeblocks(1, b"\x00" * 16, 0)
    assert dev.ioctl(6, 1) == 0
    buf = bytearray(16)
    dev.readblocks(1, buf)
    assert buf == b"\xff" * 16
    assert dev.erase_counts == [0, 1, 0]
    assert dev.readblocks(3, bytearray(16)) == -22  # the device is still 3 blocks long


def test_whole_block_write_erases_first():
    """without an offset, the blocks are erased, then written"""
    dev = RAMFlash(8, 4)
    dev.writeblocks(1, b"\x0f" * 8, 0)
    assert dev.writeblocks(1, b"\xf0" * 16) == 0
    buf = bytearray(16)
    dev.readblocks(1, buf)
    assert buf == b"\xf0" * 16
    assert dev.erase_counts == [0, 1, 1, 0]


def test_out_of_range_changes_nothing():
    """a write past the end returns -22 and leaves the bytes as they were"""
    dev = RAMFlash(16, 2)
    assert dev.writeblocks(1, b"\x00" * 8, 12) == -22
    assert dev.writeblocks(2, b"\x00" * 16) == -22
    buf = bytearray(4)
    assert dev.readblocks(1, buf, 14) == -22
    whole = bytearray(32)
    dev.readblocks(0, whole)
    assert whole == b"\xff" * 32


def test_last_bytes_can_be_written():
    """a write that ends exactly at the last byte is in range"""
    dev = RAMFlash(16, 2)
    assert dev.writeblocks(1, b"\x00" * 4, 12) == 0
    buf = bytearray(4)
    assert dev.readblocks(1, buf, 12) == 0
    assert buf == b"\x00" * 4

Then check yourself on flash and block devices, the two filesystems, mounts and statvfs, and RAM devices for tests. The questions start with the idea you have answered worst so far, and each answer is explained:

Interview questions

Warm-up (fresher): What is the difference between LittleFS and FAT on a MicroPython board?

LittleFS was designed for flash. It writes changes to free blocks instead of over old data, commits a file’s changes only when the file is closed or synced, makes rename and remove atomic, and spreads erases over the blocks it allocates, so a power cut leaves it in its last committed state. FAT is the filesystem of SD cards and USB drives, which every computer reads, but MicroPython’s documentation warns that a power failure during a write can corrupt it, and it binds data to fixed places, so it cannot spread wear. The rp2 and esp32 ports format their flash with LittleFS v2. FAT fits an SD card that people read on a PC, with the files that change often kept on LittleFS.

This lesson leads to two longer questions in the module’s bank:

Key takeaways

• Flash is erased in whole sectors, 4,096 bytes on the Pico, and programming only clears bits, so a filesystem writes new data into erased space through a block device that reads, programs and erases.
• The rp2 and esp32 ports format their flash with LittleFS v2, which survives power cuts; FAT is for storage a PC must read, such as an SD card, and a power cut during a write can corrupt it.
• vfs.mount() lists the mounts, a mount point catches a path only when a slash or the end follows it, and new code uses vfs rather than os.mount().
• Free bytes are st[0] * st[3] of os.statvfs(), and a rewrite needs free blocks for the whole new copy, because LittleFS frees the old one only after the commit.
• A RAM block device with flash’s rules gives the unix port a filesystem for tests that is stricter than the board.