Native modules in C: natmods and user C modules

Every Modbus RTU frame ends in a CRCCRC: Cyclic redundancy check: a short value computed from a block of bytes by polynomial division, sent along with them, so that a receiver that computes it again can tell when bits were changed on the way. More in “Numbers, bytes and text on small chips”-16 that takes eight shift-and-test steps per byte: 2,048 steps for a 256-byte frame, each of them several bytecode instructions in Python. In C the same steps are a few machine instructions each, and MicroPython can load C as an .mpy file, a natmodnatmod: A native .mpy module: an .mpy file whose code is machine code compiled from C, built with MicroPython's dynruntime.mk and mpy_ld.py. A board imports it like any module, but only on the architecture it was built for., with no change to the firmware. The other way in is a user C moduleuser C module: C code compiled into a MicroPython firmware image, for example through c_module() in a manifest, and imported like a built-in module. It can call the port's whole C API, and every change needs a new firmware build.: the same C, compiled into the firmware itself.

This lesson weighs a natmod against its alternatives, builds one from plain C that is tested on your computer first, and ships it so that every board gets either the file for its processor or a Python version that gives the same result.

A natmod is an .mpy file whose code is compiled C

A hot function such as the CRC can run four ways on a board, and two of them are C:

• Plain Python: the function as you wrote it, as bytecode.
• Viper: Python with machine-word types, compiled to machine code by the board or by mpy-cross, as the lessons on emitters showed.
• A natmod: C compiled to an .mpy file of machine code, which a board imports like any module; the documentation says it needs no firmware build, and that it loads only on the architecture it was built for.
• A user C module: C compiled into the firmware itself, imported like a built-in module, with the whole C interface of the port; every change needs a new firmware.

The difference between the two C ways is where the code lives. A natmod is a file you can copy, delete and replace like any .mpy file. In exchange it can call only the functions of mp_fun_tablemp_fun_table: The fixed table of MicroPython's functions that a natmod calls through, instead of linking against the firmware itself. It is set when the firmware is built, so a natmod can use only the functions it lists., a table that the firmware hands it and that is fixed when the firmware is built, so no driver or system call of the port is in its reach. A user C module is part of the firmware build and can call anything the port has, at the cost of a firmware build per change. Answer three questions about your function:

In short: viper first for short integer code; a natmod for code that is best in C, on stock firmware; a user C module for C that needs the port’s own functions, in firmware you build.

A natmod is built from plain C, tested before it meets a board

Keep the algorithm in plain C, apart from MicroPython: here crc16_core.h, the CRC of the Modbus specification, started at 0xFFFF, with 0xA001 folded in after each shift that drops a 1. The natmod wraps it in a few lines, and the Makefile names the module, the source and the architecture. Step through one build for a Pico, then its import:

The tools come from the MicroPython source tree: py/dynruntime.mk holds the rules, and tools/mpy_ld.py, which needs the pyelftools package, turns the compiled ELF object into an .mpy file. The compiler is the board’s own cross-compiler, arm-none-eabi-gcc for ARCH=armv6m, xtensa-esp32-elf-gcc for xtensawin, and Clang works from 1.29.0. The file starts with the header of any armv6m .mpy file, 4d 06 13 1f, and the board checks it the same way.

Because the core is plain C, any computer can test it before a cross-compiler is involved. This program runs the core on four inputs and prints the frame as it goes on the wire:

/* test_core.c: the C core of the natmod, run on a computer before it goes near a board.
   It prints the CRC-16 of four inputs; crc16_ref.py prints the same four with the Python version. */
#include <stdint.h>
#include <stdio.h>

#include "crc16_core.h"

static void show(const char *label, const uint8_t *data, size_t len) {
    printf("%-26s 0x%04X\n", label, (unsigned)crc16_core(data, len));
}

int main(void) {
    const uint8_t check[] = {'1', '2', '3', '4', '5', '6', '7', '8', '9'};
    const uint8_t request[] = {0x01, 0x03, 0x00, 0x00, 0x00, 0x02};
    uint8_t ramp[256];
    for (int i = 0; i < 256; i++) {
        ramp[i] = (uint8_t)i;
    }
    show("\"123456789\"", check, sizeof check);
    show("01 03 00 00 00 02", request, sizeof request);
    show("bytes 0 to 255", ramp, sizeof ramp);
    show("no bytes", ramp, 0);

    uint16_t crc = crc16_core(request, sizeof request);
    printf("the request on the wire: 01 03 00 00 00 02 %02X %02X\n", (unsigned)(crc & 0xFF), (unsigned)(crc >> 8));
    return 0;
}

"123456789"                0x4B37
01 03 00 00 00 02          0x0BC4
bytes 0 to 255             0xDE6C
no bytes                   0xFFFF
the request on the wire: 01 03 00 00 00 02 C4 0B

/* crc16_core.h: the CRC-16 that ends every Modbus RTU frame, in plain C with nothing from MicroPython. */
#ifndef CRC16_CORE_H
#define CRC16_CORE_H

#include <stddef.h>
#include <stdint.h>

static inline uint16_t crc16_core(const uint8_t *data, size_t len) {
    uint16_t crc = 0xFFFF;
    for (size_t i = 0; i < len; i++) {
        crc ^= data[i];
        for (int bit = 0; bit < 8; bit++) {
            if (crc & 1) {
                crc = (uint16_t)((crc >> 1) ^ 0xA001);
            } else {
                crc >>= 1;
            }
        }
    }
    return crc;
}

#endif

The first line is the check value of CRC-16/MODBUS in the catalogue of CRC parameters, 0x4B37, so the core is the right CRC. The last line puts the CRC on the wire as the specification orders, the low byte first: the request ends in C4 0B. The same four inputs through the Python version give the same four numbers:

"""The four inputs of test_core.c, through the Python reference: the C core must print the same values."""
from crc16_ref import crc16

request = bytes([0x01, 0x03, 0x00, 0x00, 0x00, 0x02])
for label, data in (('"123456789"', b"123456789"), ("01 03 00 00 00 02", request),
                    ("bytes 0 to 255", bytes(range(256))), ("no bytes", b"")):
    print(f"{label:26} 0x{crc16(data):04X}")

crc = crc16(request)
print("the request on the wire:", (request + bytes([crc & 0xFF, crc >> 8])).hex(" ").upper())

"123456789"                0x4B37
01 03 00 00 00 02          0x0BC4
bytes 0 to 255             0xDE6C
no bytes                   0xFFFF
the request on the wire: 01 03 00 00 00 02 C4 0B

"""crc16 in Python: the reference the natmod must match, and the fallback for boards that have no natmod."""


def crc16(data):
    crc = 0xFFFF
    for byte in data:
        crc ^= byte
        for _ in range(8):
            if crc & 1:
                crc = (crc >> 1) ^ 0xA001
            else:
                crc >>= 1
    return crc

crc16_ref.py will do two jobs: it is the reference every build of the natmod is tested against, and the fallback a board imports when no natmod fits it.

Check yourself

The questions below cover natmods and their alternatives, one at a time.

The C wrapper, the import, and what a natmod cannot do Pro: opens with a Pro, Premium or Ultimate pass

One natmod per architecture, chosen at import, with a fallback Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Key takeaways

• A natmod is an .mpy file of machine code compiled from C: a board imports it with no firmware build, only on its own architecture, and it can call only the functions of mp_fun_table.
• A user C module is compiled into the firmware and can call the port’s whole C interface, but every change needs a new firmware build.
• A natmod is built with py/dynruntime.mk and tools/mpy_ld.py, one make ARCH=… per architecture.
• Keep the algorithm in plain C and test it on a computer against the Python reference and known values before any cross-compiler runs.