Libraries with mip and micropython-lib

pip install umqtt.simple puts nothing on a Pico W: pip installs into the Python on your computer, and the board has neither pip nor PyPI. MicroPython has an installer of its own, mipmip: MicroPython's package installer. It runs on boards with networking, inside mpremote on a computer and on the unix port, fetches a package from micropython-lib or from an address, and writes it into the first /lib folder of sys.path., and a collection of its own, micropython-libmicropython-lib: The MicroPython project's collection of packages for its builds, from logging and datetime to device drivers and MQTT clients. Its index at micropython.org serves each package as .mpy files and as source, and mip installs from it by default., whose index lists more than a hundred packages written for small chips, from logging to display drivers and MQTT clients.

This lesson follows one install request by request, on the board and from your computer: which files mip fetches, the folder it writes them to, what it installs when you name no version, and how a package of your own describes itself to mip in a package.jsonpackage.json: The file that describes a package to mip: urls lists its files as [destination, source] pairs, deps its dependencies as [name, version] pairs, and version the package's own version. mip reads it to install a package from an address..

mip copies a package from micropython-lib into /lib

On a Pico W with Wi-Fi connected, two lines install the MQTT client of micropython-lib: import mip, then mip.install("umqtt.simple"). The same installer exists in two more places, and the package documentation gives all three the same features:

Only the first needs a network on the board. mip ships in the firmware of boards with network support, so a Pico without Wi-Fi gets its packages through mpremotempremote: MicroPython's command-line tool for your computer: it connects to a board over USB serial and runs code, copies, lists and hashes files, mounts a local folder and opens the REPL, all through the raw REPL. More in “mpremote: run, copy, mount and sync”, which downloads on your computer and copies over USB. Step through one install on a Pico W, from the folder mip chooses to the last file it writes:

Two things decide every install. The first is the index, https://micropython.org/pi/v2 unless you pass index=: for a package name it answers with a package file that lists the files to copy, each with the first 8 hex digits of its SHA-256 hash, and mip then fetches each file by that hash. A package the requested one depends on is part of that list: umqtt.simple requires ssl in its manifest, so its package file names ssl.mpy too. If a file is already on the board with the same hash, mip prints Exists: and skips it.

The second is the target folder. mip takes the first entry of sys.path that ends in /lib and does not start with /rom, which is /lib on a Pico W. target="/data/lib" on the board, or --target=/data/lib with mpremote, puts the files elsewhere, but then that folder must be on sys.path too, or nothing can import them. This program follows those rules for umqtt.simple 1.8.0 on three builds, with the package files that the index serves for that version written into it, so it needs no network:

"""What mip.install() asks the index for, and where it writes each file.

This follows the rules of mip in micropython-lib (the copy that MicroPython 1.29.0 ships) and makes no
network request: the replies below are the package files the index serves for umqtt.simple 1.8.0.
"""

INDEX = "https://micropython.org/pi/v2"

REPLIES = {
    INDEX + "/package/6/umqtt.simple/1.8.0.json": {
        "v": 1,
        "version": "1.8.0",
        "hashes": [["ssl.mpy", "d53a3211"], ["umqtt/simple.mpy", "9240e244"]],
    },
    INDEX + "/package/py/umqtt.simple/1.8.0.json": {
        "v": 1,
        "version": "1.8.0",
        "hashes": [["ssl.py", "89a6e5ef"], ["umqtt/simple.py", "de2fbae7"]],
    },
}

BUILDS = {
    "Pico W": (4870, ["", ".frozen", "/lib"]),
    "Pico W with a ROMFS image": (4870, ["", ".frozen", "/rom", "/rom/lib", "/lib"]),
    "unix port": (774, ["", ".frozen", "~/.micropython/lib", "/usr/lib/micropython"]),
}


def default_target(path):
    """The first sys.path entry that ends in /lib, skipping the read-only /rom ones."""
    for entry in path:
        if entry.endswith("/lib") and not entry.startswith("/rom"):
            return entry
    return None


def install(build, name, version=None, mpy=True):
    mpy_number, path = BUILDS[build]
    target = default_target(path)
    version = version or "latest"
    kind = mpy_number & 0xFF if mpy else "py"
    url = f"{INDEX}/package/{kind}/{name}/{version}.json"
    print(f"{build}, sys.path {path}")
    print(f"  mip.install({name!r}, version={version!r}, mpy={mpy})")
    print(f"  Installing {name} ({version}) from {INDEX} to {target}")
    print(f"  GET {url}")
    reply = REPLIES.get(url)
    if reply is None:
        print("  Package not found:", url)
        print("  Package may be partially installed")
        return
    for dest, short_hash in reply["hashes"]:
        print(f"  GET {INDEX}/file/{short_hash[:2]}/{short_hash}")
        print(f"    Copying: {target}/{dest}")
    print("  Done")


install("Pico W", "umqtt.simple", "1.8.0")
install("Pico W with a ROMFS image", "umqtt.simple", "1.8.0")
install("unix port", "umqtt.simple", "1.8.0", mpy=False)
install("Pico W", "umqtt.simple", "1.9.0")

Pico W, sys.path ['', '.frozen', '/lib']
  mip.install('umqtt.simple', version='1.8.0', mpy=True)
  Installing umqtt.simple (1.8.0) from https://micropython.org/pi/v2 to /lib
  GET https://micropython.org/pi/v2/package/6/umqtt.simple/1.8.0.json
  GET https://micropython.org/pi/v2/file/d5/d53a3211
    Copying: /lib/ssl.mpy
  GET https://micropython.org/pi/v2/file/92/9240e244
    Copying: /lib/umqtt/simple.mpy
  Done
Pico W with a ROMFS image, sys.path ['', '.frozen', '/rom', '/rom/lib', '/lib']
  mip.install('umqtt.simple', version='1.8.0', mpy=True)
  Installing umqtt.simple (1.8.0) from https://micropython.org/pi/v2 to /lib
  GET https://micropython.org/pi/v2/package/6/umqtt.simple/1.8.0.json
  GET https://micropython.org/pi/v2/file/d5/d53a3211
    Copying: /lib/ssl.mpy
  GET https://micropython.org/pi/v2/file/92/9240e244
    Copying: /lib/umqtt/simple.mpy
  Done
unix port, sys.path ['', '.frozen', '~/.micropython/lib', '/usr/lib/micropython']
  mip.install('umqtt.simple', version='1.8.0', mpy=False)
  Installing umqtt.simple (1.8.0) from https://micropython.org/pi/v2 to ~/.micropython/lib
  GET https://micropython.org/pi/v2/package/py/umqtt.simple/1.8.0.json
  GET https://micropython.org/pi/v2/file/89/89a6e5ef
    Copying: ~/.micropython/lib/ssl.py
  GET https://micropython.org/pi/v2/file/de/de2fbae7
    Copying: ~/.micropython/lib/umqtt/simple.py
  Done
Pico W, sys.path ['', '.frozen', '/lib']
  mip.install('umqtt.simple', version='1.9.0', mpy=True)
  Installing umqtt.simple (1.9.0) from https://micropython.org/pi/v2 to /lib
  GET https://micropython.org/pi/v2/package/6/umqtt.simple/1.9.0.json
  Package not found: https://micropython.org/pi/v2/package/6/umqtt.simple/1.9.0.json
  Package may be partially installed

The first two runs write to the same place. A Pico W built with a ROMFSROMFS: MicroPython's read-only filesystem in a flash partition, mounted at /rom on boards whose firmware has one. Its .mpy files run in place from flash, like frozen code, but a new image is written with mpremote, without new firmware. More in “Frozen modules, manifests and ROMFS” image has /rom/lib on its path before /lib, and mip passes over it, because that image is read-only. The unix port writes to ~/.micropython/lib in your home folder, the first /lib folder of its default path; its sys.path shows that folder with the ~ replaced by the full path of your home folder. The last two runs, the source files and the version that does not exist, come back in the next section.

Without a version, mip installs the newest .mpy build

mip.install("umqtt.simple") with nothing else asks the index for latest.json, the newest version of the package, and for files compiled to .mpy.mpy file: A file of precompiled MicroPython bytecode, made with mpy-cross on a computer, that a board imports like a .py module without compiling it. It may also hold machine code for one architecture. More in “What MicroPython is and where it runs” for the board’s .mpy version. That version is the low byte of sys.implementation._mpy: 4870 & 0xFF is 6 on a Pico, and 6 on every release from 1.19 on. Two arguments change what you get, and the package documentation names the command-line forms that do the same:

The source build is useful when you want to read the code on the board or step through it; the .mpy build saves the board the work of compiling at every import. Look back at the third run of the program: with mpy=False the same package file lists ssl.py and umqtt/simple.py, with hashes of their own.

A version the index does not have stops the install at its first request. The last run asked for 1.9.0 and got Package not found, followed by mip’s general warning, Package may be partially installed. Here nothing was written, because the package file itself was missing; the warning matters when a later file fails, after earlier ones were already copied.

Name a version for anything a fleet of boards depends on. latest means whatever is newest on the day of the install, and the index keeps every version it has published, so a pin keeps working while new versions arrive. For a package fetched from a Git repository instead of the index, the version names a branch or a tag, and without one mip takes the repository’s default branch.

A package.json lists the files and dependencies of your own package

Your own driver is not in micropython-lib, but mip installs it the same way from a Git repository or any web server, as long as the package has a package.json beside its files. This one describes the station package of this module, three modules that read a Modbus sensor and send the readings by MQTT, with two dependencies, each pinned. The program checks the file the way mip will read it, then prints what one install would fetch and write:

"""Checks a package.json before mip reads it, then lists what one install would fetch and write.

mip itself checks nothing first: a bad entry stops it halfway, with part of the package copied. The URL rules
are mip's (micropython-lib, the copy MicroPython 1.29.0 ships); nothing here touches the network.
"""
import json

INDEX = "https://micropython.org/pi/v2"
PREFIXES = ("http://", "https://", "github:", "gitlab:", "codeberg:")
HOSTS = {
    "github:": "https://raw.githubusercontent.com/{org}/{repo}/{ref}/{path}",
    "gitlab:": "https://gitlab.com/{org}/{repo}/-/raw/{ref}/{path}",
    "codeberg:": "https://codeberg.org/api/v1/repos/{org}/{repo}/raw/{path}?ref={ref}",
}


def rewrite(url, version):
    """github:org/repo/path and the like become HTTPS addresses; the version names a branch or tag."""
    for prefix, form in HOSTS.items():
        if url.startswith(prefix):
            org, repo, *path = url[len(prefix):].split("/")
            return form.format(org=org, repo=repo, ref=version or "HEAD", path="/".join(path))
    return url


def problems(package):
    found = []
    if not isinstance(package.get("version"), str) or not package["version"]:
        found.append("version: missing; mip ignores it, but people and tools read it")
    for i, entry in enumerate(package.get("urls", [])):
        if not (isinstance(entry, list) and len(entry) == 2 and all(isinstance(x, str) for x in entry)):
            found.append(f"urls[{i}]: must be a pair [destination, source]")
            continue
        dest, source = entry
        if "\\" in dest or "\\" in source:
            found.append(f"urls[{i}]: {entry} uses backslashes; write forward slashes")
        if dest.startswith("/") or ".." in dest.split("/"):
            found.append(f"urls[{i}]: destination {dest!r} must be relative to the target folder")
    for i, dep in enumerate(package.get("deps", [])):
        if not (isinstance(dep, list) and len(dep) == 2 and all(isinstance(x, str) for x in dep)):
            found.append(f"deps[{i}]: {dep} must be a pair [name, version]")
    return found


def plan(package, source, version, target="/lib", mpy_number=4870):
    json_url = source + "/package.json"
    print(f"mip.install({source!r}, version={version!r}), target {target}")
    print("  GET", rewrite(json_url, version))
    base = json_url.rpartition("/")[0]
    for dest, url in package["urls"]:
        if not url.startswith(PREFIXES):
            url = base + "/" + url
        print("  GET", rewrite(url, version))
        print(f"    Copying: {target}/{dest}")
    for name, dep_version in package["deps"]:
        print(f"  then {name} {dep_version}: GET {INDEX}/package/{mpy_number & 0xFF}/{name}/{dep_version}.json")


with open("package.json") as f:
    station = json.load(f)
print("package.json:", problems(station) or "no problems found")
plan(station, "github:your-name/station", "v0.3.0")

broken = {
    "urls": [["station\\modbus.py", "station\\modbus.py"], ["/lib/station/upload.py", "station/upload.py"]],
    "deps": [["umqtt.simple"]],
}
print("broken package:")
for problem in problems(broken):
    print("  " + problem)

package.json: no problems found
mip.install('github:your-name/station', version='v0.3.0'), target /lib
  GET https://raw.githubusercontent.com/your-name/station/v0.3.0/package.json
  GET https://raw.githubusercontent.com/your-name/station/v0.3.0/station/__init__.py
    Copying: /lib/station/__init__.py
  GET https://raw.githubusercontent.com/your-name/station/v0.3.0/station/modbus.py
    Copying: /lib/station/modbus.py
  GET https://raw.githubusercontent.com/your-name/station/v0.3.0/station/upload.py
    Copying: /lib/station/upload.py
  then umqtt.simple 1.8.0: GET https://micropython.org/pi/v2/package/6/umqtt.simple/1.8.0.json
  then logging 0.6.2: GET https://micropython.org/pi/v2/package/6/logging/0.6.2.json
broken package:
  version: missing; mip ignores it, but people and tools read it
  urls[0]: ['station\\modbus.py', 'station\\modbus.py'] uses backslashes; write forward slashes
  urls[1]: destination '/lib/station/upload.py' must be relative to the target folder
  deps[0]: ['umqtt.simple'] must be a pair [name, version]

{
  "version": "0.3.0",
  "urls": [
    ["station/__init__.py", "station/__init__.py"],
    ["station/modbus.py", "station/modbus.py"],
    ["station/upload.py", "station/upload.py"]
  ],
  "deps": [
    ["umqtt.simple", "1.8.0"],
    ["logging", "0.6.2"]
  ]
}

The format that the package documentation gives has three keys. urls lists each file as a pair, the destination under the target folder first and the source second. A relative source is read from the folder that holds package.json; a full address, or a github:, gitlab: or codeberg: one, is not. deps lists [name, version] pairs, which mip installs after the files, each with its own version: from the index for a plain name, from that repository for a github: address. version is the package’s own version, for people and tools: mip itself does not read it.

The output shows the rewriting. github:your-name/station with the version v0.3.0 becomes https://raw.githubusercontent.com/your-name/station/v0.3.0/ followed by the path, so every file comes from the same tag. Notice also what arrives: your own files come as the .py files you published, because mip copies what each address holds and compiles nothing, while the dependencies from the index arrive as .mpy files.

To test a package before you publish it, install it from the folder on your computer: mpremote mip install ./station/package.json reads the file and its relative sources from your disk and copies them to the board. The broken package at the end of the output shows what the check catches before mip ever runs: no version, a backslash, a destination that starts with /, and a dependency without its version.

Practise: package a soil sensor

Write the package.json of a three-module sensor package that depends on umqtt.simple. The sample tests read your file as JSON and check each rule of this lesson, the forward slashes included:

{
  "version": "",
  "urls": [],
  "deps": []
}

import json
from pathlib import Path

MODULES = {"soil/__init__.py", "soil/probe.py", "soil/calibrate.py"}


def package():
    return json.loads(Path("package.json").read_text())


def test_is_an_object():
    """package.json is a JSON object with only urls, deps and version"""
    data = package()
    assert isinstance(data, dict), "the file must hold one JSON object"
    assert set(data) <= {"urls", "deps", "version"}, f"unexpected keys: {sorted(set(data) - {'urls', 'deps', 'version'})}"


def test_version():
    """the package's version is 1.0.0"""
    assert package().get("version") == "1.0.0"


def test_three_modules():
    """urls installs the three modules into a soil folder"""
    urls = package().get("urls", [])
    assert all(isinstance(u, list) and len(u) == 2 for u in urls), "each entry is a [destination, source] pair"
    assert {u[0] for u in urls} == MODULES, "the destinations are soil/__init__.py, soil/probe.py and soil/calibrate.py"
    assert len(urls) == 3, "list each module once"


def test_sources_are_relative():
    """each source is the module's own path in the repository, with forward slashes"""
    for dest, source in package().get("urls", []):
        assert "\\" not in dest + source, "write forward slashes, even on Windows"
        assert not source.startswith(("/", "http://", "https://", "github:")), "give the path relative to package.json"
        assert source == dest, f"{dest} comes from the file of the same path, not {source}"


def test_pinned_dependency():
    """deps pins umqtt.simple to 1.8.0"""
    deps = package().get("deps", [])
    assert all(isinstance(d, list) and len(d) == 2 for d in deps), "each dependency is a [name, version] pair"
    assert ["umqtt.simple", "1.8.0"] in deps

Then check yourself on mip’s install path, its versions and package.json. Feedback follows every answer, and the questions start with the idea you know least:

Interview questions

The first question is answered here; the next two are longer, and their first steps are on the module’s page of questions.

Key takeaways

• mip installs micropython-lib packages from the board, from mpremote over USB or on the unix port, by the same rules, and only the first needs a network on the board.
• mip writes into the first sys.path folder that ends in /lib and is not under /rom, which is /lib on a Pico, unless target= names another folder that is also on the path.
• Without a version mip installs the newest .mpy build for the board’s .mpy version; version= or @1.8.0 pins it, and mpy=False or --no-mpy installs the source.
• A package of your own needs a package.json: urls as [destination, source] pairs with forward slashes, deps as [name, version] pairs, and a version for people and tools.