From e9d9a1d639c84eb61ab786deedc485b9af4552ea Mon Sep 17 00:00:00 2001 From: Brad Barnett <127794626+bdbarnett@users.noreply.github.com> Date: Thu, 8 Oct 2026 00:47:17 -0500 Subject: [PATCH] build_mp.py --interpreter circuitpython: CircuitPython-compatible builds The same command now builds OmniPython (CircuitPython-compatible) firmware: CircuitPython with our C modules compiled in through its own USER_C_MODULES, for CircuitPython's ports and boards. - deps.lock pins CircuitPython at 11.0.0-alpha.1 (520805e12), whose core is MicroPython 1.29. It is cloned into deps/circuitpython, never linked to a sibling checkout, and each build fetches only the submodules its target needs with CircuitPython's tools/ci_fetch_deps.py. Its build tools go in deps/circuitpython-venv. - --modules names modules as for MicroPython; each module's micropython.mk goes in as USER_C_MODULES. A module without C is refused for now. - mpy-cross is built first, and make gets a short BUILD= link to our build dir (raspberrypi's link step overflows the 128 KB single-argument limit with long absolute object paths). - Output in builds/circuitpython/// with the same build record; the record logic is now one function both paths use. - modules.lock: pygraphics at b98288c, which builds as a user C module in CircuitPython (pygraphics#41; also brings #36 and #38). - A CI workflow builds CircuitPython's unix port with pygraphics from a bare clone and checks it draws. --- .github/workflows/circuitpython.yml | 59 ++++++++ README.md | 16 +- ROADMAP.md | 12 +- build_mp.py | 222 +++++++++++++++++++++++++--- deps.lock | 10 +- docs/build-plan.md | 60 +++++++- modules.lock | 2 +- 7 files changed, 355 insertions(+), 26 deletions(-) create mode 100644 .github/workflows/circuitpython.yml diff --git a/.github/workflows/circuitpython.yml b/.github/workflows/circuitpython.yml new file mode 100644 index 00000000..eca86376 --- /dev/null +++ b/.github/workflows/circuitpython.yml @@ -0,0 +1,59 @@ +# The CircuitPython-compatible path from a bare clone, the way a user builds +# it: build_mp.py fetches CircuitPython at its deps.lock tag and pygraphics at +# its modules.lock commit, builds the unix port with pygraphics as a user C +# module, and the result has to import it, draw, and read the pixels back. +name: circuitpython + +on: + push: + branches: [main] + paths: + - 'build_mp.py' + - 'deps.lock' + - 'modules.lock' + - '.github/workflows/circuitpython.yml' + pull_request: + paths: + - 'build_mp.py' + - 'deps.lock' + - 'modules.lock' + - '.github/workflows/circuitpython.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + unix: + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v7 + + - name: Install the unix port's build dependencies + run: | + sudo apt-get update + sudo apt-get install -y libffi-dev pkg-config + + - name: Build CircuitPython's unix port with pygraphics + run: python3 build_mp.py --interpreter circuitpython --port unix --modules pygraphics + + - name: pygraphics imports, draws and reads back + run: | + cat > check.py <<'PY' + import sys + import pygraphics + from pygraphics import FrameBuffer, RGB565 + fb = FrameBuffer(bytearray(32 * 32 * 2), 32, 32, RGB565) + fb.fill(0) + fb.fill_rect(4, 4, 8, 8, 0xF800) + fb.circle(20, 20, 6, 0x07E0, True) + fb.line(0, 31, 31, 31, 0x001F) + got = (fb.pixel(5, 5), fb.pixel(20, 20), fb.pixel(16, 31), fb.pixel(0, 0)) + print(sys.implementation.name, pygraphics.implementation(), [hex(p) for p in got]) + assert sys.implementation.name == "circuitpython" + assert pygraphics.implementation() == "native_cmod" + assert pygraphics.capabilities()["framebuf"] == "native" + assert got == (0xF800, 0x07E0, 0x001F, 0), got + PY + ./builds/circuitpython/unix/coverage/micropython check.py diff --git a/README.md b/README.md index 06bcb745..0d259f98 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,19 @@ On Windows you can build the windows port natively from PowerShell, with MSYS2 is installed: [On Windows](docs/newcomers.md#on-windows) has the steps. Everything else builds from WSL. +The same command builds OmniPython (CircuitPython-compatible) firmware, which +is CircuitPython with our C modules compiled in, for CircuitPython's own +ports and boards: + +```bash +./build_mp.py --interpreter circuitpython --port raspberrypi \ + --board adafruit_feather_rp2040 --modules pygraphics +``` + +It takes C modules only, so far; the +[build plan](docs/build-plan.md#circuitpython-compatible-builds) has how it +works. + New here? The [newcomer's guide](docs/newcomers.md) explains the model; the [build plan](docs/build-plan.md) has the layout, the rules and the traps. @@ -38,7 +51,8 @@ New here? The [newcomer's guide](docs/newcomers.md) explains the model; the the port's own `make`. - `UPSTREAM` — the MicroPython release everything applies to (v1.29.0). - `modules.lock`, `deps.lock` — the commits and versions `build_mp.py` - fetches when a module or a toolchain isn't there. Beside sibling + fetches when a module or a toolchain isn't there. `deps.lock` also pins + CircuitPython, for `--interpreter circuitpython`. Beside sibling checkouts, both are links to those checkouts instead. - `modules/` — `manifest.py` (the manifest every build freezes), `all/` (every module), `castif/` (the ESP32-P4 cast) and `jpegio/` (the JPEG diff --git a/ROADMAP.md b/ROADMAP.md index e068fd6c..563614cd 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,11 @@ # Roadmap -No larger changes are planned for micropython-pydevices right now. Work here follows bug -reports and requests, so if something is broken or missing for you, open an -[issue](https://github.com/PyDevices/micropython-pydevices/issues). +Work here mostly follows bug reports and requests, so if something is broken +or missing for you, open an +[issue](https://github.com/PyDevices/micropython-pydevices/issues). Planned: + +- **More modules in CircuitPython-compatible builds.** audiodsp, displayif's + desktop display and LVGL next, then the ESP32 boards, which build + CircuitPython against ESP-IDF 6. +- **Freezing our Python packages into CircuitPython-compatible builds**, + through CircuitPython's own frozen-module mechanism. diff --git a/build_mp.py b/build_mp.py index bd5081fe..2512c990 100755 --- a/build_mp.py +++ b/build_mp.py @@ -10,6 +10,13 @@ by short name, anything else by full path; "all" is every module. Arguments it doesn't recognise go straight to make. + ./build_mp.py --interpreter circuitpython --port raspberrypi \\ + --board adafruit_feather_rp2040 --modules pygraphics + +--interpreter circuitpython builds CircuitPython-compatible firmware instead: +CircuitPython at its deps.lock pin, our C modules compiled in through its +USER_C_MODULES. Ports and boards are then CircuitPython's own. + This script names no version and no setting. Pins live in modules.lock, deps.lock and UPSTREAM; esp32 settings live in sdkconfig fragments found by convention. See docs/build-plan.md. @@ -192,6 +199,10 @@ def dep_version(name, path): if name == "emsdk": f = path / "upstream" / "emscripten" / "emscripten-version.txt" return f.read_text().strip().strip('"') if f.exists() else "" + if name == "circuitpython": + # The tag the checkout is exactly at; CircuitPython's own build reads + # its version the same way, from git describe. + return git_out(path, "describe", "--tags", "--exact-match") if name == "SDL2": h = path / "x86_64-w64-mingw32" / "include" / "SDL2" / "SDL_version.h" if not h.exists(): @@ -499,10 +510,201 @@ def rp2_autosize(log_text): return storage if storage >= align else None +# ---- CircuitPython: the second interpreter --------------------------------- + +CP = DEPS / "circuitpython" +CP_VENV = DEPS / "circuitpython-venv" + + +def lock_beside(path): + """One build at a time in a checkout: held until this process exits.""" + lock = open(path.parent / f".{path.name}-build.lock", "w") + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + say(f"waiting for another build to release {lock.name}") + fcntl.flock(lock, fcntl.LOCK_EX) + return lock + + +def ensure_circuitpython(): + """deps/circuitpython at its deps.lock tag, and a venv with its build tools. + + Always a checkout of our own, never a link to a sibling circuitpython/: a + sibling is at whatever version its owner needs, and may carry other + repositories' patches. Only the submodules a target needs are fetched, + by CircuitPython's own tools/ci_fetch_deps.py, at build time.""" + url, version = {row[0]: row[1:] for row in read_lock("deps.lock")}["circuitpython"] + if not CP.exists(): + DEPS.mkdir(exist_ok=True) + clone_at(url, version, CP) + have = dep_version("circuitpython", CP) + if have != version: + die(f"deps/circuitpython is {have or 'not at a tag'}, but deps.lock wants {version}") + # CircuitPython's make runs python3 from PATH and needs its dev requirements + # (cascadetoml, jinja2, typer, ...). The venv is redone when they change. + reqs = CP / "requirements-dev.txt" + stamp = CP_VENV / "requirements-dev.txt" + if not (stamp.exists() and stamp.read_text() == reqs.read_text()): + say(f"setting up {CP_VENV.relative_to(REPO)} with CircuitPython's requirements-dev.txt") + run([sys.executable, "-m", "venv", "--clear", str(CP_VENV)]) + run([str(CP_VENV / "bin" / "pip"), "install", "-q", "-r", str(reqs)]) + shutil.copy(reqs, stamp) + return CP + + +def cp_ports(): + # zephyr-cp builds through west and never reads USER_C_MODULES. + return sorted(p.name for p in (CP / "ports").iterdir() if (p / "Makefile").exists() and p.name != "zephyr-cp") + + +def cp_boards(port): + d = CP / "ports" / port / "boards" + return sorted(p.parent.name for p in d.glob("*/mpconfigboard.mk")) if d.is_dir() else [] + + +def cp_variants(port): + d = CP / "ports" / port / "variants" + return sorted(p.name for p in d.iterdir() if (p / "mpconfigvariant.mk").exists()) if d.is_dir() else [] + + +def cp_user_c_modules(module_dirs): + """The directory holding each module's micropython.mk, for USER_C_MODULES. + + CircuitPython's py/py.mk carries MicroPython 1.29's USER_C_MODULES block, + so the micropython.mk our MicroPython builds use is the one it reads. + Python is not frozen here yet: CircuitPython freezes through its own + FROZEN_MPY_DIRS, not manifest.py, so a module without C is refused rather + than silently left out.""" + dirs = [] + for d in module_dirs: + real = d.resolve() + found = next((c for c in (real, real / "code") if (c / "micropython.mk").is_file()), None) + if not found: + die(f"module '{d.name}' has no micropython.mk; CircuitPython builds take C modules only, so far") + dirs.append(found) + return dirs + + +def build_dir_for(build, want, clean): + """The build dir for a target, emptied first if it was built with + something other than `want` (another module set, flash size, ...).""" + rec_path = build / "pydevices-build.json" + if clean and build.exists(): + shutil.rmtree(build) + say(f"--clean: removed {build}") + if rec_path.exists(): + had = json.loads(rec_path.read_text()) + if {k: had.get(k) for k in want} != want: + shutil.rmtree(build) + flash = f" (flash {had.get('flash')})" if "flash" in want else "" + say(f"{build} was built with {had.get('modules')!r} {had.get('module_set') or ''}{flash}; " + f"wiped, building {want['modules']!r} {want['module_set']}") + elif build.exists() and any(build.iterdir()): + # Something built here without a record: nobody can say with what. + shutil.rmtree(build) + say(f"{build} has no build record; wiped") + build.mkdir(parents=True, exist_ok=True) + # Written before the build, so a build that fails still says what it was + # built with, and the next one with a different set wipes it. + rec_path.write_text(json.dumps(dict(want, complete=False), indent=1) + "\n") + return rec_path + + +def build_circuitpython(args, make_extra, ws): + if os.name == "nt": + die("--interpreter circuitpython builds from Linux or WSL") + if args.flash: + die("--flash is for MicroPython's esp32 port") + ensure_modules(ws) + cp = ensure_circuitpython() + lock = lock_beside(cp) # noqa: F841 -- held until exit + + port = args.port or choose("CircuitPython port:", cp_ports()) + if port not in cp_ports(): + die(f"no CircuitPython port '{port}' (have: {', '.join(cp_ports())})") + board, variant = args.board, args.variant + if cp_boards(port): + board = board or choose("Board:", cp_boards(port)) + if board not in cp_boards(port): + die(f"no board '{board}' for CircuitPython's {port}") + if variant: + die("CircuitPython boards have no variants; leave out --variant") + else: + if board: + die(f"{port} has no boards; leave out --board") + if variant is None and not args.port: + variant = choose("Variant:", cp_variants(port), allow_none=True) + if variant and variant not in cp_variants(port): + die(f"no variant '{variant}' (have: {', '.join(cp_variants(port))})") + + spec = args.modules + if spec is None: + if not sys.stdin.isatty(): + die("--modules: not given, and there is no terminal to ask on") + say("Modules: " + ", ".join(module_choices())) + spec = input('Which (comma list; full paths for others; empty for none)? ').strip() + spec, module_dirs = resolve_modules(spec) + user_c = cp_user_c_modules(module_dirs) + + # The unix port's own default variant is coverage. + build = OUT_DIR / "circuitpython" / port / (board or variant or "coverage") + module_set = sorted(d.name if d.parent == MODULES_DIR else str(d) for d in module_dirs) + want = {"interpreter": "circuitpython", "port": port, "board": board, "variant": variant, + "modules": spec, "module_set": module_set} + rec_path = build_dir_for(build, want, args.clean) + + env = {k: v for k, v in os.environ.items() + if k not in ("USER_C_MODULES", "FROZEN_MANIFEST", "BUILD", "BOARD", "VARIANT", "VARIANT_DIR", "PYTHONPATH")} + env["PATH"] = os.pathsep.join([str(CP_VENV / "bin"), env.get("PATH", "")]) + env["VIRTUAL_ENV"] = str(CP_VENV) + # Only the submodules this target needs, the way CircuitPython's CI does: + # a board name fetches its port's, "tests" what the unix port builds with. + run([str(CP_VENV / "bin" / "python"), "tools/ci_fetch_deps.py", board or "tests"], cwd=cp, env=env, + stdout=subprocess.DEVNULL) + # mpy-cross first and on its own: a port's sub-make for it would inherit + # BUILD= from our command line (the same trap as micropython#19667). + run(["make", "-C", str(cp / "mpy-cross"), "-j", JOBS], env=env, stdout=subprocess.DEVNULL) + + # make is given a short relative BUILD, a link in the port dir to ours: the + # raspberrypi link rule echoes every object path in one shell argument, + # and with our absolute build path in front of each that passes Linux's + # 128 KB limit on a single argument ("Argument list too long"). + port_dir = cp / "ports" / port + short = port_dir / f"build-{board or variant or 'coverage'}" + if short.is_symlink() or short.is_file(): + short.unlink() + elif short.is_dir(): + shutil.rmtree(short) + short.symlink_to(build, target_is_directory=True) + make = ["make", "-C", str(port_dir), "-j", JOBS, f"BUILD={short.name}", + "USER_C_MODULES=" + " ".join(str(d) for d in user_c)] + if board: + make.append(f"BOARD={board}") + elif variant: + make.append(f"VARIANT={variant}") + make += make_extra + say(f"CircuitPython {dep_version('circuitpython', cp)} {port} {board or variant or 'coverage'}; " + f"USER_C_MODULES: {' '.join(str(d) for d in user_c) or '(none)'}") + rc = subprocess.run(make, env=env).returncode + if rc != 0: + die(f"the build failed (make exit {rc})") + rec = dict(want, complete=True) + rec["circuitpython"] = git_out(cp, "describe", "--tags", "--always", "--dirty") + rec["module_revisions"] = record(module_dirs) + rec_path.write_text(json.dumps(rec, indent=1) + "\n") + say(f"\nBuilt into {build}") + for name in ("firmware.uf2", "firmware.bin", "firmware.elf", "micropython"): + if (build / name).exists(): + say(f" {build / name}") + + # ---- the build ------------------------------------------------------------- def main(): ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0], allow_abbrev=False) + ap.add_argument("--interpreter", choices=("micropython", "circuitpython"), default="micropython", + help="circuitpython: CircuitPython-compatible firmware, from CircuitPython's own ports and boards") ap.add_argument("--port") ap.add_argument("--board") ap.add_argument("--variant") @@ -516,6 +718,8 @@ def main(): if msys2: say(f"using MSYS2 at {msys2}") ws = workspace() + if args.interpreter == "circuitpython": + return build_circuitpython(args, make_extra, ws) ensure_micropython(ws) ensure_modules(ws) mp = MP.resolve() @@ -592,23 +796,7 @@ def main(): module_set = sorted(d.name if d.parent == MODULES_DIR else str(d) for d in module_dirs) want = {"port": port, "board": board, "variant": variant, "flash": flash, "modules": spec, "module_set": module_set} - if args.clean and build.exists(): - shutil.rmtree(build) - say(f"--clean: removed {build}") - if rec_path.exists(): - had = json.loads(rec_path.read_text()) - if {k: had.get(k) for k in want} != want: - shutil.rmtree(build) - say(f"{build} was built with {had.get('modules')!r} {had.get('module_set') or ''} (flash {had.get('flash')}); " - f"wiped, building {spec!r} {module_set}") - elif build.exists() and any(build.iterdir()): - # Something built here without a record: nobody can say with what. - shutil.rmtree(build) - say(f"{build} has no build record; wiped") - build.mkdir(parents=True, exist_ok=True) - # Written before the build, so a build that fails still says what it was - # built with, and the next one with a different set wipes it. - rec_path.write_text(json.dumps(dict(want, complete=False), indent=1) + "\n") + build_dir_for(build, want, args.clean) env = dict(os.environ) for leak in ("USER_C_MODULES", "FROZEN_MANIFEST", "BUILD", "BOARD", "VARIANT", "BOARD_DIR", "VARIANT_DIR"): diff --git a/deps.lock b/deps.lock index 9ac1c674..2568add7 100644 --- a/deps.lock +++ b/deps.lock @@ -1,8 +1,14 @@ # Toolchain pins: name, URL, version. A build fetches and checks only the one # its port uses: esp-idf for esp32, emsdk for webassembly, SDL2 for windows -# when displayif is selected. In a workspace deps/ may be a symlink to a -# shared checkout, and a build refuses if it isn't the version below. +# when displayif is selected, circuitpython for --interpreter circuitpython. +# In a workspace deps/ may be a symlink to a shared checkout, and a +# build refuses if it isn't the version below. # The ESP-IDF version is our choice, not upstream's recommendation. esp-idf https://github.com/espressif/esp-idf.git v5.5.4 emsdk https://github.com/emscripten-core/emsdk.git 6.0.1 SDL2 https://github.com/libsdl-org/SDL/releases/download/release-2.30.10/SDL2-devel-2.30.10-mingw.tar.gz 2.30.10 +# CircuitPython is the second interpreter, and the one pin that is a source +# tree rather than a toolchain. Always our own checkout in deps/, never a link +# to a sibling. 11.0.0-alpha.1 is commit 520805e12, the first release of the +# 11.0 line, whose core is MicroPython 1.29, like UPSTREAM. +circuitpython https://github.com/adafruit/circuitpython.git 11.0.0-alpha.1 diff --git a/docs/build-plan.md b/docs/build-plan.md index 4c5f0e50..2bdb4476 100644 --- a/docs/build-plan.md +++ b/docs/build-plan.md @@ -211,6 +211,62 @@ that wants one (the wasm one) requires it by name. tidier, not possible; it already works without it. It isn't planned as an upstream PR. +## CircuitPython-compatible builds + +The same command builds OmniPython (CircuitPython-compatible) firmware: +CircuitPython, with our C modules compiled in. + +```bash +./build_mp.py --interpreter circuitpython --port raspberrypi \ + --board adafruit_feather_rp2040 --modules pygraphics +./build_mp.py --interpreter circuitpython --port unix --modules pygraphics +``` + +Ports and boards are CircuitPython's own (`raspberrypi`, `atmel-samd`, `unix` +and the rest; boards by their directory names under `ports//boards/`). +Output lands in `builds/circuitpython///`, with the +same build record; unix builds CircuitPython's default variant, `coverage`, +unless you name another. + +It is a flag rather than a second command because everything around the +interpreter is shared: the module names and `modules.lock`, the workspace +links, the output layout and the build record. What differs is one function: + +- **The checkout.** CircuitPython is pinned in `deps.lock` like a toolchain + (`11.0.0-alpha.1`, whose core is MicroPython 1.29) and cloned into + `deps/circuitpython`, always our own clone, never a link to a sibling + checkout. A build fetches only the submodules its target needs, with + CircuitPython's own `tools/ci_fetch_deps.py` (a board's name, or `tests` + for unix), and its build tools go in `deps/circuitpython-venv` from its + `requirements-dev.txt`. +- **The modules go in as `USER_C_MODULES`**, not a manifest. From the 1.29 + merge on, CircuitPython's `py/py.mk` carries MicroPython 1.29's + `USER_C_MODULES` block unchanged, so a module's `micropython.mk` is the one + our MicroPython builds read. CircuitPython freezes Python its own way + (`FROZEN_MPY_DIRS`), so a module with no `micropython.mk` is refused for + now rather than left out quietly. +- **mpy-cross is built first, on its own**, for the same reason as on + MicroPython (the trap below). +- **`BUILD=` is a short link.** make is given `BUILD=build-`, a link in + the port directory to our build dir: raspberrypi's link step echoes every + object path in one shell argument, and with a long absolute path in front + of each it passes Linux's 128 KB limit on one argument. + +The banner is CircuitPython's, unchanged: `Adafruit CircuitPython +11.0.0-alpha.1 on ; with `. A module that has to +behave differently here can test `CIRCUITPY`, which every CircuitPython port +sets, unix included; pygraphics also passes its own define from its +`micropython.mk`, so it knows it came in as a user C module and not through +CircuitPython's shared-bindings. + +Two things a module's C has to absorb on CircuitPython's MCU ports: they +append `-Werror` warnings (`sign-compare`, `float-equal`, `cast-align` on +atmel-samd) after the module's flags, so a `-Wno-` meant for them goes on +each object, where it lands last; and atmel-samd has no C heap, so +`malloc()` doesn't link there (use `m_malloc()`). + +Builds from Linux or WSL only. + ## Traps we already know **`BUILD=` and the mpy-cross sub-make.** Passed on its own, `BUILD=` reaches @@ -249,5 +305,5 @@ Deferred, because the build script doesn't need them: sensor and H.264 replace cameraif (and castif's encoder) stays open. esp-vision's `tflite` and image stack are in as modules; where `sensor` stands is in [esp-vision.md](esp-vision.md). -- **CircuitPython.** It doesn't read our manifests, and none of this reaches - it yet. +- **CircuitPython-compatible builds** take C modules only, so far; see + [CircuitPython-compatible builds](#circuitpython-compatible-builds). diff --git a/modules.lock b/modules.lock index f442123d..f44fee75 100644 --- a/modules.lock +++ b/modules.lock @@ -12,6 +12,6 @@ lvgl-micropython https://github.com/PyDevices/lvgl-micropython.git 2c74ef56a9 palettes https://github.com/PyDevices/palettes.git 6f0870f0c8a0b48c56d2150b3f800bbbee7761e1 pdwidgets https://github.com/PyDevices/pdwidgets.git 306c1e20e0c3c63066dab6d10155fcf165a6ba13 pydevices https://github.com/PyDevices/pydevices.git ab3f7743aa266bd12a758e1780f66ce1948f6851 -pygraphics https://github.com/PyDevices/pygraphics.git b4d4c49cc2a22cdd2632f2eea2593b4cc4df4db1 +pygraphics https://github.com/PyDevices/pygraphics.git b98288c2e4cfcac10482ff71bd9f3b243c2f03c1 ulab https://github.com/v923z/micropython-ulab.git 1d3ddd8f5228e2c7335dac71f4846e6abb0ec9f9 usbif https://github.com/PyDevices/usbif.git 95bcb3b29933009299abdf59d6b81cf742e95778