diff --git a/src/field-notes/2026-06-24-hardened-boot-imx93.mdx b/src/field-notes/2026-06-24-hardened-boot-imx93.mdx index c7f53897..525049b7 100644 --- a/src/field-notes/2026-06-24-hardened-boot-imx93.mdx +++ b/src/field-notes/2026-06-24-hardened-boot-imx93.mdx @@ -8,7 +8,7 @@ category: 'Security & Compliance' featured: false # Featured selection is Kristen/Bill's call image: /img/field-notes/2026-06-24-hardened-boot-imx93-thumb.png image_source: local:hardened-boot-imx93.jpg -tested_against: 'Avocado 0.41.2, avocado-imx93-frdm (AHAB Secure Boot + OP-TEE fTPM), avocado-qemuarm64 (fTPM via QEMU)' +tested_against: 'Avocado 0.41.2, avocado-qemuarm64 (fTPM via QEMU). LUKS2 /var and phase-2 OP-TEE fTPM PCR-7 sealing both validated on avocado-imx93-frdm hardware (2026-08-12 and 2026-08-19, meta-avocado #279). dm-verity and module signing NOT yet validated on that hardware; IMA/EVM is design only and not implemented in the layer.' poster: jtia lift_for_blog: 'CRA Annex I and IEC 62443-4-2 both require demonstrable data-at-rest protection and integrity; this is the boot-chain half a compliance mapping builds on.' promote_to_track: 'Secure-boot / storage-encryption reference candidate once the i.MX93 hardware pass is fully signed off.' @@ -18,7 +18,7 @@ import TestStatus from '@site/src/components/TestStatus' import PullQuote from '@site/src/components/PullQuote' all zeros +``` + +So the order is burn `SRK_HASH` while still open, _then_ check for zero events, +_then_ close. A plan that validates zero events before the burn cannot be +satisfied - which is what this note used to imply. + +### The gap in that log (2026-08-14) + +Read the pass above again and one line gives it away: `Loading Ramdisk`. AHAB +covered the kernel and the device tree. The initramfs was loaded separately and +handed to `booti` as `argv[1]`, which `booti` never authenticates - and on this +image that initrd is what derives the LUKS key and opens `/var`. The single +unsigned component stood directly in front of the storage control. + +The obvious fix is to add it as a second `--data` payload, and it fails +quietly. `container_get_image_dst()` returns the first exec image and the first +data image and stops, so a second data payload is authenticated, placed, and +then never located; `booti` still reads `argv[1]`. Keeping the separate load +just to obtain its size is worse: `${filesize}` comes from the unauthenticated +copy on disk, so a larger file leaves unverified bytes past the authenticated +region - and appending a cpio archive is the documented way to overlay files in +an initrd. + +Bundling it into the kernel closes it by construction. The initramfs rides +inside the exec payload the container already signs, and `booti` is handed no +ramdisk at all: + +```text +Booting A +176028672 bytes read # os_cntr_signed.bin - kernel, dtb and initramfs +Authenticate OS container at 0xa8000000 +Starting kernel ... +[ OK ] Found device /dev/mapper/var. + +# findmnt -no SOURCE,FSTYPE /var +/dev/mapper/var btrfs +``` + +No `Loading Ramdisk` line. That absence is the whole result. + +**The cost is size, and it lands in three places.** Bundling embeds the archive +uncompressed - the build decompresses the `.cpio.zst` and the kernel stores what +it is handed - so the kernel goes from 33 MB to about 200 MB. Re-compressing is +not reachable from a config fragment, because the compression choice depends on +`INITRAMFS_SOURCE`, which only reaches the kernel as a make argument during the +bundling step and so is unset when the config is written. + +That forces the boot partitions from 128 to 256 MiB, `recovery` included since +it holds a boot filesystem too. It also breaks the memory map, twice. The +device tree ended up inside the kernel's unpack range (`FDT image overlaps OS +image`). Then the container, moved out of the way, landed across a hole: this +board's DRAM is not contiguous - `bdinfo` reports `0x80000000+0x16000000` and +`0x98000000+0x68000000` with 32 MiB missing between them - and a load spanning +it is refused with `Reading file would overwrite reserved memory`, which names +neither the hole nor the size. The container now stages in the second bank, +above the carve-out Linux later claims as `ele-reserved`. + +The guard that was supposed to catch this compared two addresses for equality, +which is true of any layout that fails this way and false of every layout that +did. It now measures the built kernel and refuses when it reaches either the +device tree or the container. It earned that immediately: turning on encrypted +`/var` grew the initramfs and pushed the kernel past a device tree that had +already been moved once, and the build stopped with both addresses named rather +than the board stopping at a prompt. + +Worth being plain about two limits. The initramfs is loaded outside the container +and passed to `booti` as a separate argument, so it is **not** authenticated - +and on this image it is what unlocks LUKS `/var`. And nothing here is enforced +yet: the part is open, so a signature failure is logged rather than fatal. +Enforcement is what closing the device buys, and closing is the step that can +leave a board unable to boot anything you hold keys for. + +### The hardware pass (2026-08-12) + +The board pass ran on an FRDM-IMX93 against the wrynose BSP, and it closed the LUKS2 row. Three defects had to be fixed first, and every one of them was invisible on the proxy because `avocado-qemuarm64` carries a machine override that `avocado-imx93-frdm` did not: + +- **`CONFIG_DM_CRYPT` was silently `=m`.** The fragment asks for `=y`, but kconfig caps a tristate at its weakest parent and the i.MX defconfig left both `BLK_DEV_DM` and `TRUSTED_KEYS` at `=m`. The initramfs ships `cryptsetup` with no module and nothing that loads one, so the unlock had no dm-crypt to call. The task check greps the fragment _text_, so it passed while the built kernel disagreed - verify against the produced `.config`, never the fragment. +- **`AVOCADO_VAR_PART_DEV` was unset.** Both qemu machines point `/var` at `/dev/mapper/var` when `encrypted-var` is on; imx93 fell through to the base-files raw-PARTUUID default, so the container would be created and opened in the initrd while fstab mounted `/var` from the raw partition underneath it. +- **The key provider read a secret that does not exist.** The shared `var-key.sh` reads `/var/private/var-key-secret` - on the volume being unlocked, not mounted when the provider runs in the initramfs, and created by nothing in the tree. The Argon2id path had therefore never once succeeded on this board. + +What the pass showed, on the board: + +```text +# cryptsetup status var +/dev/mapper/var is active and is in use. + type: LUKS2 cipher: aes-xts-plain64 keysize: 512 [bits] + device: /dev/mmcblk1p8 + +# findmnt /var +/var /dev/mapper/var btrfs rw,relatime,ssd,space_cache=v2 + +# dmsetup targets | grep crypt +crypt v1.28.0 +``` + +First boot ran the whole path - open, no filesystem found, filesystem created at 29.01 GiB, `Finished LUKS2 /var unlock and first-boot format` - and the partition expanded 112M to 29G. A second boot then exercised the **unlock** path rather than first-boot format: `/var` reopened with the same btrfs UUID across a reboot confirmed by a changed `boot_id`. That is the reboot survival the proxy structurally cannot show, though only for the Argon2id slot. + +**Phase-2 TPM2 remains unshown, and not for want of running it.** There is no TPM on this board to seal to. The FRDM-IMX93 has no discrete TPM (confirmed against UM12181 and, for its i.MX95 siblings, the schematic BOMs), and the i.MX9 route to a TPM 2.0 interface is a firmware TPM on OP-TEE, which `optee-ftpm-init` currently builds for `avocado-qemuarm64` only. Nothing seals to PCR 7 on an i.MX93 today. The board does have what that route needs - OP-TEE enabled, 32 GB eMMC so RPMB exists, and a `recovery` partition for the TEE store - so this is a porting task rather than a hardware limitation, and it needs no AHAB work. Worth knowing when reading NXP material: on i.MX, "TPM" in a datasheet or device tree usually means the **Timer/PWM Module** peripheral, not a trusted platform module. + +The intent still holds: both paths run the same recipes, kernel fragments and initramfs logic, and both seal to the same OP-TEE fTPM (`MACHINE_FEATURES += " optee-ftpm"` on `avocado-qemuarm64`, per `meta-avocado-qemu/conf/machine/avocado-qemuarm64.conf`) - what differs is the platform underneath it, QEMU `virt` against i.MX93 silicon. So a cheap proxy gate should guard the expensive hardware path. It is an intent rather than a seal until the cases merge and the board pass runs. + +### The fTPM hardware pass (2026-08-19) + +The porting task named above - `optee-ftpm` for imx93, gated behind `kas/feature/ftpm.yml` - +landed and ran on the board. Built `avocado-imx93-frdm` with +`kas/feature/encrypted-var.yml:kas/feature/ftpm.yml`, `ahab` feature absent entirely. +Confirmed live on the board, not inferred from reading the layer: + +```text +# /dev/tpm0 and /dev/tpmrm0 registered at boot +crw------- 1 root root 10, 224 /dev/tpm0 +crw------- 1 root root 252, 65536 /dev/tpmrm0 + +# journal, first boot +cryptsetup-var: first boot - formatting /dev/disk/by-partlabel/var as LUKS2 +cryptsetup-var: opening LUKS2 container on /dev/disk/by-partlabel/var +cryptsetup-var: enrolling TPM2 keyslot (PCR 7) + +# cryptsetup luksDump /dev/mmcblk1p8 (excerpt) +Tokens: + 0: systemd-tpm2 + tpm2-hash-pcrs: 7 + tpm2-srk: true + tpm2-primary-alg: ecc + Keyslot: 1 +``` + +Slot 0 stays `argon2i` - the recovery keyslot - and is never removed; see the corrected +"Phase-2 TPM2 enrollment" section above. `/var` mounts live from `/dev/mapper/var` +(btrfs), no emergency-mode fallback. + +fTPM has zero dependency on AHAB: `kas/feature/ftpm.yml` never references `ahab` +anywhere, and this exact boot had the `ahab` feature off. That independence is worth +stating plainly because a bug briefly obscured it (next). + +**A separate u-boot bug, unrelated to fTPM itself, blocked this build until fixed.** +`u-boot-imx_%.bbappend` applied `ahab.cfg` (`CONFIG_AHAB_BOOT=y`) unconditionally on +`imx93-frdm`/`imx91-frdm` - gated on the machine, not on the `ahab` DISTRO_FEATURE. With +that compiled in, `booti` always tries to authenticate `argv[1]` as a signed AHAB +container; a build with `ahab` off never constructs one, so `booti` got a bare kernel +Image and failed with `Error: Wrong container header`. Fixed on +[meta-avocado #279](https://github.com/avocado-linux/meta-avocado/pull/279), gating +both `SRC_URI:append` lines on `bb.utils.contains('DISTRO_FEATURES', 'ahab', ...)` to +match the sibling gate two lines below it in the same file. Confirmed on hardware: +same board, same non-`ahab` build, clean boot with no AHAB authentication attempt. + +**This specific board can no longer validate AHAB at all**, independent of the bug +above. Reading fuse bank 16 (SRK_HASH) directly off this board's u-boot console shows +the correct SRK table hash burned in the wrong byte order - an out-of-band `fuse-srk` +tool unpacked the fuse file big-endian instead of NXP's native order (documented in +meta-avocado commit `0f3fc697`, 2026-08-13). The fuse is OTP; no key choice, old or +new, can make the ELE's native-order comparison match a byte-reversed value. The board +remains OEM Open (never closed) and boots fine for everything non-AHAB, fTPM included - +it just cannot re-run the AHAB row of the coverage table above. A different, unfused +FRDM-IMX93 unit is needed for that. ## Security properties Pulling the eMMC off the board gives an attacker AES-256-XTS ciphertext. Without the Argon2id recovery passphrase (device-bound, not in the image) or the TPM2 (physically in the device), they cannot decrypt it, and the two factors are independent. +**Updated 2026-08-19: the i.MX93 now has both factors, via the OP-TEE fTPM.** Earlier text here said the board had one factor, not two, because it had no TPM to seal to and the Argon2id slot - derived from the SoC's readable, non-secret OCOTP UID - was the only slot. That gap is closed: fTPM PCR-7 sealing is verified on this board (previous section), and the Argon2id slot is now genuinely a recovery path behind a sealed primary, not the sole key. One claim from that earlier text still holds and is not yet re-tested: the cloned-eMMC scenario in "Reproduce it" below (enroll on one board, boot the eMMC on a different one) has not been run against real fTPM hardware, so PCR-7 sealing blocking a cross-board clone is confirmed by design, not yet by an adversarial hardware test. + The storage chip alone is worthless: the key is sealed to the Secure Boot state, and the TPM is not on the chip. -The LUKS key is sealed to PCR 7 (UEFI Secure Boot state). A tampered bootloader or disabled Secure Boot changes PCR 7 before the initramfs runs, so the TPM refuses to unseal, and you cannot clone the eMMC to another board and boot it. dm-verity verifies every rootfs block against a build-time hash tree, so an offline patch is caught before the patched code executes. IMA/EVM extends that to mutable files: a binary dropped into `/var` or an overlay has no valid `security.ima` signature, so the kernel refuses to execute it. Module signing closes the in-band injection path: even with a shell, the attacker cannot load a custom kernel module. +The LUKS key is sealed to PCR 7 (UEFI Secure Boot state). A tampered bootloader or disabled Secure Boot changes PCR 7 before the initramfs runs, so the TPM refuses to unseal, and you cannot clone the eMMC to another board and boot it. That half is built, and verified on this board. + +The integrity half is design rather than delivery, and the coverage table above says which is which. The intent: dm-verity verifies every rootfs block against a build-time hash tree, so an offline patch is caught before the patched code executes; IMA/EVM extends that to mutable files, so a binary dropped into `/var` or an overlay has no valid `security.ima` signature and the kernel refuses to execute it; and module signing closes the in-band injection path, so even with a shell an attacker cannot load a custom kernel module. None of those three is implemented in the layer today. ## Issues and open questions +- **Unseal fails on reboot on the proxy; the TPM2-specific reboot claim is still open even on hardware.** The two-phase enrollment lands both slots correctly on QEMU, and then the very next boot cannot unseal from the TPM2 slot and falls back to Argon2. The layer already documents why, and it is a property of the emulator rather than of the design: the QEMU `virt` machine has no RPMB, so OP-TEE cannot commit the rollback counter its secure storage depends on, will not trust the fTPM's saved state on the next boot, and the fTPM re-derives its seed. **The 2026-08-12 board pass showed reboot survival for the Argon2id slot** - `/var` reopened across a reboot from an existing LUKS2 container, same filesystem UUID either side - and **the 2026-08-19 pass showed the eMMC-backed board _does_ have RPMB and a working fTPM, with the PCR-7 keyslot enrolling cleanly on first boot.** What remains genuinely untested on any platform is the specific case that matters: reopen `/var` via the TPM2 token specifically (not the Argon2id fallback) on a _second_ boot of an _already-enrolled_ image on real hardware. `cryptsetup-var.sh` is written to prefer the TPM2 token on open and fall back to Argon2id only on failure, so this should work - it just has not been independently re-tested this way yet. +- **An interrupted first boot wedges the device permanently.** `cryptsetup-var.sh` only runs `mkfs.btrfs` in its first-boot branch. A first boot that dies after `luksFormat` but before `mkfs.btrfs` leaves a LUKS2 header with no filesystem inside, so every later boot takes the open-existing branch, never creates the filesystem, and drops to emergency forever. Re-provisioning does not clear it, because `fwup -t complete` does not touch the `var` partition; recovery means zeroing the LUKS header by hand. The fix is either to detect "header present, no filesystem on the opened mapper" and re-`mkfs`, or to make format-plus-mkfs atomic behind a completion marker. - **`luksKillSlot` guard.** The `|| true` on slot 0 removal is intentional but means a failed kill leaves both slots active. A monitoring check after enrollment should confirm slot 0 is gone; the current implementation does not warn if the kill is skipped. - **IMA/EVM raises a labeling and update cost.** Every appraised file must carry a valid `security.ima` signature, so the build signs the rootfs and shipped mutable content, and every OS update re-signs and shifts the measured hashes. That makes the attestation server's known-good list an ongoing ops dependency. First boot also runs roughly 5-15% slower while IMA labels and measures, which is why IMA is gated on `ima` rather than always on. - **Software fallback Argon2id tier.** Boards without a TPM or fTPM (Renesas RZ/V2N, Synaptics SL1680, Qualcomm QCS6490, Raspberry Pi) fall back to a software-derived key bound to a provisioned secret and the hardware serial. This is a lower-security tier; the per-device provisioning of the secret is not addressed here. @@ -215,8 +485,10 @@ The LUKS key is sealed to PCR 7 (UEFI Secure Boot state). A tampered bootloader ## Reproduce it -- Boot the FRDM board with a rootfs where one block has been flipped with `dd`. dm-verity halts the boot before any userspace code runs. -- Enroll phase-2 TPM2 keys on one board, then clone the eMMC to a second board with a different TPM. PCR 7 on the clone does not match what was sealed; the clone cannot decrypt `/var`. -- Copy a stock binary into `/var/tmp` and run it. IMA appraisal denies the exec (no valid `security.ima` signature), while the same binary in its signed rootfs location runs normally. +Three adversarial tests, stated as predictions the hardware pass should confirm or break. None has been run on a board yet, and each one failing would say something specific about which control is not doing its job: + +- Boot the FRDM board with a rootfs where one block has been flipped with `dd`. dm-verity should halt the boot before any userspace code runs. The proxy cannot answer this one at all, since dm-verity is not active there. +- Enroll phase-2 TPM2 keys on one board, then clone the eMMC to a second board with a different TPM. PCR 7 on the clone should not match what was sealed, so the clone should not decrypt `/var`. This is the test that matters most right now, because the proxy's unseal already fails on a plain reboot of the _same_ machine. +- Copy a stock binary into `/var/tmp` and run it. IMA appraisal should deny the exec for want of a valid `security.ima` signature, while the same binary in its signed rootfs location runs normally. The controls live in `meta-avocado` and `meta-avocado-nxp` in the [avocado-linux organization on GitHub](https://github.com/avocado-linux). It composes cleanly with `hardened-security` (runtime posture), and the boot-chain controls satisfy CRA Annex I Part I items (b), (d), and (f) that a compliance mapping can build on.