Skip to content

Testing.md

Latest commit

 

History

History
221 lines (168 loc) · 8.2 KB

Testing.md

File metadata and controls

221 lines (168 loc) · 8.2 KB
Raw

Testing

wolfTrust separates host behavior, architecture-accurate emulation, and physical-hardware evidence. A result from one environment must not be reported as a result from another.

Validation layers

Environment What it validates What it does not validate
Native host State machines, manifests, IPC ownership, copied transfers, services, storage, crypto integration, recovery decisions, and negative inputs Cortex-M exception return, CMSE, SAU, MPU, GTZC, or physical flash behavior
M33MU Cortex-M33 instruction flow, TrustZone transitions, CMSE gateway calls, MPU faults, guest scheduling, authenticated boot, and target service interactions STM32H563 peripherals, real option bytes, WRP, ST-Link, or silicon timing
STM32H563 The actual NUCLEO-H563ZI boot chain, Secure/Non-secure attribution, faults, flash, UART, and selected end-to-end behavior Other devices, other provisioning states, peer-flash confidentiality, or adversarial peripheral and Non-secure NVIC ownership

Host tests

Run the complete native set:

make test

The suite list is generated by:

make -s -C tests/host print-suites

Current suites cover domain and manifest validation, lifecycle, guest verification, rollback decisions, IPC and FF-M behavior, SPM policy, gateway vectors, Secure Partition layout and recovery, HSM relay and key isolation, vault and storage services, attestation and COSE integration, firmware update, runtime remeasurement, VNET, public PSA headers, boot-handoff record consumption, and negative paths. The attestation IAK suite runs wolfHSM NVM with both the default 8-byte and STM32H5 16-byte flash programming units.

Additional host checks:

make test-compilers
make test-sanitize
make test-valgrind

Valgrind requires the tool on the host. make test-compilers reruns all suites with the selected CC (default cc); it does not select multiple compilers. Run it once per compiler, for example make test-compilers CC=clang. Sanitizer support depends on the local toolchain.

PSA FF conformance

make test-conformance

This target fetches the pinned Arm PSA architecture tests. When M33MU is available it runs the target FF-M IPC suite. Without M33MU it runs only the host-side client and policy subset and prints explicit warnings. Do not treat that fallback as target conformance.

The dedicated guest configurations also exercise the PSA Crypto, Storage, and Initial Attestation validation applications. Those are target scenarios, not part of a plain make test result.

M33MU scenarios

The baseline target command is:

make test-target

It runs the positive lifecycle, guest restart, cross-domain Secure fault, and FF-M conformance scenarios. Detection accepts m33mu on PATH or a path in M33MU. If the emulator is unavailable, the target reports a skip rather than a pass.

Additional focused runs use:

tests/target/run_m33mu_scenario.sh positive

The runner's usage output is the authoritative scenario list. It includes authenticated-boot failure, rollback, runtime remeasurement, Secure Partition recovery, key and vault isolation, storage recovery, attestation negatives, firmware update, manifest rejection, GTZC behavior, and VNET paths.

VNET has convenience targets:

make test-vnet
make test-vnet-target

test-vnet is host-only. test-vnet-target launches two authenticated wolfIP guests under M33MU.

STM32H563 hardware

The published hardware run requires:

  • a Linux host with Bash and GNU userland;
  • Docker for the documented container build;
  • Python 3 on the host for the flash-phase measurement check;
  • STM32_Programmer_CLI, found at its default install path or through STM32_CLI;
  • pyOCD with STM32H563 support;
  • a host-visible arm-none-eabi-nm, or an override in ARM_NM;
  • a serial VCP, default /dev/ttyACM0 or H5_SERIAL; and
  • a connected ST-Link.

The detector checks the board/programmer path, not every required host tool. It verifies the ST-Link USB ID only when lsusb is available. Without lsusb, a missing probe can be reported later as a flash failure rather than an initial skip.

Run the default hardware set with:

WT_H5_DOCKER_IMAGE=ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 make test-hardware

The target skips if hardware detection fails. It builds and flashes the positive, restart, cross-domain, and conformance scenarios by default. Select a narrower set with WT_H5_SCENARIOS:

WT_H5_SCENARIOS="positive bootupdate" \
WT_H5_DOCKER_IMAGE=ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 \
make test-hardware

The optional gtzcneg scenario must be selected explicitly. It verifies the GTZC peer-RAM curtain after a Non-secure MPU bypass; it does not establish peer flash confidentiality or adversarial peripheral and Non-secure NVIC ownership.

For the hardened guest-flash configuration, explicitly forward the build flag into the container and repeat it for the host flash run:

docker run --rm \
    -e WT_GUEST_FLASH_WRP=1 \
    -v "$PWD":/workspace \
    -w /workspace \
    ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 \
    bash tests/target/run_h5_hardware.sh build positive
WT_GUEST_FLASH_WRP=1 \
    tests/target/run_h5_hardware.sh flash positive

With that variable set, the flash runner clears WRP so it can program the guests, then re-applies WRP before boot. The Secure image independently reads the live WRP register and refuses an incompletely protected guest. The supported provisioning_ctrl.sh set-wrp workflow deliberately applies WRP only while the board is Open; that is a conservative project workflow, not the complete silicon rule. The hardware runner does not check product state before clearing or reapplying WRP. Current ST guidance makes WRP nonmodifiable in TZ-Closed, Closed, and Locked, and RM0481 separately defines the FLASH_WRPSGNxR.UNLOCK condition. The current suite wrapper does not forward this variable into its Docker build, so the shorter make test-hardware form does not build a WRP-enforcing Secure image.

Use the disposable container for Secure and guest builds. The current direct-host build path adds safe.directory '*' to the user's global Git configuration and is not suitable as a published workflow.

Hardware test commands write flash and can reset the board. Review STM32H5 Guide before running them.

CI coverage

The workflows under .github/workflows/ separately run:

  • host unit tests;
  • compiler variants, sanitizers, and Valgrind;
  • Cortex-M33 cross-compilation;
  • dependency integration;
  • the core/port split guard and the docs guard (no internal-ledger or home-directory references in the published docs);
  • fuzz targets; and
  • selected and nightly M33MU scenarios.

Trigger routing

The host, compiler, sanitizer, Valgrind, cross-compilation, integration, and core/port split checks run on every pull request, including drafts. The fuzz target also runs on pull requests as a 60-second libFuzzer smoke pass; the nightly schedule and manual dispatch run the 600-second soak instead.

The full M33MU matrix (the M33MU workflow: wolfBoot plus both guest lifecycles and every scenario) runs on every pull request, on a push to master, main, or wolfTrust-dev, on the nightly schedule, and on manual dispatch. Every PR gets the full emulator matrix automatically — no label or opt-in step.

Running M33MU off a pull request

To run the matrix against a branch without a PR, dispatch the workflow:

gh workflow run m33mu.yml --ref <branch>

To run a single scenario locally, use tests/target/run_m33mu_scenario.sh <key> (for example positive or crossdomain).

The GitHub-hosted workflows do not establish a physical-board result. Hardware output must come from the STM32H563 runner attached to a board.

Interpreting failures

  • A host pass plus target failure usually points to architecture glue, image assembly, linker placement, or hardware policy rather than a neutral state machine.
  • A missing target prints SKIP. It is not a successful run.
  • Preserve the first failing marker and the generated log before rebuilding; target runners place detailed output under logs/ or the configured scenario log.
  • Confirm that Secure, guest, manifest, flash, and emulator addresses were built from the same configuration.