From fb242be98728176e93167e2a9d441d7f07dd1a99 Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Sun, 18 Oct 2026 09:59:05 -0700 Subject: [PATCH] docs(test): document and guard the local test entry point MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scripts/bootstrap_local_tests.sh` makes the suite runnable locally, but nothing pointed at it: the README had build instructions only, there was no Makefile, and the Tier 3 note in #128 already counted the bootstrap as done. A contributor who runs a bare `swift test` on a fresh checkout can still get a run that reports success while almost nothing executed — MLX aborts the process when it cannot load its Metal library, and `swift test` exits 0 anyway. Add `make test`, a one-command entry point over the bootstrap plus `swift test --skip-build`, and document it under Build from Source along with the Swift 6.4 / Swift 6.2 difference and the Metal Toolchain requirement. `scripts/run-tests.sh` also rejects a short run: an unfiltered run that executed fewer than 200 tests fails with the metallib explanation instead of reporting a pass. Filtered runs are exempt, because narrowing is the point there. Verified on Swift 6.4 / Xcode 27: `make test` runs 326 tests, exit 0; `make test-filter FILTER=PromptCacheTests` runs 25, exit 0; forcing MIN_TESTS=100000 turns the same green run into the intended failure. Refs #128. --- Makefile | 16 +++++++++++++ README.md | 33 ++++++++++++++++++++++++++ scripts/run-tests.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 Makefile create mode 100755 scripts/run-tests.sh diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..2aef4ec --- /dev/null +++ b/Makefile @@ -0,0 +1,16 @@ +# Local entry points. +# +# `make test` is the supported way to run the unit suites. A bare `swift test` on a fresh +# checkout can die before the suite runs (MLX cannot find its Metal library) and still +# exit 0, which looks like a short green pass; scripts/run-tests.sh bootstraps the metallib +# and fails the run when too few tests executed. + +.PHONY: test test-filter + +test: + ./scripts/run-tests.sh + +# make test-filter FILTER=PromptCacheTests +test-filter: + @test -n "$(FILTER)" || { echo "usage: make test-filter FILTER="; exit 2; } + ./scripts/run-tests.sh --filter "$(FILTER)" diff --git a/README.md b/README.md index 49f8ab2..9f69482 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,39 @@ Then start the server (models download automatically if not cached): *(Add `--stream-experts` when running oversized MoE models to bypass macOS virtual memory swapping and stream expert layers directly from NVMe SSD.)* +### Run the Tests + +```bash +make test +``` + +That is the supported entry point. It is a thin wrapper over: + +```bash +./scripts/bootstrap_local_tests.sh # build the harness, make mlx.metallib findable +swift test --skip-build +``` + +Do **not** trust a bare `swift test` on a fresh checkout. MLX aborts the test process when it +cannot load its Metal library, and `swift test` can still exit 0 after printing +`Failed to load the default metallib` — on the pre-6.4 layout that happens at the second +test, so the run looks like a short green pass with almost nothing executed. `make test` +runs the bootstrap and then **fails if fewer than 200 tests executed**, so this failure mode +is reported instead of hidden. + +The bootstrap handles both toolchains automatically: + +- **Swift 6.4+ (Xcode 27)** — SwiftPM compiles the Cmlx kernels and embeds `default.metallib` + in each test bundle, so nothing is copied. It only checks the Metal Toolchain is installed + (`xcodebuild -downloadComponent MetalToolchain` if it is not). Copying a metallib into + `.xctest/Contents/MacOS` here invalidates the bundle signature and breaks the next + incremental build at CodeSign. +- **Swift 6.2 (Xcode 26, the CI runner)** — the metallib is not embedded; the bootstrap + fetches `mlx.metallib` from the `mlx` Python wheel and copies it next to the test binaries. + +To run a subset, `make test-filter FILTER=PromptCacheTests` (or pass `swift test` arguments +to `scripts/run-tests.sh` directly). + ## 📊 Performance: Mac mini M6 (base, 32 GB) The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README comes from 64 GB hardware. Running the same context lengths on this M6 exposed memory bugs that 64 GB machines had been hiding, and they are fixed in this release (see [What 32 GB exposed](#what-32-gb-exposed)). diff --git a/scripts/run-tests.sh b/scripts/run-tests.sh new file mode 100755 index 0000000..373763a --- /dev/null +++ b/scripts/run-tests.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# run-tests.sh — run the local unit suites and refuse to report a run that barely ran. +# +# Usage: scripts/run-tests.sh [extra `swift test` arguments...] +# scripts/run-tests.sh --filter PromptCacheTests +# +# Why this exists: MLX aborts the test process when it cannot load its Metal library. On the +# pre-6.4 build layout that happens on the second test, and `swift test` can still exit 0 +# after printing "Failed to load the default metallib" — the run looks like a short pass +# while ~270 tests never executed. `scripts/bootstrap_local_tests.sh` installs the metallib; +# this wrapper calls it and then checks the run actually executed the suite. +set -euo pipefail +cd "$(dirname "$0")/.." + +# The suite is a few hundred tests (326 at the time of writing). The floor sits well below +# that so ordinary additions and removals never need a bump, but far above the one or two a +# broken metallib leaves behind. +MIN_TESTS="${MIN_TESTS:-200}" + +bash scripts/bootstrap_local_tests.sh + +LOG="$(mktemp -t swiftlm-tests.XXXXXX)" +trap 'rm -f "$LOG"' EXIT + +set +e +swift test --skip-build --disable-swift-testing "$@" 2>&1 | tee "$LOG" +status=${PIPESTATUS[0]} +set -e + +# XCTest prints "Executed N tests" per suite and once for the whole run; the largest number +# is the total for the run. +executed=$(grep -oE "Executed [0-9]+ tests" "$LOG" | grep -oE "[0-9]+" | sort -n | tail -1) +executed=${executed:-0} + +if [ "$status" -ne 0 ]; then + exit "$status" +fi + +# The floor only makes sense for an unfiltered full run. A caller passing --filter is +# deliberately narrowing, and a dozen passing tests is a correct result there. +if [ "$#" -eq 0 ] && [ "$executed" -lt "$MIN_TESTS" ]; then + cat >&2 < ${executed} tests executed (floor ${MIN_TESTS})."