diff --git a/.github/qualify-rocm/FEED.md b/.github/qualify-rocm/FEED.md new file mode 100644 index 000000000000..ce67f4584108 --- /dev/null +++ b/.github/qualify-rocm/FEED.md @@ -0,0 +1,176 @@ +# Qualification feed — published contract + +Every release run publishes its ROCm qualification records to the +**`qualification-data`** branch of this repository, in a fixed layout an +external dashboard can poll. This file is that contract. A copy is kept on the +branch as `qualification/SCHEMA.md`, refreshed on every publish, so the data and +its description never drift apart. Treat it as versioned (`schema_version`). + +## Why a branch and not the releases + +The records are already attached to each release, but that copy cannot serve a +dashboard: + +- A release only exists when something shipped. The dashboard needs every run, + including the ones whose ROCm archives were withheld. +- Reading history out of releases means one API call per release, and the + Releases API allows only 60 unauthenticated requests an hour. + +A file on a branch is one plain HTTP GET with no authentication and no rate +limit. The branch is only the transport; the contract below would hold +unchanged if the files moved to GitHub Pages, a bucket or a database. + +## Layout + +``` +qualification/ + index.json # the feed: one compact row per archive per run + reports/.json # the full record for that row (drill-down) + latest/-rocm-.json # the newest full record for each archive + SCHEMA.md # a copy of this file +``` + +Public raw base URL (no authentication, about 5 minutes of CDN caching): + +``` +https://raw.githubusercontent.com/lemonade-sdk/llama.cpp/qualification-data/qualification/ +``` + +The dashboard's one regular fetch is `.../qualification/index.json`. + +## One row per archive + +A release run builds six ROCm archives — Windows and Linux for each of ROCm +7.14, 10.0 and 10.1 — and each is qualified on its own. Each one gets its own +row, because an archive is what a user downloads and what a bug report names. +Rows from the same run share `run_id` and `tag`; group on those to show a run as +a whole. + +## `build_id` and re-runs + +``` +build_id = "--rocm-" e.g. 39775718636-win-rocm-10.1 +``` + +Unique per archive per run. Publishing is an upsert keyed on `build_id`: +re-running the release job replaces that run's rows and files instead of adding +new ones. `run_attempt` records which attempt the row came from. + +## Which runs appear + +Runs of `.github/workflows/release.yml` that reach its `release` job: the +scheduled runs, and manual runs with `create_release` ticked. Pass or fail, +every record that job writes is published. + +Not in the feed: + +- **Pull requests**, and manual runs without `create_release`. They run the + checks, but never reach the step that turns them into records. +- **An archive whose build failed.** There was nothing to check, so there is no + record and no row. A run normally has six rows; fewer means a build leg + failed (see `run_url`). +- **A run that failed before qualifying anything**, for example because no + artifact at all was built. + +## `index.json` + +```jsonc +{ + "schema_version": 1, + "repo": "lemonade-sdk/llama.cpp", + "updated_at": "2026-10-22T14:51:14Z", // generated_at of the newest row + "builds": [ // newest run first + { + "build_id": "39775718636-win-rocm-10.1", + "tag": "b11819", // llama.cpp build number = the release tag + "upstream_sha": "0693dbfb915f33233818373b5f6969396658d02c", + "artifact": "llama-b11819-bin-win-rocm-10.1-x64.zip", + "os": "win", // win | ubuntu + "rocm_version": "10.1", // currently 7.14 | 10.0 | 10.1 + "gfx_target": "gfx1151", // the target tested on real hardware, or null + "hardware_validated": true, // did Tier 1 run on a GPU + "gfx": { "requested": 20, "present": 20, "missing": [] }, + "generated_at": "2026-10-22T14:51:14Z", // ISO-8601 UTC, the time axis + "run_id": "39775718636", + "run_attempt": 1, + "run_url": "https://github.com/lemonade-sdk/llama.cpp/actions/runs/39775718636", + "event": "schedule", // schedule | workflow_dispatch + "tests": { "passed": 13, "failed": 0, "warned": 1, "skipped": 3, "missing": 0, "total": 17 }, + "qualified": true, // this archive passed its checks + "version_qualified": true, // both archives of this ROCm version qualified + "overall": "warn", // pass | warn | fail + "blocked_by": [], // e.g. ["tier0=fail"] + "blocking_tests": [], // e.g. ["T0.3"] + "results": { + "T0.1": "warn", "T0.2": "pass", "T0.3": "pass", "T0.4": "pass", + "T0.5": "pass", "T0.6": "skip", "T0.7": "skip", "T0.8": "pass", + "T0.9": "skip", "T0.10": "pass", "T0.11": "pass", "T0.12": "pass", + "T1.1": "pass", "T1.2": "pass", "T1.3": "pass", "T1.4": "pass", "T1.5": "pass" + }, + "report_url": "reports/39775718636-win-rocm-10.1.json" + } + ] +} +``` + +### Field notes + +- **`results`** has one entry per test: T0.1–T0.12 and T1.1–T1.5, as listed in + [README.md](README.md). Values are `pass | fail | warn | skip | missing`. A + test that did not run at all is `missing`. A test id added to the suite later + appears here as soon as it runs, so the failure-frequency and heatmap views + can be built from `index.json` alone. +- **`tests`** counts the values in `results`, so + `passed + failed + warned + skipped + missing == total` always holds. +- **`missing` is not `failed`.** This is the one deliberate difference from the + vllm-rocm feed, which counts missing tests as failed. Here, Tier 1 needs a + machine with an AMD GPU; when none ran it, all five T1 tests are `missing`, + `hardware_validated` is `false`, and the archive can still pass + (`qualified: true`). Showing that as a failure would paint a shipped archive + red. Use `qualified`, not the counts, to decide pass or fail. +- **`qualified`** is the verdict for this one archive: Tier 0 ran and did not + fail, and Tier 1 did not fail if it ran. A `warn` does not block, and tests + marked non-gating in README.md never affect it. +- **`version_qualified`** is the verdict the release acts on. A ROCm version is + withheld from the release unless its Windows **and** Linux archives both + qualify, so an archive can have `qualified: true` and still not ship because + its counterpart failed. `null` means that run did not record the decision. + (A version can also be withheld because one of its archives failed to build; + that archive then has no row at all.) +- **`blocked_by`** gives the reason per tier (`tier0=fail`, `tier0 missing`, + `tier1=fail`); **`blocking_tests`** lists the release-blocking tests that + failed. Both are empty when the archive qualified. +- **`gfx`** summarises the check that every gfx target was compiled into the + GPU backend (T0.3): how many were requested, how many are present, and the + names of any that are missing. +- **`gfx_target` / `hardware_validated`** say which single target was actually + tested on a GPU. Every other target in `gfx.requested` was only checked for + presence, never run. Don't present those as tested. +- **`report_url`** is relative to the base URL; it holds every check's + `status`, `gating`, `error` and `details`. + +## `reports/.json` + +The `qualification-.json` record exactly as `aggregate.py` wrote it +(`kind: "qualification_record"`): the `build` metadata, `overall`, `promoted`, +`promotion.blocked_by`, raw `counts`, the `gfx` audit, and `tiers` with every +test. The same content is attached to the release when the run published one. +Its own `build.build_id` (`@`) differs from the feed's +`build_id`; use the feed's. + +## `latest/-rocm-.json` + +The newest full record for one archive (for example `latest/win-rocm-10.1.json`), +for status tiles that want the current state without scanning the index. It is +replaced only by a newer record, so re-running an old release cannot move it +backwards. + +## Versioning and growth + +Bump `schema_version` on any breaking change to `index.json` or the row shape. +Adding a field is not breaking. The dashboard should read `schema_version` and +show a notice, rather than break, on a version it does not know. + +`index.json` grows by about six rows per run, roughly a dozen a day. That is +comfortable to poll for a long time, especially compressed. If it ever is not, +older rows will move into per-month files under a new `schema_version`. diff --git a/.github/qualify-rocm/README.md b/.github/qualify-rocm/README.md index 1d4c1a5a1cb6..2f52ef5c5650 100644 --- a/.github/qualify-rocm/README.md +++ b/.github/qualify-rocm/README.md @@ -217,6 +217,13 @@ were actually tested on hardware, and the result of every check — so questions like "was this gfx target ever verified in that build?" stay answerable long after the CI logs have expired. +## Published feed + +The release job's records are also published, pass or fail, to the +`qualification-data` branch by `publish_qualification.py`, so a dashboard can +follow the history of every run without reading each release. The layout and +every field are described in [FEED.md](FEED.md). + ## Out of scope - **Performance.** Tracked downstream against these same artifacts; a second diff --git a/.github/qualify-rocm/publish_qualification.py b/.github/qualify-rocm/publish_qualification.py new file mode 100644 index 000000000000..3b8cb2ce55eb --- /dev/null +++ b/.github/qualify-rocm/publish_qualification.py @@ -0,0 +1,267 @@ +#!/usr/bin/env python3 +""" +Publish the qualification records from one release run to the dashboard feed +on the `qualification-data` branch. + +Reads the records `aggregate.py` wrote (one qualification-.json per +archive, plus qualification-summary.json) and upserts them into --data-dir, +which is the `qualification/` folder of a checkout of that branch: + + index.json one compact row per archive per run, newest first + reports/.json the full record, for drill-down + latest/-rocm-.json the newest full record for that archive + +build_id is "--rocm-". +Publishing the same run again (a re-run of the release job) replaces its rows and files +rather than duplicating them, so this script is safe to run any number of times. + +A test the record does not contain is reported as "missing" and counted separately -- NOT folded into "failed". +A missing Tier 1 is a legitimate state: the archive was not tested on hardware, and aggregate.py lets it ship. +Counting it as a failure would show a published archive as red. + +The contract this writes is FEED.md, copied to the branch as SCHEMA.md. + +Usage: + python3 publish_qualification.py --records-dir ./qualification \ + --data-dir ./data/qualification --event schedule +""" + +import argparse +import glob +import json +import os +import re +import sys + +SCHEMA_VERSION = 1 + +# Every test the suite runs, in the order the README lists them. +# A test absent from a record shows up as "missing", so the dashboard always has the same columns to draw. +# A test id not listed here is still published, never dropped. +TIER_TESTS = { + "tier0": ["T0.1", "T0.2", "T0.3", "T0.4", "T0.5", "T0.6", + "T0.7", "T0.8", "T0.9", "T0.10", "T0.11", "T0.12"], + "tier1": ["T1.1", "T1.2", "T1.3", "T1.4", "T1.5"], +} +KNOWN_TESTS = [test_id for ids in TIER_TESTS.values() for test_id in ids] + +STATUS_MISSING = "missing" +SUMMARY_FILE = "qualification-summary.json" + +# build_id becomes a filename on the data branch, so it may only contain +# characters that cannot escape the folder it is written to. +_SAFE_ID = re.compile(r"^[A-Za-z0-9._-]+$") + + +def _int(value): + try: + return int(value) + except (TypeError, ValueError): + return None + + +def _test_order(test_id): + """Sort "T0.10" after "T0.9", and unknown ids after known ones.""" + match = re.match(r"^T(\d+)\.(\d+)$", test_id or "") + if match: + return (int(match.group(1)), int(match.group(2)), "") + return (sys.maxsize, sys.maxsize, test_id or "") + + +def load_records(records_dir): + """Return (records, summary) from the folder aggregate.py wrote.""" + records = [] + summary = None + for path in sorted(glob.glob(os.path.join(records_dir, "*.json"))): + try: + with open(path, encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, ValueError) as exc: + print(f"::warning::skipping unreadable record {path}: {exc}") + continue + if os.path.basename(path) == SUMMARY_FILE: + summary = data + elif data.get("kind") == "qualification_record": + records.append(data) + return records, summary + + +def build_id_for(record): + build = record.get("build", {}) + run_id = build.get("run_id") + os_name = build.get("os") + rocm_version = build.get("rocm_version") + if not (run_id and os_name and rocm_version): + raise SystemExit( + f"::error::record for {build.get('artifact')!r} has no run_id, os or " + "rocm_version, so it cannot be given a build_id") + build_id = f"{run_id}-{os_name}-rocm-{rocm_version}" + if not _SAFE_ID.match(build_id): + raise SystemExit(f"::error::refusing unsafe build_id {build_id!r}") + return build_id + + +def collect_results(record): + """Per-test status across the known suite, plus any test it does not list.""" + by_id = {} + for tier in record.get("tiers", {}).values(): + for test in tier.get("tests", []): + by_id.setdefault(test.get("id"), test) + + results = {test_id: by_id[test_id]["status"] if test_id in by_id else STATUS_MISSING + for test_id in KNOWN_TESTS} + for test_id in sorted((t for t in by_id if t not in results), key=_test_order): + results[test_id] = by_id[test_id]["status"] + + blocking = sorted((test_id for test_id, test in by_id.items() + if test.get("gating") and test.get("status") == "fail"), + key=_test_order) + return results, blocking + + +def build_entry(record, build_id, summary, run_url, event): + """Derive the compact, dashboard-facing index row from a full record.""" + build = record.get("build", {}) + results, blocking = collect_results(record) + statuses = list(results.values()) + + version_qualified = None + if summary is not None: + blocked = summary.get("blocked_rocm_versions") or [] + version_qualified = build.get("rocm_version") not in blocked + + gfx = record.get("gfx", {}) + return { + "build_id": build_id, + "tag": build.get("llamacpp_tag"), + "upstream_sha": build.get("upstream_sha"), + "artifact": build.get("artifact"), + "os": build.get("os"), + "rocm_version": build.get("rocm_version"), + "gfx_target": build.get("gfx_target"), + "hardware_validated": bool(build.get("hardware_validated")), + "gfx": { + "requested": len(gfx.get("requested", [])), + "present": len(gfx.get("present_in_binary", [])), + "missing": gfx.get("missing_from_binary", []), + }, + "generated_at": record.get("generated_at"), + "run_id": build.get("run_id"), + "run_attempt": _int(build.get("run_attempt")), + "run_url": run_url, + "event": event, + "tests": { + "passed": statuses.count("pass"), + "failed": statuses.count("fail"), + "warned": statuses.count("warn"), + "skipped": statuses.count("skip"), + "missing": statuses.count(STATUS_MISSING), + "total": len(statuses), + }, + "qualified": bool(record.get("promoted")), + "version_qualified": version_qualified, + "overall": record.get("overall"), + "blocked_by": record.get("promotion", {}).get("blocked_by", []), + "blocking_tests": blocking, + "results": results, + "report_url": f"reports/{build_id}.json", + } + + +def write_json(path, data): + with open(path, "w", encoding="utf-8") as handle: + json.dump(data, handle, indent=2) + handle.write("\n") + + +def update_latest(path, record): + """Write the record unless the file already holds a newer one. + + Re-publishing an old run (a late re-run) must not roll the status tile + back to an older result. + """ + if os.path.exists(path): + try: + with open(path, encoding="utf-8") as handle: + current = json.load(handle).get("generated_at") or "" + except (OSError, ValueError): + current = "" + if current > (record.get("generated_at") or ""): + return False + write_json(path, record) + return True + + +def upsert_index(index_path, repo, entries): + """Replace-or-append each entry by build_id, newest first.""" + if os.path.exists(index_path): + with open(index_path, encoding="utf-8") as handle: + index = json.load(handle) + else: + index = {} + + replaced = {entry["build_id"] for entry in entries} + builds = [b for b in index.get("builds", []) if b.get("build_id") not in replaced] + builds.extend(entries) + # Newest run first; the archives within one run in a fixed, readable order. + builds.sort(key=lambda b: b.get("build_id") or "") + builds.sort(key=lambda b: b.get("generated_at") or "", reverse=True) + + return { + "schema_version": SCHEMA_VERSION, + "repo": repo or index.get("repo"), + "updated_at": builds[0].get("generated_at") if builds else None, + "builds": builds, + } + + +def main(): + parser = argparse.ArgumentParser(description="Publish qualification records to the dashboard feed") + parser.add_argument("--records-dir", required=True, + help="folder holding the qualification-*.json records aggregate.py wrote") + parser.add_argument("--data-dir", required=True, + help="the qualification/ folder of the qualification-data branch") + parser.add_argument("--event", default=os.environ.get("GITHUB_EVENT_NAME"), + help="what triggered the run, e.g. schedule or workflow_dispatch") + parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"), + help="owner/name, used to build run_url") + parser.add_argument("--server-url", default=os.environ.get("GITHUB_SERVER_URL", "https://github.com")) + args = parser.parse_args() + + records, summary = load_records(args.records_dir) + if not records: + raise SystemExit(f"::error::no qualification records in {args.records_dir}; nothing to publish") + if summary is None: + print(f"::warning::no {SUMMARY_FILE} in {args.records_dir}; version_qualified will be null") + + os.makedirs(os.path.join(args.data_dir, "reports"), exist_ok=True) + os.makedirs(os.path.join(args.data_dir, "latest"), exist_ok=True) + + entries = [] + for record in records: + build = record["build"] + build_id = build_id_for(record) + run_url = (f"{args.server_url}/{args.repo}/actions/runs/{build['run_id']}" + if args.repo else None) + entry = build_entry(record, build_id, summary, run_url, args.event) + entries.append(entry) + + write_json(os.path.join(args.data_dir, "reports", f"{build_id}.json"), record) + latest = os.path.join(args.data_dir, "latest", + f"{build['os']}-rocm-{build['rocm_version']}.json") + moved = update_latest(latest, record) + + tests = entry["tests"] + print(f"{build_id}: {tests['passed']}/{tests['total']} passed, " + f"{tests['failed']} failed, {tests['missing']} missing, " + f"qualified={entry['qualified']} version_qualified={entry['version_qualified']}" + f"{'' if moved else ' (latest/ kept: it already holds a newer run)'}") + + index_path = os.path.join(args.data_dir, "index.json") + index = upsert_index(index_path, args.repo, entries) + write_json(index_path, index) + print(f"Published {len(entries)} record(s); {len(index['builds'])} build(s) in the index") + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 8767d3e9e827..29622840761d 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1646,6 +1646,16 @@ jobs: --output-summary ./qualification/qualification-summary.json \ --fail-on-no-promote + # Hand the records to publish-qualification, which adds them to the dashboard feed. + # Uploaded straight after they are written so that a failure in any later step of this job cannot keep them out of the feed. + - name: Upload qualification records + if: always() + uses: actions/upload-artifact@v6 + with: + name: qualification-records + path: qualification/*.json + if-no-files-found: warn + - name: Drop unqualified ROCm versions id: rocm_gate env: @@ -1858,3 +1868,96 @@ jobs: run: | echo "::error::ROCm ${{ steps.rocm_gate.outputs.blocked }} did not qualify — ${{ steps.rocm_gate.outputs.dropped }} artifact(s) were dropped from this release. Every other ROCm version and every other backend was published normally. The qualification-*.json records attached to the release list each failing check." exit 1 + + # --------------------------------------------------------------------------- + # Publish this run's ROCm qualification records to the qualification-data branch, where the dashboard polls them. + # Every record is published irrespective of it passing or failing. + # A feed that only held the runs that shipped would show nothing but successes. + # The format is described in .github/qualify-rocm/FEED.md. + # --------------------------------------------------------------------------- + publish-qualification: + # The release job goes red whenever any backend family is incomplete, which says + # nothing about the ROCm records it wrote, so publish after either outcome. + # Skipped along with the release job on pull requests and on dispatches + # without create_release so neither produces records. + if: ${{ always() && (needs.release.result == 'success' || needs.release.result == 'failure') }} + + needs: + - resolve-upstream + - release + + runs-on: ubuntu-24.04 + + permissions: + contents: write # to push to the qualification-data branch + + steps: + - name: Checkout qualification suite + uses: actions/checkout@v6 + with: + path: lemonade-ci + + - name: Download qualification records + continue-on-error: true + uses: actions/download-artifact@v7 + with: + name: qualification-records + path: ./records + + - name: Publish to qualification-data branch + env: + GH_TOKEN: ${{ github.token }} + TAG: ${{ needs.resolve-upstream.outputs.tag }} + run: | + set -euo pipefail + shopt -s nullglob + records=(records/qualification-*.json) + if [ ${#records[@]} -eq 0 ]; then + echo "::warning::The release job produced no qualification records, so there is nothing to publish." + exit 0 + fi + + git config --global user.name "github-actions[bot]" + git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" + REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" + BRANCH="qualification-data" + + pushed=0 + for attempt in 1 2 3 4 5; do + rm -rf data + found=0 + git ls-remote --exit-code --heads "$REPO_URL" "$BRANCH" >/dev/null || found=$? + if [ "$found" -eq 0 ]; then + if ! git clone --quiet --depth 1 --branch "$BRANCH" "$REPO_URL" data; then + echo "Clone failed (attempt $attempt); retrying"; sleep $((attempt * 3)); continue + fi + elif [ "$found" -eq 2 ]; then + echo "$BRANCH does not exist yet; creating it" + git init --quiet data + git -C data checkout --quiet --orphan "$BRANCH" + git -C data remote add origin "$REPO_URL" + else + echo "Could not reach the repository (attempt $attempt); retrying"; sleep $((attempt * 3)); continue + fi + + python3 lemonade-ci/.github/qualify-rocm/publish_qualification.py \ + --records-dir records \ + --data-dir data/qualification + # Keep the format description next to the data it describes. + cp lemonade-ci/.github/qualify-rocm/FEED.md data/qualification/SCHEMA.md + + git -C data add -A + if git -C data diff --cached --quiet; then + echo "Nothing changed; the feed already holds this run." + pushed=1; break + fi + git -C data commit --quiet -m "qualification: run ${{ github.run_id }} (${TAG})" + if git -C data push --quiet origin "$BRANCH"; then + pushed=1; break + fi + echo "Push rejected (attempt $attempt), most likely because another run published first; retrying" + sleep $((attempt * 3)) + done + + [ "$pushed" = 1 ] || { echo "::error::Could not publish to $BRANCH after 5 attempts."; exit 1; } + echo "Feed: https://raw.githubusercontent.com/${{ github.repository }}/${BRANCH}/qualification/index.json"