From 9e4e937d25668a80ccd172982556796695bd25d7 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 2 Sep 2026 13:20:07 +0800 Subject: [PATCH 01/10] fix(frontierchallenge): make open-track selection reproducible --- benchmarks/frontierchallenge/README.md | 14 ++- benchmarks/frontierchallenge/TASKS.md | 5 +- .../docs/huggingface-release.md | 15 ++- .../docs/providers/docker.md | 2 +- .../frontierchallenge/docs/quickstart.md | 23 +++- benchmarks/frontierchallenge/docs/running.md | 7 +- .../frontierchallenge/docs/troubleshooting.md | 5 + .../frontierchallenge/release/datasets.json | 11 ++ .../frontierchallenge/scripts/run_eval.sh | 112 ++++++++++-------- .../scripts/setup_release.py | 48 ++++++-- .../scripts/task_selection.py | 108 +++++++++++++++++ .../tests/test_run_eval_contract.py | 29 +++++ .../tests/test_setup_release.py | 33 +++++- .../tests/test_task_selection.py | 103 ++++++++++++++++ 14 files changed, 432 insertions(+), 83 deletions(-) create mode 100644 benchmarks/frontierchallenge/release/datasets.json create mode 100755 benchmarks/frontierchallenge/scripts/task_selection.py create mode 100644 benchmarks/frontierchallenge/tests/test_run_eval_contract.py create mode 100644 benchmarks/frontierchallenge/tests/test_task_selection.py diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index a4a654c..73f59b1 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -29,7 +29,7 @@ simulation, electrochemistry, quantitative imaging, and molecular biology. Tasks97 (74 hard, 23 medium) Taxonomy6 domains, 21 subdomains - Runtime81 open-image tasks, 16 user-supplied ORCA tasks + Runtime81 open-image tasks, 16 tasks executing user-supplied ORCA Gradingdeterministic checks; 77 tasks also judge the report HarnessHarbor 0.20.0 Outputnamed files under /app/output @@ -59,10 +59,10 @@ its SHA-256, and load it into Docker: HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -The full track adds 16 normally released ORCA tasks. FrontierChallenge does -not distribute ORCA or an image containing it. After obtaining ORCA 6.0.1 from -its official provider, build and smoke-test the private local runtime, then -validate the full track: +The full track adds 16 normally released tasks that execute ORCA. +FrontierChallenge does not distribute ORCA or an image containing it. After +obtaining ORCA 6.0.1 from its official provider, build and smoke-test the +private local runtime, then validate the full track: ```bash ./scripts/build_orca_runtime.sh \ @@ -74,6 +74,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track full Do not push, export, publish, or share the resulting ORCA image. See the [ORCA setup tutorial](docs/providers/orca.md). +Track membership describes what a task executes, not where its input files +came from. For example, `task_098_orca_claisen_thermochemistry` reads supplied +ORCA output but does not run ORCA, so it belongs to the open track. + ### 2. Run a real task Fill `.env`, then run Harbor with the Claude Code agent: diff --git a/benchmarks/frontierchallenge/TASKS.md b/benchmarks/frontierchallenge/TASKS.md index 6fa43fd..527bcb6 100644 --- a/benchmarks/frontierchallenge/TASKS.md +++ b/benchmarks/frontierchallenge/TASKS.md @@ -14,7 +14,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` The table below previews benchmark coverage without exposing evaluator data. -Keywords come from each task and describe technique rather than answers. +Keywords come from each task and describe technique rather than answers. The +`Image` column records what the task executes, not software named in supplied +files: `task_098_orca_claisen_thermochemistry`, for example, reads precomputed +ORCA output and therefore uses the open image. | Task | Difficulty | Image | Judge | Agent budget | Techniques | |---|---|---|---|---|---| diff --git a/benchmarks/frontierchallenge/docs/huggingface-release.md b/benchmarks/frontierchallenge/docs/huggingface-release.md index 5b83e3a..9304e09 100644 --- a/benchmarks/frontierchallenge/docs/huggingface-release.md +++ b/benchmarks/frontierchallenge/docs/huggingface-release.md @@ -19,9 +19,12 @@ workspace. HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -Setup downloads the current `main` branches, runs the verification tool bundled -with each dataset, and requires both `source_registry.json` files to equal this -checkout's `registry.json`. A mixed or incomplete dataset is refused. +Setup downloads the exact solve and reference commits declared in +`release/datasets.json`, runs the verification tool bundled with each dataset, +and requires both `source_registry.json` files to equal this checkout's +`registry.json`. A mixed or incomplete dataset is refused. Release maintainers +may test newer snapshots with `--revision` and `--reference-revision`; published +runtime changes should update both pins together. Use local directories instead of HF repository IDs for an offline handoff: @@ -44,3 +47,9 @@ the encrypted verifier hash. GitHub contains neither payload. The solve dataset must contain no `tests/`, verifier archive, rubric, fixture, or reference output; the reference dataset must contain no instruction, input, or runtime environment. + +The top-level Hugging Face `README.md` is intentionally outside +`checksums.sha256`: it is a mutable dataset card whose citation and links may be +edited without changing the benchmark payload. Task files, task-level READMEs, +registries, manifests, image artifacts, and verifier archives remain covered by +the checksum manifests and registry commitments. diff --git a/benchmarks/frontierchallenge/docs/providers/docker.md b/benchmarks/frontierchallenge/docs/providers/docker.md index 34370d8..c791448 100644 --- a/benchmarks/frontierchallenge/docs/providers/docker.md +++ b/benchmarks/frontierchallenge/docs/providers/docker.md @@ -92,7 +92,7 @@ path. ## Two things that bite **ORCA is user-supplied and writes beside its input.** Before selecting one of -the 16 ORCA tasks, create the licensed local runtime described in +the 16 tasks that execute ORCA, create the licensed local runtime described in [orca.md](orca.md). Copy ORCA inputs into a writable directory (`/app/data`, `/tmp`) before running; invoking ORCA directly on a read-only bind-mounted file fails. diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 8a11fe9..85f3b63 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -61,19 +61,25 @@ This is the shortest path for most evaluators: HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -Setup downloads the solve and reference datasets from their current `main` branches, -verifies both packages, binds them to this checkout's `registry.json`, then +Setup downloads the solve and reference revisions pinned by this Git checkout in +`release/datasets.json`, verifies both packages, binds them to this checkout's +`registry.json`, then downloads `images/frontierchallenge-cpu-open-2026.08.docker.tar.zst` from the solve dataset. It checks the declared size, SHA-256 and image ID before loading the `linux/amd64` image into Docker. No container registry is used. Evaluator- local paths are written to `.frontierchallenge/config.env`. +For release development only, `--revision main` overrides both pins; +`--reference-revision` can override the reference revision independently. Normal +evaluation should keep the checkout's pins so later dataset changes cannot alter +an otherwise identical run. + ### Full track: build the private ORCA runtime -All 16 ORCA task statements and inputs are released normally. Only ORCA and a -configured ORCA image are absent. Obtain ORCA 6.0.1 from its official provider, -install it outside this checkout, and keep the complete directory together. -Then run: +All statements and inputs for the 16 tasks that execute ORCA are released +normally. Only ORCA and a configured ORCA image are absent. Obtain ORCA 6.0.1 +from its official provider, install it outside this checkout, and keep the +complete directory together. Then run: ```bash ./scripts/build_orca_runtime.sh \ @@ -118,6 +124,11 @@ tasks into evaluator staging, decrypts the matching verifier there, starts the agent, and invokes Harbor's verifier after the agent exits. By default Claude Code's `WebSearch` and `WebFetch` tools are disabled. +Selection comes from each task's declared `task.json.environment`, validated +against the registry. Include/exclude filters are applied before image preflight, +staging, verifier decryption, resume, and Harbor invocation; stale directories +from an older run cannot add tasks to the effective run. + A healthy run reaches messages like: ```text diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index faff611..1de1e0a 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -12,7 +12,8 @@ The open track contains the 81 tasks that use the redistributable image: ./scripts/run_eval.sh --agent claude-code --model ``` -The full track adds 16 ORCA tasks. Prepare the licensed local runtime first: +The full track adds 16 tasks that execute ORCA. Prepare the licensed local +runtime first: ```bash ./scripts/build_orca_runtime.sh --orca-root /path/to/orca-6.0.1 @@ -22,6 +23,10 @@ The full track adds 16 ORCA tasks. Prepare the licensed local runtime first: Setup writes verified local paths under `.frontierchallenge/`. The runner validates the GitHub/solve/reference registries again before staging anything. +Track membership follows each task's declared execution environment, not +software names in its instruction or supplied files. Thus +`task_098_orca_claisen_thermochemistry`, which only reads precomputed ORCA +output, remains an open-track task. ## Runtime diff --git a/benchmarks/frontierchallenge/docs/troubleshooting.md b/benchmarks/frontierchallenge/docs/troubleshooting.md index 9bece3f..1782740 100644 --- a/benchmarks/frontierchallenge/docs/troubleshooting.md +++ b/benchmarks/frontierchallenge/docs/troubleshooting.md @@ -40,6 +40,11 @@ or distribute it. After obtaining and installing ORCA officially, run: The build helper runs a real ORCA calculation, and the runner checks the local image again before any selected ORCA task starts. +An open task may still mention ORCA because it reads supplied output files. +That does not require the licensed runtime: preflight follows the task's +registry-backed execution environment, not instruction text. For example, +`task_098_orca_claisen_thermochemistry` belongs to the open track. + ## Every task fails during agent setup Check that `.env` contains the key required by the selected agent and that the diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json new file mode 100644 index 0000000..f7bdea4 --- /dev/null +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -0,0 +1,11 @@ +{ + "schema_version": 1, + "solve": { + "repo": "apodex/FrontierChallenge", + "revision": "8a73bb02a5709aa007e1a146d0cdc5d36ef5ea4d" + }, + "reference": { + "repo": "apodex/FrontierChallenge-reference", + "revision": "59fbf007ad5c6816caf7d379eef32d9fb384edea" + } +} diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index b522b63..433fa47 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -233,33 +233,55 @@ fi mkdir -p "$STAGE_DIR" "$JOBS_DIR" -# Mirror Harbor's include/exclude matching so only selected tasks are staged. -task_selected() { - local task_id="$1" - if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then - local matched=0 - for pattern in "${INCLUDE_PATTERNS[@]}"; do - [[ "$task_id" == $pattern ]] && { matched=1; break; } - done - [[ "$matched" -eq 1 ]] || return 1 - fi +# Resolve selection once from the verified solve package. Persistent staging may +# contain tasks from older runs, so it must never define preflight or grading. +SELECTION_ARGS=( + --tasks-root "$SOLVE_TASKS" + --registry "$SOLVE_DIR/source_registry.json" + --track "$TRACK" +) +if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then + for pattern in "${INCLUDE_PATTERNS[@]}"; do + SELECTION_ARGS+=(--include "$pattern") + done +fi +if [[ ${#EXCLUDE_PATTERNS[@]} -gt 0 ]]; then for pattern in "${EXCLUDE_PATTERNS[@]}"; do - [[ "$task_id" == $pattern ]] && return 1 + SELECTION_ARGS+=(--exclude "$pattern") done - return 0 -} +fi + +if ! SELECTION_OUTPUT="$( + python3 "$ROOT/scripts/task_selection.py" "${SELECTION_ARGS[@]}" +)"; then + echo "FATAL: task selection is invalid." >&2 + exit 1 +fi +EFFECTIVE_TASK_IDS=() +EFFECTIVE_TASK_ENVS=() +EFFECTIVE_TASK_SOURCES=() +while IFS=$'\t' read -r task_id task_environment task_source; do + [[ -n "$task_id" ]] || continue + EFFECTIVE_TASK_IDS+=("$task_id") + EFFECTIVE_TASK_ENVS+=("$task_environment") + EFFECTIVE_TASK_SOURCES+=("$task_source") +done <<< "$SELECTION_OUTPUT" +unset SELECTION_OUTPUT +if [[ ${#EFFECTIVE_TASK_IDS[@]} -eq 0 ]]; then + echo "FATAL: task selection is empty." >&2 + exit 1 +fi echo "== Staging $TRACK-track tasks from $SOLVE_TASKS into $STAGE_DIR ==" staged=0 skipped=0 -for task_dir in "$SOLVE_TASKS"/*/; do - task_id="$(basename "$task_dir")" - [[ -f "$task_dir/task.toml" ]] || continue - if [[ "$TRACK" == "open" ]] && ! grep -q '"environment": "open"' "$task_dir/task.json"; then - continue - fi - task_selected "$task_id" || continue +EFFECTIVE_TASK_DIRS=() +for index in "${!EFFECTIVE_TASK_IDS[@]}"; do + task_id="${EFFECTIVE_TASK_IDS[$index]}" + task_environment="${EFFECTIVE_TASK_ENVS[$index]}" + task_dir="${EFFECTIVE_TASK_SOURCES[$index]}" dest="$STAGE_DIR/$task_id" + EFFECTIVE_TASK_DIRS+=("$dest") source_identity="$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE" if [[ "$FORCE_RESTAGE" -eq 0 && -f "$dest/task.toml" \ && -f "$dest/instruction.md" && ! -e "$dest/statement.fcref" \ @@ -271,9 +293,12 @@ for task_dir in "$SOLVE_TASKS"/*/; do # Copy the verified solve task and pin its open-image Dockerfile to setup's # selected reference. The HF source is immutable; only the staged copy changes. rm -rf "$dest" - cp -a "$task_dir" "$dest" + # Hugging Face snapshots may expose files as cache-relative symlinks. Copy + # their contents so the evaluator stage cannot contain broken links after it + # leaves the snapshot directory hierarchy. + cp -aL "$task_dir" "$dest" printf '%s\n' "$source_identity" > "$dest/.frontier-source" - if grep -q '"environment": "open"' "$dest/task.json"; then + if [[ "$task_environment" == "open" ]]; then OPEN_IMAGE="$OPEN_IMAGE" python3 - "$dest/environment/Dockerfile" <<'PIN_OPEN_IMAGE' import os import pathlib @@ -316,15 +341,15 @@ if [[ "$AGENT" == "claude-code" && ${#AGENT_KWARGS[@]} -eq 0 ]]; then # Prevent direct web lookup unless the evaluator deliberately overrides this. AGENT_KWARG_ARGS+=(--agent-kwarg "disallowed_tools=WebSearch WebFetch") fi -AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}") +if [[ ${#AGENT_KWARGS[@]} -gt 0 ]]; then + AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}") +fi INCLUDE_ARGS=() -for pattern in "${INCLUDE_PATTERNS[@]}"; do - INCLUDE_ARGS+=("--include-task-name" "$pattern") -done -EXCLUDE_ARGS=() -for pattern in "${EXCLUDE_PATTERNS[@]}"; do - EXCLUDE_ARGS+=("--exclude-task-name" "$pattern") +# Give Harbor the exact resolved IDs. This makes stale directories in a reused +# stage invisible even when the user supplied no include/exclude flags. +for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + INCLUDE_ARGS+=("--include-task-name" "$task_id") done # A command-line verifier env takes precedence over the task declaration. @@ -388,18 +413,9 @@ fi # Fail before evaluation if a selected task needs ORCA but the evaluator-local # licensed runtime is unavailable. orca_tasks=() -for task_dir in "$STAGE_DIR"/*/; do - [[ -d "$task_dir" ]] || continue - task_name="$(basename "$task_dir")" - if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then - matched=0 - for pattern in "${INCLUDE_PATTERNS[@]}"; do - [[ "$task_name" == *"$pattern"* ]] && { matched=1; break; } - done - [[ "$matched" -eq 1 ]] || continue - fi - if grep -qil 'orca' "$task_dir/task.toml" "$task_dir/instruction.md" "$task_dir/environment/Dockerfile" 2>/dev/null; then - orca_tasks+=("$task_name") +for index in "${!EFFECTIVE_TASK_IDS[@]}"; do + if [[ "${EFFECTIVE_TASK_ENVS[$index]}" == "licensed-orca" ]]; then + orca_tasks+=("${EFFECTIVE_TASK_IDS[$index]}") fi done if [[ ${#orca_tasks[@]} -gt 0 ]]; then @@ -434,15 +450,14 @@ fi # completed trial directories but archive stale job-level metadata. RESUME_JOB=0 if [[ -f "$JOBS_DIR/$JOB_NAME/config.json" ]]; then - if REQUESTED="${INCLUDE_PATTERNS[*]-}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY' + if REQUESTED="${EFFECTIVE_TASK_IDS[*]}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY' import json, os, sys requested = set(os.environ.get("REQUESTED", "").split()) try: recorded = {t["task"]["name"] for t in json.load(open(sys.argv[1]))["trials"]} except Exception: sys.exit(1) # unreadable lock -> treat as new work -# No --include means "the whole staged set", which resume also covers. -sys.exit(0 if not requested or requested == recorded else 1) +sys.exit(0 if requested == recorded else 1) PY then RESUME_JOB=1 @@ -469,10 +484,8 @@ if [[ -f "$REFERENCE_DIR/tools/verify_reference_dataset.py" ]]; then fi echo "== Injecting encrypted verifier archives from $REFERENCE_DIR ==" injected=0 -for task_dir in "$STAGE_DIR"/*/; do - [[ -f "$task_dir/task.toml" ]] || continue +for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do task_id="$(basename "$task_dir")" - task_selected "$task_id" || continue source_verifier="$REFERENCE_TASKS/$task_id/verifier.fcref" if [[ ! -f "$source_verifier" ]]; then echo "FATAL: encrypted verifier missing for $task_id: $source_verifier" >&2 @@ -497,8 +510,7 @@ echo "Injected $injected encrypted verifier archive(s)." if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "== Unsealing encrypted verifiers with the published archive password ==" unsealed=0 - for task_dir in "$STAGE_DIR"/*/; do - [[ -f "$task_dir/task.toml" ]] || continue + for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do if [[ ! -f "$task_dir/instruction.md" || ! -f "$task_dir/verifier.fcref" ]]; then echo "FATAL: $(basename "$task_dir") lacks plaintext instruction or verifier archive." >&2 exit 1 @@ -534,7 +546,7 @@ else --artifact /app/output \ --jobs-dir "$JOBS_DIR" --job-name "$JOB_NAME" \ --env-file "$ENV_FILE" \ - "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${EXCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \ + "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \ --yes fi diff --git a/benchmarks/frontierchallenge/scripts/setup_release.py b/benchmarks/frontierchallenge/scripts/setup_release.py index f8cc997..48ea539 100755 --- a/benchmarks/frontierchallenge/scripts/setup_release.py +++ b/benchmarks/frontierchallenge/scripts/setup_release.py @@ -13,11 +13,22 @@ import sys from pathlib import Path - ROOT = Path(__file__).resolve().parents[1] -DEFAULT_SOLVE_REPO = "apodex/FrontierChallenge" -DEFAULT_REFERENCE_REPO = "apodex/FrontierChallenge-reference" -DEFAULT_REVISION = "main" + + +def select_revisions( + release: dict, + revision_override: str | None, + reference_revision_override: str | None, +) -> tuple[str, str]: + """Return pinned defaults, preserving the legacy --revision override.""" + solve_revision = revision_override or release["solve"]["revision"] + reference_revision = ( + reference_revision_override + or revision_override + or release["reference"]["revision"] + ) + return solve_revision, reference_revision def digest(path: Path) -> str: @@ -284,14 +295,16 @@ def write_config( reference: Path, track: str, open_image: str, - revision: str, + solve_revision: str, + reference_revision: str, ) -> None: values = { "FRONTIER_SOLVE_DIR": str(solve), "FRONTIER_REFERENCE_DIR": str(reference), "FRONTIER_TRACK": track, "FRONTIER_OPEN_IMAGE": open_image, - "FRONTIER_DATASET_REVISION": revision, + "FRONTIER_SOLVE_REVISION": solve_revision, + "FRONTIER_REFERENCE_REVISION": reference_revision, } path.parent.mkdir(parents=True, exist_ok=True) path.write_text( @@ -303,11 +316,16 @@ def write_config( def main() -> int: image_manifest = json.loads((ROOT / "release" / "images.json").read_text()) + dataset_release = json.loads((ROOT / "release" / "datasets.json").read_text()) default_image = image_manifest["images"]["open"]["ref"] parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--solve-source", default=DEFAULT_SOLVE_REPO) - parser.add_argument("--reference-source", default=DEFAULT_REFERENCE_REPO) - parser.add_argument("--revision", default=DEFAULT_REVISION) + parser.add_argument("--solve-source", default=dataset_release["solve"]["repo"]) + parser.add_argument("--reference-source", default=dataset_release["reference"]["repo"]) + parser.add_argument( + "--revision", + default=None, + help="override both pinned dataset revisions (for example, main)", + ) parser.add_argument("--reference-revision", default=None) parser.add_argument("--track", choices=("open", "full"), default="open") parser.add_argument("--cache-dir", type=Path, default=Path.home() / ".cache/frontierchallenge") @@ -316,18 +334,21 @@ def main() -> int: ) parser.add_argument("--skip-image", action="store_true", help="verify datasets only") args = parser.parse_args() + solve_revision, reference_revision = select_revisions( + dataset_release, args.revision, args.reference_revision + ) token = os.environ.get("HF_TOKEN") solve = resolve_dataset( args.solve_source, - args.revision, + solve_revision, args.cache_dir, token=token, ignore_patterns=["images/*.tar.zst"], ) reference = resolve_dataset( args.reference_source, - args.reference_revision or args.revision, + reference_revision, args.cache_dir, token=token, ) @@ -340,7 +361,7 @@ def main() -> int: load_hf_image_archive( solve=solve, solve_source=args.solve_source, - revision=args.revision, + revision=solve_revision, cache_dir=args.cache_dir, archive_config=image_manifest["images"]["open"]["hf_archive"], token=token, @@ -357,7 +378,8 @@ def main() -> int: reference=reference, track=args.track, open_image=default_image, - revision=args.revision, + solve_revision=solve_revision, + reference_revision=reference_revision, ) print(f"ready: {len(selected)} tasks; configuration written to {args.config.resolve()}") return 0 diff --git a/benchmarks/frontierchallenge/scripts/task_selection.py b/benchmarks/frontierchallenge/scripts/task_selection.py new file mode 100755 index 0000000..aafa65c --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/task_selection.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Resolve the exact FrontierChallenge task set for an evaluation run.""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +from dataclasses import dataclass +from pathlib import Path + +VALID_ENVIRONMENTS = {"open", "licensed-orca"} + + +@dataclass(frozen=True) +class SelectedTask: + task_id: str + environment: str + path: Path + + +def select_tasks( + tasks_root: Path, + *, + track: str, + include: tuple[str, ...] = (), + exclude: tuple[str, ...] = (), + registry_path: Path | None = None, +) -> list[SelectedTask]: + """Select from verified solve tasks, never from persistent staging.""" + if track not in {"open", "full"}: + raise ValueError(f"unsupported track: {track}") + + registry_environments: dict[str, str] | None = None + if registry_path is not None: + registry = json.loads(registry_path.read_text(encoding="utf-8")) + registry_environments = {} + for record in registry.get("tasks", []): + task_id = record.get("id") + environment = record.get("image") + if not isinstance(task_id, str) or environment not in VALID_ENVIRONMENTS: + raise ValueError(f"invalid registry task record: {record!r}") + if task_id in registry_environments: + raise ValueError(f"duplicate registry task ID: {task_id}") + registry_environments[task_id] = environment + + selected: list[SelectedTask] = [] + for task_path in sorted(tasks_root.iterdir()): + if not task_path.is_dir() or not (task_path / "task.toml").is_file(): + continue + metadata_path = task_path / "task.json" + if not metadata_path.is_file(): + raise ValueError(f"task metadata missing: {metadata_path}") + metadata = json.loads(metadata_path.read_text(encoding="utf-8")) + task_id = metadata.get("task_id") + environment = metadata.get("environment") + if task_id != task_path.name: + raise ValueError(f"task ID/path mismatch: {task_path.name} != {task_id!r}") + if environment not in VALID_ENVIRONMENTS: + raise ValueError(f"invalid environment for {task_id}: {environment!r}") + if registry_environments is not None: + registered = registry_environments.get(task_id) + if registered != environment: + raise ValueError( + f"task/registry environment mismatch for {task_id}: " + f"{environment!r} != {registered!r}" + ) + if track == "open" and environment != "open": + continue + if include and not any(fnmatch.fnmatchcase(task_id, pattern) for pattern in include): + continue + if any(fnmatch.fnmatchcase(task_id, pattern) for pattern in exclude): + continue + selected.append(SelectedTask(task_id, environment, task_path.resolve())) + return selected + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tasks-root", type=Path, required=True) + parser.add_argument("--registry", type=Path, required=True) + parser.add_argument("--track", choices=("open", "full"), required=True) + parser.add_argument("--include", action="append", default=[]) + parser.add_argument("--exclude", action="append", default=[]) + args = parser.parse_args() + + try: + selected = select_tasks( + args.tasks_root, + track=args.track, + include=tuple(args.include), + exclude=tuple(args.exclude), + registry_path=args.registry, + ) + except (OSError, ValueError, json.JSONDecodeError) as exc: + parser.error(str(exc)) + if not selected: + parser.error("task selection is empty") + for task in selected: + path = str(task.path) + if any(character in path for character in ("\t", "\n", "\r")): + parser.error(f"task path contains a control character: {task.path}") + print(f"{task.task_id}\t{task.environment}\t{path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_contract.py b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py new file mode 100644 index 0000000..96fc7db --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py @@ -0,0 +1,29 @@ +from pathlib import Path + +SCRIPT = Path(__file__).parents[1] / "scripts" / "run_eval.sh" + + +def test_orca_preflight_uses_declared_environment_not_instruction_text(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'EFFECTIVE_TASK_ENVS[$index]' in text + assert "grep -qil 'orca'" not in text + + +def test_harbor_receives_exact_effective_task_ids(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'for task_id in "${EFFECTIVE_TASK_IDS[@]}"' in text + assert 'INCLUDE_ARGS+=("--include-task-name" "$task_id")' in text + + +def test_staging_dereferences_hugging_face_cache_symlinks(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'cp -aL "$task_dir" "$dest"' in text + + +def test_verifier_work_is_limited_to_effective_task_directories(): + text = SCRIPT.read_text(encoding="utf-8") + + assert text.count('for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"') == 2 diff --git a/benchmarks/frontierchallenge/tests/test_setup_release.py b/benchmarks/frontierchallenge/tests/test_setup_release.py index 38bd35f..b1b2391 100644 --- a/benchmarks/frontierchallenge/tests/test_setup_release.py +++ b/benchmarks/frontierchallenge/tests/test_setup_release.py @@ -1,12 +1,11 @@ from __future__ import annotations import json -from pathlib import Path import sys +from pathlib import Path from types import SimpleNamespace import pytest - import setup_release @@ -91,16 +90,44 @@ def test_write_config_quotes_evaluator_paths(tmp_path): reference=tmp_path / "reference package", track="open", open_image="example/image@sha256:123", - revision="main", + solve_revision="solve-sha", + reference_revision="reference-sha", ) text = config.read_text() assert "FRONTIER_SOLVE_DIR=" in text assert "'" in text assert "example/image@sha256:123" in text + assert "FRONTIER_SOLVE_REVISION=solve-sha" in text + assert "FRONTIER_REFERENCE_REVISION=reference-sha" in text assert "FRONTIER_IMAGE_SOURCE" not in text +def test_select_revisions_uses_independent_pins_by_default(): + release = { + "solve": {"revision": "solve-sha"}, + "reference": {"revision": "reference-sha"}, + } + + assert setup_release.select_revisions(release, None, None) == ( + "solve-sha", + "reference-sha", + ) + + +def test_revision_override_preserves_legacy_both_dataset_behavior(): + release = { + "solve": {"revision": "solve-sha"}, + "reference": {"revision": "reference-sha"}, + } + + assert setup_release.select_revisions(release, "main", None) == ("main", "main") + assert setup_release.select_revisions(release, "solve-next", "ref-next") == ( + "solve-next", + "ref-next", + ) + + def test_validate_orca_runtime_accepts_wrapper_contract(monkeypatch): calls = [] diff --git a/benchmarks/frontierchallenge/tests/test_task_selection.py b/benchmarks/frontierchallenge/tests/test_task_selection.py new file mode 100644 index 0000000..6f089b0 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_task_selection.py @@ -0,0 +1,103 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import task_selection + + +def make_task(root: Path, task_id: str, environment: str, instruction: str = "") -> Path: + task = root / task_id + (task / "environment").mkdir(parents=True) + (task / "task.toml").write_text('schema_version = "1.1"\n') + (task / "task.json").write_text( + json.dumps({"task_id": task_id, "environment": environment}) + ) + (task / "instruction.md").write_text(instruction) + (task / "environment" / "Dockerfile").write_text("FROM example/open\n") + return task + + +def test_open_task_can_mention_orca_without_requiring_licensed_runtime(tmp_path): + make_task( + tmp_path, + "task_098_orca_claisen_thermochemistry", + "open", + "Read the supplied ORCA output files; do not execute ORCA.", + ) + + selected = task_selection.select_tasks(tmp_path, track="open") + + assert [(task.task_id, task.environment) for task in selected] == [ + ("task_098_orca_claisen_thermochemistry", "open") + ] + + +def test_open_track_excludes_declared_licensed_tasks(tmp_path): + make_task(tmp_path, "task_011_open", "open") + make_task(tmp_path, "task_199_orca", "licensed-orca") + + assert [task.task_id for task in task_selection.select_tasks(tmp_path, track="open")] == [ + "task_011_open" + ] + assert [task.task_id for task in task_selection.select_tasks(tmp_path, track="full")] == [ + "task_011_open", + "task_199_orca", + ] + + +def test_include_and_exclude_use_glob_semantics(tmp_path): + for task_id in ("task_011_alpha", "task_012_beta", "task_199_orca"): + make_task(tmp_path, task_id, "open") + + selected = task_selection.select_tasks( + tmp_path, + track="open", + include=("task_0*",), + exclude=("*_beta",), + ) + + assert [task.task_id for task in selected] == ["task_011_alpha"] + + +def test_selection_reads_solve_source_not_stale_stage(tmp_path): + solve = tmp_path / "solve" + stale_stage = tmp_path / "stage" + make_task(solve, "task_011_open", "open") + make_task(stale_stage, "task_199_orca", "licensed-orca") + + selected = task_selection.select_tasks(solve, track="open") + + assert [task.task_id for task in selected] == ["task_011_open"] + + +def test_invalid_environment_is_rejected(tmp_path): + make_task(tmp_path, "task_011_bad", "orca-by-text-search") + + with pytest.raises(ValueError, match="invalid environment"): + task_selection.select_tasks(tmp_path, track="full") + + +def test_task_environment_must_match_registry_commitment(tmp_path): + make_task(tmp_path, "task_098_orca_claisen_thermochemistry", "open") + registry = tmp_path / "source_registry.json" + registry.write_text( + json.dumps( + { + "tasks": [ + { + "id": "task_098_orca_claisen_thermochemistry", + "image": "licensed-orca", + } + ] + } + ) + ) + + with pytest.raises(ValueError, match="task/registry environment mismatch"): + task_selection.select_tasks( + tmp_path, + track="full", + registry_path=registry, + ) From 4fbefe92a918ff203d2f062010473becea27bdb9 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 14:03:48 +0800 Subject: [PATCH 02/10] fix: align public setup and FrontierChallenge full-score metrics --- README.md | 6 +- apodex/tests/test_deployment_config.py | 41 +++++- benchmarks/frontierchallenge/README.md | 8 +- .../docs/huggingface-release.md | 5 + .../frontierchallenge/docs/quickstart.md | 9 +- benchmarks/frontierchallenge/docs/running.md | 15 ++- benchmarks/frontierchallenge/docs/scoring.md | 40 ++++-- .../frontierchallenge/docs/submitting.md | 5 +- .../frontierchallenge/docs/task-format.md | 5 +- .../frontierchallenge/release/datasets.json | 4 +- .../frontierchallenge/scripts/run_eval.sh | 22 +++- .../scripts/summarize_results.py | 84 ++++++++---- benchmarks/frontierchallenge/site/index.html | 2 +- .../tests/test_run_eval_integration.py | 112 ++++++++++++++++ .../tests/test_summarize_results.py | 121 ++++++++++++++++++ compose.yaml | 13 +- docker/run.sh | 7 +- docs/install/docker.md | 35 +++-- 18 files changed, 459 insertions(+), 75 deletions(-) mode change 100644 => 100755 benchmarks/frontierchallenge/scripts/summarize_results.py create mode 100644 benchmarks/frontierchallenge/tests/test_run_eval_integration.py create mode 100644 benchmarks/frontierchallenge/tests/test_summarize_results.py diff --git a/README.md b/README.md index 77273a9..359dcf7 100644 --- a/README.md +++ b/README.md @@ -199,11 +199,13 @@ Chinese-speaking macOS users can use the ## Containers and local models -Pre-built `linux/amd64` and `linux/arm64` images are published to the GitHub -Container Registry, so no local Python environment is needed: +Build the Docker image from this checkout; no local Python environment is +needed. The organization's GHCR image is private, so the public quickstart +uses a local image and does not require registry credentials: ```bash cp .env.example .env +docker compose build docker compose run --rm agent ``` diff --git a/apodex/tests/test_deployment_config.py b/apodex/tests/test_deployment_config.py index b412924..f9ebf3e 100644 --- a/apodex/tests/test_deployment_config.py +++ b/apodex/tests/test_deployment_config.py @@ -1,8 +1,12 @@ from __future__ import annotations +import os import re +import shutil +import subprocess from pathlib import Path +import pytest import yaml ROOT = Path(__file__).resolve().parents[2] @@ -30,13 +34,13 @@ def _dotenv(name: str) -> dict[str, str]: return values -def test_default_compose_pulls_release_image_and_preserves_cli_state() -> None: +def test_default_compose_uses_public_local_build_and_preserves_cli_state() -> None: compose = _yaml("compose.yaml") agent = compose["services"]["agent"] - assert "build" not in agent - assert IMAGE in agent["image"] - assert agent["pull_policy"] == "always" + assert agent["build"]["context"] == "." + assert _fallback(agent["image"]) == "frontieragent:local" + assert agent["pull_policy"] == "never" assert agent["environment"]["APODEX_IN_CONTAINER"] == "1" assert agent["environment"]["SANDBOX_BACKEND"] == "container" assert "security_opt" not in agent @@ -59,16 +63,37 @@ def test_default_compose_pulls_release_image_and_preserves_cli_state() -> None: assert agent["environment"]["APODEX_WORKSPACE_LINK"] == "/workspace" -def test_development_compose_is_the_only_compose_file_that_builds() -> None: +def test_development_compose_forces_rebuild_of_public_local_image() -> None: compose = _yaml("compose.yaml") development = _yaml("compose.dev.yaml") - assert all("build" not in service for service in compose["services"].values()) + for service in compose["services"].values(): + assert service["build"]["context"] == "." + assert service["pull_policy"] == "never" + assert _fallback(service["image"]) == "frontieragent:local" assert development["services"]["agent"]["build"]["context"] == "." assert development["services"]["eval"]["build"]["context"] == "." assert development["services"]["agent"]["pull_policy"] == "build" +@pytest.mark.parametrize("arguments,service", [(["-p", "hello"], "agent"), (["eval", "--limit", "5"], "eval")]) +def test_docker_helper_reuses_image_from_repository_directory(tmp_path, arguments, service): + repo = tmp_path / "repo" + (repo / "docker").mkdir(parents=True) + shutil.copy2(ROOT / "docker/run.sh", repo / "docker/run.sh") + binaries = tmp_path / "bin" + binaries.mkdir() + fake = binaries / "docker" + fake.write_text('#!/bin/sh\npwd\nprintf "%s\\n" "$@"\n') + fake.chmod(0o755) + result = subprocess.run(["bash", str(repo / "docker/run.sh"), *arguments], + cwd=tmp_path, capture_output=True, text=True, check=True, + env={**os.environ, "PATH": f"{binaries}:{os.environ['PATH']}"}) + lines = result.stdout.splitlines() + assert Path(lines[0]).resolve() == repo.resolve() + assert lines[1:7] == ["compose", "run", "--pull", "never", "--rm", service] + + def test_sglang_compose_mounts_an_optional_local_checkpoint_read_only() -> None: compose = _yaml("compose.sglang.yaml") model = compose["services"]["model"] @@ -311,7 +336,9 @@ def test_user_docs_use_the_published_registry_name() -> None: # rather than repeating them. paths = [ROOT / "docs/install/docker.md", ROOT / "compose.yaml"] - assert all(IMAGE in path.read_text(encoding="utf-8") for path in paths) + assert IMAGE in paths[0].read_text(encoding="utf-8") + assert "frontieragent:local" in paths[1].read_text(encoding="utf-8") + assert "docker compose build" in paths[0].read_text(encoding="utf-8") assert "docs/install/docker.md" in (ROOT / "README.md").read_text(encoding="utf-8") # The hyphenated spelling is not the published name and must appear nowhere. diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index 73f59b1..40ee3e4 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -105,8 +105,12 @@ cat results/harbor///verifier/reward.json cat results/harbor//summary.json ``` -`passed` is the task's own pass decision; do not derive it from a global score -threshold. `task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms +Official **Pass Rate** counts completed evaluations with **`task_score == 1.0`** +over all 97 tasks. **Score** is the mean `task_score` over 97, multiplied by 100. +Missing or failed evaluations contribute zero. The raw reward's `passed` is a +legacy task-specific decision and is not used for official metrics; summaries +retain it as `native_passed`. No rounding or `>= 0.999` tolerance is applied. +`task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms that grading finished. See [Quickstart](docs/quickstart.md) for credentials and expected output, and [Scoring](docs/scoring.md) for aggregate reporting. diff --git a/benchmarks/frontierchallenge/docs/huggingface-release.md b/benchmarks/frontierchallenge/docs/huggingface-release.md index 9304e09..583bfee 100644 --- a/benchmarks/frontierchallenge/docs/huggingface-release.md +++ b/benchmarks/frontierchallenge/docs/huggingface-release.md @@ -53,3 +53,8 @@ The top-level Hugging Face `README.md` is intentionally outside edited without changing the benchmark payload. Task files, task-level READMEs, registries, manifests, image artifacts, and verifier archives remain covered by the checksum manifests and registry commitments. + +After payload edits, regenerate the affected checksum entries and run both +bundled verification tools before publishing. Dataset-card-only edits require +no payload checksum change. Update the runtime's pinned HF revisions after +publishing; see [Scoring](scoring.md) for the metric contract shared by both cards. diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 85f3b63..226093e 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -157,10 +157,15 @@ cat results/harbor///verifier/reward.json ``` - `evaluation_complete = 1` means the verifier finished; -- `passed` is the task's own pass decision and must not be recomputed from a - global threshold; +- official Pass Rate counts completed `task_score == 1.0` evaluations over 97; +- raw `passed` is the native grader's diagnostic decision, ignored by official + metrics and retained as `native_passed` in summaries; - `task_score` is a continuous score in `[0, 1]`. +Score is the mean `task_score` over 97, times 100. Missing and failed evaluations +contribute zero. No `>= 0.999` tolerance or rounding is used for Pass Rate. See +[Scoring](scoring.md) for subsets, repeated attempts, and historical results. + The job aggregate is: ```bash diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index 1de1e0a..83f698d 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -49,7 +49,9 @@ missing. See [Docker](providers/docker.md). ``` Use disjoint include lists and distinct job names to shard across machines. -`summarize_results.py` accepts multiple job directories and merges them. +`summarize_results.py` accepts one job directory at a time; it does not merge +shards. A shard's report is partial, not a separate full-benchmark result. +Do not average shard Pass Rates as though they were full runs. Concurrency must fit both machine resources and model-provider rate limits. Start with one task, then increase gradually. @@ -67,6 +69,10 @@ lowering longer ones: Use a separate stage directory for each concurrent run that changes timeouts. Verifier timeouts use `--verifier-timeout-multiplier` (default 40). +Legacy staging caches containing Hugging Face symlinks are automatically +rebuilt once. Only the effective include/exclude selection is staged, checked +for ORCA, unsealed, and passed to Harbor; leftover stage directories are ignored. + ## Resume and results Reusing a job name resumes completed work when the requested task set matches: @@ -82,6 +88,13 @@ python3 scripts/summarize_results.py results/harbor/ cat results/harbor//summary.json ``` +Official Pass Rate requires completed `task_score == 1.0`; native `passed` is +diagnostic only. The default denominator is 97, including missing tasks. +See [Scoring](scoring.md) for partial scores and explicitly labeled subsets. +Automatic summaries use the runner's current selection. If a job directory +contains trials from an older selection, use repeatable `--task-id ` with +the standalone summarizer, or start a fresh job directory. + Before a long run, verify one task reaches `evaluation_complete = 1`, confirm the selected backend in the startup banner, and confirm the local ORCA runtime before selecting the full track. diff --git a/benchmarks/frontierchallenge/docs/scoring.md b/benchmarks/frontierchallenge/docs/scoring.md index 229fd4a..5404b9e 100644 --- a/benchmarks/frontierchallenge/docs/scoring.md +++ b/benchmarks/frontierchallenge/docs/scoring.md @@ -2,12 +2,17 @@ FrontierChallenge reports two numbers over a fixed denominator of 97 tasks: -- **Pass Rate:** tasks whose verifier writes `passed = 1`, divided by 97. +- **Pass Rate:** completed evaluations with `task_score == 1.0`, divided by 97. - **Score:** the mean of `task_score` across all 97, usually reported times 100. Unrun tasks and harness failures count as zero in the fixed denominator. The -summarizer marks an incomplete run as partial instead of averaging only the -tasks that happened to finish. +summarizer marks an incomplete run as partial while retaining the denominator +97. It never drops missing or failed tasks from the headline metrics. + +Equality is exact: `0.999` and `0.999999` do not pass. No rounding, epsilon, +per-task pass threshold, or native `passed` decision enters this calculation. +`evaluation_complete == 1` is required for a valid score. Invalid scores +(non-numeric, non-finite, or outside `[0, 1]`) contribute zero and are flagged. ## Authoritative fields @@ -15,19 +20,38 @@ Each trial writes `verifier/reward.json`: | Field | Meaning | |---|---| -| `passed` | the task's own pass decision; do not derive it from a global threshold | +| `passed` | legacy native grader decision; diagnostic only, ignored by official metrics | | `task_score` | score from 0 to 1 | | `evaluation_complete` | whether verification completed | -The 97 verifiers do not share one pass threshold. For full-mark counts, use -`task_score >= 0.999`; judge averaging can produce a value just below 1. +The encrypted native graders and their partial-credit rubrics are unchanged. +Their historical thresholds vary across tasks. The summarizer now writes +official full-score decisions to `summary.csv`/`summary.json` as `passed` and +preserves the raw reward field separately as `native_passed`. A completed +score of 1 passes even when the native flag is false; a score of 0.8 fails +even when the native flag is true. Raw `verifier/reward.json` stays unchanged. + +This policy is identified by `metric_definition: exact-full-score` in the +summary. Recompute historical results from raw rewards before comparing them; +results computed with native thresholds or `>= 0.999` are not interchangeable. -Summarize one or more Harbor job directories with: +Summarize a Harbor job directory with: ```bash python3 scripts/summarize_results.py results/harbor/ ``` +Use one predeclared attempt per task. Duplicate task trials are rejected rather +than silently counted twice or selected by their score. For a deliberately +separate subset report (for example, the 81-task open track), specify +`--expected-total 81` and report that denominator explicitly; it is not the +97-task headline metric. `--expected-total` must not be reduced to the number +of tasks that happened to succeed. Missing tasks still contribute zero. + +For repeated trials, use `run_eval.sh --n-attempts N --no-summary` and summarize +each predeclared attempt separately. Harbor's own aggregation of the raw +`passed` reward is not the official Pass Rate; use this runtime's summary. + ## Verifiers and judges Each task has a frozen verifier. Deterministic checks validate submitted files, @@ -66,7 +90,7 @@ solve-side hash and an encrypted-verifier hash. Setup refuses mixed releases. Report: - denominator 97, with missing tasks counted as zero; -- Pass Rate from `passed`, not from a new threshold; +- Pass Rate from completed `task_score == 1.0`, ignoring native `passed`; - mean `task_score` times 100; - agent, model, judge model, and judge repetitions; - pinned Docker image identity and ORCA version for full-track runs; diff --git a/benchmarks/frontierchallenge/docs/submitting.md b/benchmarks/frontierchallenge/docs/submitting.md index c893cea..f7e17d1 100644 --- a/benchmarks/frontierchallenge/docs/submitting.md +++ b/benchmarks/frontierchallenge/docs/submitting.md @@ -31,8 +31,9 @@ ORCA version; see [Scoring](scoring.md). listed if it is labelled as one, with the count of attempted tasks. It cannot be listed as a score over a smaller denominator. -**The `passed` field, not a threshold.** See -[Scoring](scoring.md#the-two-headline-numbers). +**Full-score Pass Rate.** Count only completed evaluations with +`task_score == 1.0`; ignore the raw native `passed` field and do not round scores. +See [Scoring](scoring.md). **Judge configuration stated.** `gpt-5.6-sol`, `reasoning_effort=high`, `JUDGE_REPEATS=3` with `--no-judge-override` is the definitional setting. Any diff --git a/benchmarks/frontierchallenge/docs/task-format.md b/benchmarks/frontierchallenge/docs/task-format.md index ca35ea7..3bd6299 100644 --- a/benchmarks/frontierchallenge/docs/task-format.md +++ b/benchmarks/frontierchallenge/docs/task-format.md @@ -97,7 +97,10 @@ the task's source/provenance definition, and should be described that way: 1. runs the LLM judge `JUDGE_REPEATS` times, if the task has one (77 do), 2. combines them with `statistics.fmean`, 3. calls the task's own grader with that value as the rubric component, -4. emits `task_score` and `passed`. +4. emits `task_score` and the legacy native `passed` diagnostic. + +Official Pass Rate is computed by the summarizer from completed +`task_score == 1.0` evaluations, independently of native thresholds. The full tree is stored in the gated dataset's `verifier.fcref`, including the solved reference run, reference fixtures, grader source, judge prompt, and diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json index f7bdea4..2130c7a 100644 --- a/benchmarks/frontierchallenge/release/datasets.json +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -2,10 +2,10 @@ "schema_version": 1, "solve": { "repo": "apodex/FrontierChallenge", - "revision": "8a73bb02a5709aa007e1a146d0cdc5d36ef5ea4d" + "revision": "99288f3848ad9ddd3df0d4e19ec6a8a2e4865ea9" }, "reference": { "repo": "apodex/FrontierChallenge-reference", - "revision": "59fbf007ad5c6816caf7d379eef32d9fb384edea" + "revision": "9e6ee51d24b0d1f5435dd7d450119cab0a7cd49c" } } diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index 433fa47..817dc0e 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -72,7 +72,8 @@ Options: (default: unset, follows --n-concurrent). A lower cap here than --n-concurrent adds headroom against agent-setup timeouts at high concurrency. - --n-attempts N Attempts per task (default: 1) + --n-attempts N Attempts per task (default: 1); use --no-summary + for repeated trials, then report each attempt separately --job-name NAME Harbor job name (default: derived) --jobs-dir PATH Where Harbor writes results (default: results/harbor) --stage-dir PATH Scratch dir for evaluator-owned task copies @@ -161,6 +162,11 @@ if [[ "$TRACK" != "open" && "$TRACK" != "full" ]]; then exit 1 fi +if [[ "$N_ATTEMPTS" != "1" && "$NO_SUMMARY" -eq 0 ]]; then + echo "error: automatic metrics require one attempt per task; use --no-summary for repeated trials" >&2 + exit 1 +fi + if [[ -z "$SOLVE_DIR" || ! -d "$SOLVE_DIR" ]]; then echo "error: a downloaded solve-side HF package is required." >&2 echo " Run ./scripts/setup.sh, pass --solve-dir PATH, or set FRONTIER_SOLVE_DIR." >&2 @@ -282,7 +288,9 @@ for index in "${!EFFECTIVE_TASK_IDS[@]}"; do task_dir="${EFFECTIVE_TASK_SOURCES[$index]}" dest="$STAGE_DIR/$task_id" EFFECTIVE_TASK_DIRS+=("$dest") - source_identity="$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE" + # Invalidate stages produced by the legacy cp -a runner. Their nested HF + # cache-relative symlinks can be broken even when task.toml remains readable. + source_identity="dereferenced-v2|$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE" if [[ "$FORCE_RESTAGE" -eq 0 && -f "$dest/task.toml" \ && -f "$dest/instruction.md" && ! -e "$dest/statement.fcref" \ && -f "$dest/.frontier-source" ]] \ @@ -540,17 +548,21 @@ else --path "$STAGE_DIR" \ --env docker \ --agent "$AGENT" --model "$MODEL" \ - --n-attempts "$N_ATTEMPTS" --n-concurrent "$N_CONCURRENT" "${CONCURRENT_AGENTS_ARGS[@]}" \ + --n-attempts "$N_ATTEMPTS" --n-concurrent "$N_CONCURRENT" ${CONCURRENT_AGENTS_ARGS[@]+"${CONCURRENT_AGENTS_ARGS[@]}"} \ --verifier-timeout-multiplier "$VERIFIER_TIMEOUT_MULTIPLIER" \ --agent-setup-timeout-multiplier "$AGENT_SETUP_TIMEOUT_MULTIPLIER" \ --artifact /app/output \ --jobs-dir "$JOBS_DIR" --job-name "$JOB_NAME" \ --env-file "$ENV_FILE" \ - "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \ + ${AGENT_KWARG_ARGS[@]+"${AGENT_KWARG_ARGS[@]}"} "${INCLUDE_ARGS[@]}" ${VERIFIER_ENV_ARGS[@]+"${VERIFIER_ENV_ARGS[@]}"} \ --yes fi if [[ "$NO_SUMMARY" -eq 0 ]]; then echo "== Summarizing $JOBS_DIR/$JOB_NAME ==" - python3 scripts/summarize_results.py "$JOBS_DIR/$JOB_NAME" + SUMMARY_TASK_ARGS=() + for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + SUMMARY_TASK_ARGS+=(--task-id "$task_id") + done + python3 scripts/summarize_results.py "$JOBS_DIR/$JOB_NAME" "${SUMMARY_TASK_ARGS[@]}" fi diff --git a/benchmarks/frontierchallenge/scripts/summarize_results.py b/benchmarks/frontierchallenge/scripts/summarize_results.py old mode 100644 new mode 100755 index 24e8a84..a9d8ebf --- a/benchmarks/frontierchallenge/scripts/summarize_results.py +++ b/benchmarks/frontierchallenge/scripts/summarize_results.py @@ -14,12 +14,32 @@ import argparse import csv import json +import math from pathlib import Path from typing import Any #: 100 tasks minus task_065 (needs a GPU) and task_047 / task_049 (their #: deterministic grader only runs in a nested container, unavailable here). EXPECTED_TOTAL_TASKS = 97 +METRIC_DEFINITION = "exact-full-score" + + +def valid_score(score: Any) -> bool: + return ( + isinstance(score, (int, float)) + and not isinstance(score, bool) + and 0.0 <= score <= 1.0 + and math.isfinite(score) + ) + + +def official_pass(row: dict[str, Any]) -> bool: + """Exact full credit on a completed evaluation; native passed is diagnostic.""" + return ( + row.get("evaluation_complete") == 1 + and valid_score(row.get("task_score")) + and row["task_score"] == 1.0 + ) #: Judge stderr wording (both casings occur across the frozen graders) for the #: one "verifier failed" cause that is not an infrastructure fault at all: the @@ -38,7 +58,8 @@ def load_json(path: Path) -> dict[str, Any] | None: try: - return json.loads(path.read_text(encoding="utf-8")) + value = json.loads(path.read_text(encoding="utf-8")) + return value if isinstance(value, dict) else None except (OSError, json.JSONDecodeError): return None @@ -97,6 +118,7 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: "model": agent.get("model_name"), "task_score": None, "passed": None, + "native_passed": None, "evaluation_complete": None, "error": None, "scored_zero_missing_artifact": False, @@ -104,8 +126,12 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: reward = load_json(trial_dir / "verifier" / "reward.json") if reward is not None: row["task_score"] = reward.get("task_score") - row["passed"] = reward.get("passed") + row["native_passed"] = reward.get("passed") row["evaluation_complete"] = reward.get("evaluation_complete") + if not valid_score(row["task_score"]): + row["task_score"] = None + row["evaluation_complete"] = 0.0 + row["error"] = "invalid task_score: expected a finite number in [0, 1]" else: row["error"] = "no reward.json (trial errored or is still running)" if row["evaluation_complete"] != 1.0 and missing_required_artifact(trial_dir): @@ -114,35 +140,38 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: row["evaluation_complete"] = 1.0 row["scored_zero_missing_artifact"] = True row["error"] = "required artifact missing from submission - genuine 0" + row["passed"] = float(official_pass(row)) rows.append(row) return rows def summarize(rows: list[dict[str, Any]], expected_total: int) -> dict[str, Any]: - # A trial can report task_score=0.0/passed=0.0 alongside - # evaluation_complete=0.0 - that 0 is the verifier crashing (e.g. a Judge - # request the endpoint rejected, a missing-artifact hard error), not a - # real graded attempt. Averaging it in as if it were a genuine 0/100 - # silently deflates the score. Only evaluation_complete=1 rows are a - # trustworthy signal; everything else is reported separately so a run's - # real grading coverage is visible instead of hidden inside the average. - graded = [r for r in rows if r["evaluation_complete"] == 1.0 and r["task_score"] is not None] - ungraded = [ - r for r in rows - if r["task_score"] is not None and r["evaluation_complete"] != 1.0 - ] + if expected_total <= 0: + raise ValueError("expected_total must be positive") + ids = [r["task_id"] for r in rows] + if len(set(ids)) != len(ids): + raise ValueError("duplicate task trials: select one predeclared attempt per task; do not select by score") + if len(rows) > expected_total: + raise ValueError("more tasks than expected_total; check the evaluation scope") + # Incomplete/invalid evaluations contribute zero to the fixed denominator, + # but remain visible as failures rather than successful zero-score grading. + graded = [r for r in rows if r["evaluation_complete"] == 1.0 and valid_score(r["task_score"])] + ungraded = [r for r in rows if r["evaluation_complete"] == 0.0] n = len(rows) - mean_score = (sum(r["task_score"] for r in graded) / len(graded)) if graded else None - pass_rate = (sum(r["passed"] for r in graded) / len(graded)) if graded else None + mean_score = sum(r["task_score"] for r in graded) / expected_total + n_passed = sum(official_pass(r) for r in rows) + pass_rate = n_passed / expected_total return { + "metric_definition": METRIC_DEFINITION, + "pass_rule": "evaluation_complete == 1 and task_score == 1.0", "n_tasks_expected": expected_total, "n_trials_found": n, "n_graded": len(graded), "n_zero_missing_artifact": sum(1 for r in graded if r["scored_zero_missing_artifact"]), "n_verifier_failed": len(ungraded), - "n_missing_or_errored": n - len(graded) - len(ungraded), + "n_missing_or_errored": expected_total - len(graded) - len(ungraded), "complete": n == expected_total and len(graded) == expected_total, - "n_passed": sum(1 for r in graded if r["passed"]) if graded else 0, + "n_passed": n_passed, "pass_rate": pass_rate, "mean_task_score": mean_score, "mean_task_score_100": (mean_score * 100) if mean_score is not None else None, @@ -161,6 +190,7 @@ def write_outputs(job_dir: Path, rows: list[dict[str, Any]], summary: dict[str, "model", "task_score", "passed", + "native_passed", "evaluation_complete", "scored_zero_missing_artifact", "error", @@ -191,17 +221,17 @@ def print_report(job_dir: Path, summary: dict[str, Any]) -> None: print( f"NOTE: {summary['n_verifier_failed']} trial(s) had the verifier itself " "fail (e.g. a Judge request the endpoint rejected, a missing-artifact " - "hard error) - excluded from Pass Rate / Score below, not counted as 0." + "hard error) - contribute 0 to the fixed-denominator metrics below." ) if summary["pass_rate"] is not None: print( - f"Pass Rate: {summary['n_passed']}/{summary['n_graded']} " + f"Pass Rate (task_score == 1): {summary['n_passed']}/{summary['n_tasks_expected']} " f"= {summary['pass_rate'] * 100:.1f}%" ) else: print("Pass Rate: n/a (no graded trials)") if summary["mean_task_score_100"] is not None: - print(f"Mean Score: {summary['mean_task_score_100']:.1f} / 100 (over {summary['n_graded']} graded trials)") + print(f"Mean Score: {summary['mean_task_score_100']:.1f} / 100 (denominator {summary['n_tasks_expected']})") else: print("Mean Score: n/a (no graded trials)") print(f"Wrote {job_dir / 'summary.csv'} and {job_dir / 'summary.json'}") @@ -210,6 +240,10 @@ def print_report(job_dir: Path, summary: dict[str, Any]) -> None: def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("job_dir", type=Path, help="Harbor jobs-dir job, e.g. results/harbor/") + parser.add_argument( + "--task-id", action="append", + help="only include this task ID (repeatable); excludes stale trials from a reused job", + ) parser.add_argument( "--expected-total", type=int, @@ -223,7 +257,13 @@ def main() -> None: raise SystemExit(f"not a directory: {job_dir}") rows = collect_rows(job_dir) - summary = summarize(rows, args.expected_total) + if args.task_id is not None: + selected = set(args.task_id) + rows = [row for row in rows if row["task_id"] in selected] + try: + summary = summarize(rows, args.expected_total) + except ValueError as exc: + parser.error(str(exc)) write_outputs(job_dir, rows, summary) print_report(job_dir, summary) diff --git a/benchmarks/frontierchallenge/site/index.html b/benchmarks/frontierchallenge/site/index.html index 49789cc..277793a 100644 --- a/benchmarks/frontierchallenge/site/index.html +++ b/benchmarks/frontierchallenge/site/index.html @@ -60,7 +60,7 @@

FrontierChallenge

-

13 model–scaffold configurations · 97 tasks · Pass Rate = native Score ≥ 99.9.

+

13 model–scaffold configurations · 97 tasks · Historical table: Pass Rate used native Score ≥ 99.9. The current evaluation policy requires completed task_score == 1.0 (no tolerance); these historical numbers have not been recomputed.

diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py new file mode 100644 index 0000000..c5187cf --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py @@ -0,0 +1,112 @@ +"""Exercise the shell runner with real archives and stubbed Docker/Harbor APIs. + +This verifies orchestration, not container isolation or real model execution. +""" +from __future__ import annotations + +import json +import os +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest +import reference_archive + +ROOT = Path(__file__).parents[1] +OPEN = "task_098_orca_claisen_thermochemistry" +LICENSED = "task_199_orca" + + +@pytest.fixture +def runtime(tmp_path): + runtime = tmp_path / "runtime" + shutil.copytree(ROOT / "scripts", runtime / "scripts") + solve, reference = tmp_path / "solve", tmp_path / "reference" + records = [] + for name, environment in [(OPEN, "open"), (LICENSED, "licensed-orca")]: + task = solve / "tasks" / name + (task / "environment/data").mkdir(parents=True) + (task / "task.toml").write_text('[agent]\ntimeout_sec = 600\n') + (task / "instruction.md").write_text("Analyze supplied ORCA output.") + (task / "task.json").write_text(json.dumps({ + "task_id": name, "environment": environment, "source_task_sha256": "fixture", + }, indent=2)) + (task / "environment/Dockerfile").write_text("FROM fixture/open\n") + blob = tmp_path / f"{name}.blob" + blob.write_text("precomputed output") + (task / "environment/data/output.txt").symlink_to(os.path.relpath(blob, task / "environment/data")) + verifier = reference / "tasks" / name + (verifier / "tests").mkdir(parents=True) + (verifier / "tests/test.sh").write_text("#!/bin/sh\nexit 0\n") + reference_archive.pack(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"], + "frontier-challenge-reference", force=True) + reference_archive.strip(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"]) + records.append({"id": name, "image": environment}) + registry = {"name": "fixture", "n_tasks": 2, "tasks": records} + for root, filename in [(runtime, "registry.json"), (solve, "source_registry.json"), + (reference, "source_registry.json")]: + (root / filename).write_text(json.dumps(registry)) + envfile = tmp_path / "credentials.env" + envfile.write_text("ANTHROPIC_API_KEY=fixture-not-a-key\n") + binaries = tmp_path / "bin" + binaries.mkdir() + (binaries / "python3").symlink_to(sys.executable) + docker = binaries / "docker" + docker.write_text('#!/bin/sh\nprintf "%s\\n" "$*" >> "$DOCKER_LOG"\n' + 'case "$*" in *orca-user-local*) exit 1;; esac\nexit 0\n') + docker.chmod(0o755) + harbor = binaries / "harbor" + harbor.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "$HARBOR_LOG"\n') + harbor.chmod(0o755) + env = {**os.environ, "PATH": f"{binaries}:{os.environ['PATH']}", + "DOCKER_LOG": str(tmp_path / "docker.log"), "HARBOR_LOG": str(tmp_path / "harbor.log"), + "FRONTIER_CONFIG_FILE": str(tmp_path / "no-config")} + stage = tmp_path / "stage" + cmd = ["bash", str(runtime / "scripts/run_eval.sh"), "--agent", "claude-code", + "--model", "fixture", "--solve-dir", str(solve), "--reference-dir", str(reference), + "--stage-dir", str(stage), "--env-file", str(envfile), "--no-judge-override", "--no-summary"] + return tmp_path, solve, stage, cmd, env + + +def test_open_selection_migrates_legacy_cache_and_ignores_stale_orca(runtime): + root, solve, stage, cmd, env = runtime + stale = stage / OPEN + shutil.copytree(solve / "tasks" / OPEN, stale, symlinks=True) + # Exactly the identity the old runner wrote, but nested input symlink now broken. + (stale / ".frontier-source").write_text( + f"{solve}|source_task_sha256:fixture|frontierchallenge/cpu-open:2026.08\n") + assert (stale / "environment/data/output.txt").is_symlink() + assert not (stale / "environment/data/output.txt").exists() + shutil.copytree(solve / "tasks" / LICENSED, stage / LICENSED, symlinks=True) + result = subprocess.run(cmd + ["--track", "open"], env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert "Staged 1 task(s), reused 0" in result.stdout + assert (stale / "environment/data/output.txt").read_text() == "precomputed output" + assert not (stale / "environment/data/output.txt").is_symlink() + assert (stale / "tests/test.sh").is_file() + assert "orca-user-local" not in (root / "docker.log").read_text() + args = (root / "harbor.log").read_text().splitlines() + assert args[args.index("--include-task-name") + 1] == OPEN + assert LICENSED not in args + result = subprocess.run(cmd, env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert "Staged 0 task(s), reused 1" in result.stdout + + +def test_full_track_exclusion_prevents_orca_preflight(runtime): + root, _, _, cmd, env = runtime + result = subprocess.run(cmd + ["--track", "full", "--exclude-task-name", LICENSED], + env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert "orca-user-local" not in (root / "docker.log").read_text() + + +def test_selected_licensed_task_still_requires_orca(runtime): + root, _, _, cmd, env = runtime + result = subprocess.run(cmd + ["--track", "full", "--include-task-name", LICENSED], + env=env, capture_output=True, text=True, check=False) + assert result.returncode != 0 + assert "1 task(s) require ORCA" in result.stderr + assert not (root / "harbor.log").exists() diff --git a/benchmarks/frontierchallenge/tests/test_summarize_results.py b/benchmarks/frontierchallenge/tests/test_summarize_results.py new file mode 100644 index 0000000..0d43912 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_summarize_results.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +import csv +import json +import subprocess +import sys +from pathlib import Path + +import pytest +import summarize_results as metrics + + +def trial(root, name, score, native_passed=1, complete=1): + directory = root / name + (directory / "verifier").mkdir(parents=True) + (directory / "config.json").write_text(json.dumps({"task": {"path": f"/tasks/{name}"}})) + (directory / "verifier/reward.json").write_text(json.dumps({ + "task_score": score, "passed": native_passed, "evaluation_complete": complete, + })) + return directory + + +@pytest.mark.parametrize("score,native,complete,passed", [ + (1.0, 0, 1, 1), (1, 1, 1, 1), (0.999999999, 1, 1, 0), + (0.999, 1, 1, 0), (0.8, 1, 1, 0), (0.0, 1, 1, 0), + (1.0, 1, 0, 0), (1.0, 1, None, 0), +]) +def test_official_pass_ignores_native_threshold(tmp_path, score, native, complete, passed): + directory = trial(tmp_path, "task_one", score, native, complete) + before = (directory / "verifier/reward.json").read_bytes() + row = metrics.collect_rows(tmp_path)[0] + assert row["passed"] == passed + assert row["native_passed"] == native + assert (directory / "verifier/reward.json").read_bytes() == before + assert metrics.summarize([row], 97)["pass_rate"] == passed / 97 + + +@pytest.mark.parametrize("score", [None, "1", True, -0.1, 1.001, 10**400, float("inf"), float("nan")]) +def test_invalid_scores_are_not_passes_or_partial_credit(tmp_path, score): + trial(tmp_path, "task_bad", score) + rows = metrics.collect_rows(tmp_path) + assert rows[0]["task_score"] is None + assert rows[0]["evaluation_complete"] == 0 + summary = metrics.summarize(rows, 97) + assert summary["n_passed"] == summary["mean_task_score"] == 0 + assert summary["n_verifier_failed"] == 1 + + +def test_fixed_denominator_and_partial_credit(tmp_path): + trial(tmp_path, "task_one", 1, 0) + trial(tmp_path, "task_two", 0.7, 1) + trial(tmp_path, "task_failed", 1, 1, 0) + summary = metrics.summarize(metrics.collect_rows(tmp_path), 97) + assert summary["n_passed"] == 1 + assert summary["pass_rate"] == 1 / 97 + assert summary["mean_task_score_100"] == pytest.approx(170 / 97) + assert summary["n_missing_or_errored"] == 94 + assert summary["complete"] is False + assert summary["metric_definition"] == "exact-full-score" + + +def test_empty_run_is_zero_and_incomplete(): + summary = metrics.summarize([], 97) + assert summary["n_missing_or_errored"] == 97 + assert summary["pass_rate"] == summary["mean_task_score"] == 0 + assert not summary["complete"] + + +def test_repeated_trials_and_wrong_denominator_are_rejected(tmp_path): + trial(tmp_path, "task_one", 1) + rows = metrics.collect_rows(tmp_path) + with pytest.raises(ValueError, match="duplicate"): + metrics.summarize(rows * 2, 97) + with pytest.raises(ValueError, match="positive"): + metrics.summarize(rows, 0) + trial(tmp_path, "task_two", 0) + with pytest.raises(ValueError, match="more tasks"): + metrics.summarize(metrics.collect_rows(tmp_path), 1) + + +def test_missing_submission_still_counts_as_zero(tmp_path): + directory = trial(tmp_path, "task_one", 0, 0, 0) + (directory / "verifier/native_grader.stderr.txt").write_text("submission directory not found") + summary = metrics.summarize(metrics.collect_rows(tmp_path), 1) + assert summary["complete"] is True + assert summary["n_zero_missing_artifact"] == 1 + assert summary["pass_rate"] == 0 + + +def test_cli_outputs_official_and_native_flags_separately(tmp_path): + trial(tmp_path, "task_one", 0.9, 1) + trial(tmp_path, "task_two", 1, 0) + subprocess.run([sys.executable, str(Path(metrics.__file__)), str(tmp_path), + "--expected-total", "2"], check=True, capture_output=True) + summary = json.loads((tmp_path / "summary.json").read_text()) + assert summary["pass_rate"] == 0.5 + assert summary["mean_task_score_100"] == 95 + assert summary["complete"] is True + with (tmp_path / "summary.csv").open() as handle: + rows = list(csv.DictReader(handle)) + assert [(r["passed"], r["native_passed"]) for r in rows] == [("0.0", "1"), ("1.0", "0")] + + +def test_cli_excludes_stale_trials_from_reused_job(tmp_path): + trial(tmp_path, "task_selected", 0.5) + trial(tmp_path, "task_stale", 1) + subprocess.run([sys.executable, str(Path(metrics.__file__)), str(tmp_path), + "--task-id", "task_selected"], check=True, capture_output=True) + summary = json.loads((tmp_path / "summary.json").read_text()) + assert summary["n_trials_found"] == 1 + assert summary["n_passed"] == 0 + assert summary["mean_task_score"] == 0.5 / 97 + + +@pytest.mark.parametrize("value", [[], 1, "broken", None]) +def test_non_object_reward_is_reported_as_missing(tmp_path, value): + directory = trial(tmp_path, "task_bad", 1) + (directory / "verifier/reward.json").write_text(json.dumps(value)) + rows = metrics.collect_rows(tmp_path) + assert rows[0]["task_score"] is None + assert metrics.summarize(rows, 97)["n_passed"] == 0 diff --git a/compose.yaml b/compose.yaml index d8b0644..226840b 100644 --- a/compose.yaml +++ b/compose.yaml @@ -1,7 +1,10 @@ services: agent: - image: ${FRONTIER_AGENT_IMAGE:-ghcr.io/apodexai/frontieragent:latest} - pull_policy: always + image: ${FRONTIER_AGENT_IMAGE:-frontieragent:local} + # Public users build locally; the organization's GHCR image is private. + pull_policy: never + build: + context: . env_file: - path: .env required: false @@ -41,8 +44,10 @@ services: tty: true eval: - image: ${FRONTIER_AGENT_IMAGE:-ghcr.io/apodexai/frontieragent:latest} - pull_policy: always + image: ${FRONTIER_AGENT_IMAGE:-frontieragent:local} + pull_policy: never + build: + context: . env_file: - path: .env required: false diff --git a/docker/run.sh b/docker/run.sh index c355f3b..53286cc 100755 --- a/docker/run.sh +++ b/docker/run.sh @@ -8,6 +8,9 @@ set -euo pipefail # ./docker/run.sh eval --limit 1 # Run benchmark evaluation repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$repo_root" +# Run `docker compose build` once and after source updates. The default local +# image is reused without requiring access to the organization's private GHCR. export APODEX_HOST_UID="$(id -u)" export APODEX_HOST_GID="$(id -g)" export APODEX_LOCAL_UTC_OFFSET="$(date +%z)" @@ -18,10 +21,10 @@ if [ "${1:-}" = "eval" ]; then shift # `docker compose run SERVICE ARGS...` replaces the service command, so # include the required benchmark defaults before forwarding overrides. - exec docker compose run --rm eval \ + exec docker compose run --pull never --rm eval \ --benchmark browsecomp \ --out /app/results/smoke \ "$@" else - exec docker compose run --rm agent "$@" + exec docker compose run --pull never --rm agent "$@" fi diff --git a/docs/install/docker.md b/docs/install/docker.md index 4a8f433..6041a6e 100644 --- a/docs/install/docker.md +++ b/docs/install/docker.md @@ -1,16 +1,17 @@ # Run FrontierAgent in Docker -FrontierAgent publishes pre-built `linux/amd64` and `linux/arm64` images to the -GitHub Container Registry. Using them requires no local Python environment and -no system dependencies beyond Docker itself. The default `compose.yaml` pulls -that published image; it does not build the repository locally. +The public Docker workflow builds FrontierAgent from this checkout for your +host architecture (`linux/amd64` or `linux/arm64`). It requires Docker and +network access to download build dependencies, but no local Python environment. +The organization's GHCR package is private. The default `compose.yaml` builds +`frontieragent:local` and reuses it with `pull_policy: never`. This page covers the CPU agent container. For a **local NVIDIA model server**, the GPU belongs to a separate SGLang container or process — use [Docker SGLang on a Linux NVIDIA host](linux-nvidia.md) or [Native SGLang without nested Docker](linux-nvidia-native.md) instead. -## One-click Compose run +## Build and run with Compose `compose.yaml` marks `.env` as optional, which requires Docker Compose 2.24 or newer; older versions reject the file outright. @@ -19,6 +20,7 @@ newer; older versions reject the file outright. git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent cp .env.example .env +docker compose build # Interactive CLI docker compose run --rm agent @@ -35,7 +37,7 @@ Its named state volume is retained for legacy sessions. Attached inputs are copied into a separate volume that tools can only read. See [run artifacts and timestamps](../run-artifacts.md) for the on-disk layout. -The convenience helper wraps the same thing: +After building, the convenience helper reuses that local image: ```bash ./docker/run.sh -p "analyze repository structure" @@ -44,16 +46,21 @@ The convenience helper wraps the same thing: ## Pin a release or another image -Set `FRONTIER_AGENT_IMAGE` before running Compose: +Users with access to the private GHCR package can explicitly log in and pull +an image. Set the same `FRONTIER_AGENT_IMAGE` when running Compose, which then +uses the downloaded image without pulling or building it. Do not run +`docker compose build` with this override: that would replace the local tag. ```bash +docker login ghcr.io +docker pull ghcr.io/apodexai/frontieragent:latest FRONTIER_AGENT_IMAGE=ghcr.io/apodexai/frontieragent:latest \ docker compose run --rm agent -p "explain pyproject.toml" ``` ## Direct `docker run` -Compose is the supported path; this is the equivalent for environments that +First run `docker compose build`. Compose is the supported path; this is the equivalent for environments that cannot use it. The environment variables and mounts are not optional — they are what tells the runtime it is inside a container and where the three sandbox roots live. @@ -77,7 +84,7 @@ docker run --rm -it \ -v frontier-agent-state:/root/.apodex \ -v frontier-agent-config:/root/.config/apodex \ -w /workspace \ - ghcr.io/apodexai/frontieragent:latest \ + frontieragent:local \ -p "explain main workflow" ``` @@ -87,26 +94,26 @@ For a terminal deployment accessed over SSH: 1. Provision an EC2 or ECS Linux instance with Docker and the Compose plugin. 2. Clone this repository and create `.env` from `.env.example`. -3. Pull and launch the pre-built container: +3. Build and launch the local container: ```bash git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent cp .env.example .env # Edit .env, then: -docker compose pull agent +docker compose build docker compose run --rm agent ``` The container itself is disposable; Compose persists sessions, configuration, attachments, and deliverables in volumes or the checked-out workspace. Pull the -image again to upgrade. This is an interactive SSH/TUI deployment, not a +source updates and rebuild with `docker compose build` to upgrade. This is an interactive SSH/TUI deployment, not a long-running HTTP service. ## Build from the current checkout -To run your own changes instead of the published image, add the development -override: +The default build already uses the checkout. For development, the override +forces a rebuild on each launch: ```bash cp .env.example .env From 9c577d6d927d73fe52b16a4226e085667237798a Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 14:10:56 +0800 Subject: [PATCH 03/10] fix(frontierchallenge): emit only the full-score pass decision --- benchmarks/frontierchallenge/README.md | 6 +-- .../frontierchallenge/docs/quickstart.md | 3 +- benchmarks/frontierchallenge/docs/running.md | 4 +- benchmarks/frontierchallenge/docs/scoring.md | 23 +++++++----- .../frontierchallenge/docs/submitting.md | 2 +- .../frontierchallenge/docs/task-format.md | 7 ++-- .../frontierchallenge/release/datasets.json | 4 +- .../scripts/apply_score_policy.py | 37 +++++++++++++++++++ .../frontierchallenge/scripts/run_eval.sh | 3 ++ .../scripts/summarize_results.py | 7 +--- .../tests/test_apply_score_policy.py | 27 ++++++++++++++ .../tests/test_run_eval_integration.py | 5 +++ .../tests/test_summarize_results.py | 8 ++-- 13 files changed, 105 insertions(+), 31 deletions(-) create mode 100644 benchmarks/frontierchallenge/scripts/apply_score_policy.py create mode 100644 benchmarks/frontierchallenge/tests/test_apply_score_policy.py diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index 40ee3e4..aca47fc 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -107,9 +107,9 @@ cat results/harbor//summary.json Official **Pass Rate** counts completed evaluations with **`task_score == 1.0`** over all 97 tasks. **Score** is the mean `task_score` over 97, multiplied by 100. -Missing or failed evaluations contribute zero. The raw reward's `passed` is a -legacy task-specific decision and is not used for official metrics; summaries -retain it as `native_passed`. No rounding or `>= 0.999` tolerance is applied. +Missing or failed evaluations contribute zero. `passed` has this single meaning +in both `reward.json` and summaries; no alternate pass field is emitted. +No rounding or `>= 0.999` tolerance is applied. `task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms that grading finished. See [Quickstart](docs/quickstart.md) for credentials and expected output, and [Scoring](docs/scoring.md) for aggregate reporting. diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 226093e..e9c91ca 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -158,8 +158,7 @@ cat results/harbor///verifier/reward.json - `evaluation_complete = 1` means the verifier finished; - official Pass Rate counts completed `task_score == 1.0` evaluations over 97; -- raw `passed` is the native grader's diagnostic decision, ignored by official - metrics and retained as `native_passed` in summaries; +- `passed` uses this same full-score rule in both rewards and summaries; - `task_score` is a continuous score in `[0, 1]`. Score is the mean `task_score` over 97, times 100. Missing and failed evaluations diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index 83f698d..874c394 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -88,8 +88,8 @@ python3 scripts/summarize_results.py results/harbor/ cat results/harbor//summary.json ``` -Official Pass Rate requires completed `task_score == 1.0`; native `passed` is -diagnostic only. The default denominator is 97, including missing tasks. +Official Pass Rate requires completed `task_score == 1.0`; `passed` means the +same thing in rewards and summaries. The default denominator is 97, including missing tasks. See [Scoring](scoring.md) for partial scores and explicitly labeled subsets. Automatic summaries use the runner's current selection. If a job directory contains trials from an older selection, use repeatable `--task-id ` with diff --git a/benchmarks/frontierchallenge/docs/scoring.md b/benchmarks/frontierchallenge/docs/scoring.md index 5404b9e..c746581 100644 --- a/benchmarks/frontierchallenge/docs/scoring.md +++ b/benchmarks/frontierchallenge/docs/scoring.md @@ -20,16 +20,19 @@ Each trial writes `verifier/reward.json`: | Field | Meaning | |---|---| -| `passed` | legacy native grader decision; diagnostic only, ignored by official metrics | +| `passed` | 1 only when evaluation completed and `task_score == 1.0`; otherwise 0 | | `task_score` | score from 0 to 1 | | `evaluation_complete` | whether verification completed | -The encrypted native graders and their partial-credit rubrics are unchanged. -Their historical thresholds vary across tasks. The summarizer now writes -official full-score decisions to `summary.csv`/`summary.json` as `passed` and -preserves the raw reward field separately as `native_passed`. A completed -score of 1 passes even when the native flag is false; a score of 0.8 fails -even when the native flag is true. Raw `verifier/reward.json` stays unchanged. +There is only one pass field, `passed`, with the same meaning in +`verifier/reward.json`, `summary.csv`, and `summary.json`. A completed score +of 1 passes; a score of 0.8 does not. No alternate pass field is emitted. + +After authenticating and unsealing the reference, the runtime applies the +full-score rule to the staged reward adapter before Harbor runs it. The +encrypted reference archives and partial-credit rubrics remain unchanged. +The summarizer also derives `passed` from score and completion when processing +older results, discarding their old pass decision rather than copying it. This policy is identified by `metric_definition: exact-full-score` in the summary. Recompute historical results from raw rewards before comparing them; @@ -49,8 +52,8 @@ separate subset report (for example, the 81-task open track), specify of tasks that happened to succeed. Missing tasks still contribute zero. For repeated trials, use `run_eval.sh --n-attempts N --no-summary` and summarize -each predeclared attempt separately. Harbor's own aggregation of the raw -`passed` reward is not the official Pass Rate; use this runtime's summary. +each predeclared attempt separately. Use this runtime's summary for the +fixed-denominator headline metric; Harbor may aggregate only attempted trials. ## Verifiers and judges @@ -90,7 +93,7 @@ solve-side hash and an encrypted-verifier hash. Setup refuses mixed releases. Report: - denominator 97, with missing tasks counted as zero; -- Pass Rate from completed `task_score == 1.0`, ignoring native `passed`; +- Pass Rate from completed `task_score == 1.0`; - mean `task_score` times 100; - agent, model, judge model, and judge repetitions; - pinned Docker image identity and ORCA version for full-track runs; diff --git a/benchmarks/frontierchallenge/docs/submitting.md b/benchmarks/frontierchallenge/docs/submitting.md index f7e17d1..d048c8b 100644 --- a/benchmarks/frontierchallenge/docs/submitting.md +++ b/benchmarks/frontierchallenge/docs/submitting.md @@ -32,7 +32,7 @@ listed if it is labelled as one, with the count of attempted tasks. It cannot be listed as a score over a smaller denominator. **Full-score Pass Rate.** Count only completed evaluations with -`task_score == 1.0`; ignore the raw native `passed` field and do not round scores. +`task_score == 1.0`; do not apply per-task thresholds or round scores. See [Scoring](scoring.md). **Judge configuration stated.** `gpt-5.6-sol`, `reasoning_effort=high`, diff --git a/benchmarks/frontierchallenge/docs/task-format.md b/benchmarks/frontierchallenge/docs/task-format.md index 3bd6299..8cca06c 100644 --- a/benchmarks/frontierchallenge/docs/task-format.md +++ b/benchmarks/frontierchallenge/docs/task-format.md @@ -97,10 +97,11 @@ the task's source/provenance definition, and should be described that way: 1. runs the LLM judge `JUDGE_REPEATS` times, if the task has one (77 do), 2. combines them with `statistics.fmean`, 3. calls the task's own grader with that value as the rubric component, -4. emits `task_score` and the legacy native `passed` diagnostic. +4. emits `task_score` and `passed`, which is 1 only for a completed full score. -Official Pass Rate is computed by the summarizer from completed -`task_score == 1.0` evaluations, independently of native thresholds. +The runtime applies this single pass rule to the staged reward adapter after +unsealing. Official Pass Rate is computed by the summarizer from completed +`task_score == 1.0` evaluations, independently of per-task thresholds. The full tree is stored in the gated dataset's `verifier.fcref`, including the solved reference run, reference fixtures, grader source, judge prompt, and diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json index 2130c7a..80bedf4 100644 --- a/benchmarks/frontierchallenge/release/datasets.json +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -2,10 +2,10 @@ "schema_version": 1, "solve": { "repo": "apodex/FrontierChallenge", - "revision": "99288f3848ad9ddd3df0d4e19ec6a8a2e4865ea9" + "revision": "8d20e59f504b8c988825b4f0d7b91cf0d71b1444" }, "reference": { "repo": "apodex/FrontierChallenge-reference", - "revision": "9e6ee51d24b0d1f5435dd7d450119cab0a7cd49c" + "revision": "c66054ef3e76eec2b53367448aa81429ee8bb209" } } diff --git a/benchmarks/frontierchallenge/scripts/apply_score_policy.py b/benchmarks/frontierchallenge/scripts/apply_score_policy.py new file mode 100644 index 0000000..115ac01 --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/apply_score_policy.py @@ -0,0 +1,37 @@ +"""Set the sole pass rule in an evaluator-owned, freshly unsealed task. + +The encrypted reference and partial-credit rubric are not modified. Apply this +after archive verification/unsealing and before Harbor reads the verifier. +Fail closed if a future reference changes the known reward adapter contract. +""" +from __future__ import annotations + +import argparse +from pathlib import Path + +OLD_RULE = '"passed": 1.0 if passed is True else 0.0,' +FULL_SCORE_RULE = ( + '"passed": 1.0 if complete and float(score or 0.0) / 100.0 == 1.0 else 0.0,' +) + + +def apply_policy(task_dir: Path) -> None: + adapter = task_dir / "tests" / "run_frontier_verifier.py" + text = adapter.read_text(encoding="utf-8") + if text.count(OLD_RULE) != 1 or FULL_SCORE_RULE in text: + raise ValueError(f"unsupported or already modified reward adapter: {adapter}") + adapter.write_text(text.replace(OLD_RULE, FULL_SCORE_RULE), encoding="utf-8") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("task_dir", type=Path) + args = parser.parse_args() + try: + apply_policy(args.task_dir) + except (OSError, ValueError) as exc: + parser.error(str(exc)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index 817dc0e..62e8d88 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -533,6 +533,9 @@ if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "FATAL: $(basename "$task_dir") has no verifier entrypoint after unsealing." >&2 exit 1 fi + # Emit only the benchmark-wide full-score pass decision, before Harbor + # reads reward.json. No alternate native pass metric is retained. + python3 "$ROOT/scripts/apply_score_policy.py" "$task_dir" unsealed=$((unsealed + 1)) done echo "Unsealed $unsealed task(s)." diff --git a/benchmarks/frontierchallenge/scripts/summarize_results.py b/benchmarks/frontierchallenge/scripts/summarize_results.py index a9d8ebf..c1cf1a4 100755 --- a/benchmarks/frontierchallenge/scripts/summarize_results.py +++ b/benchmarks/frontierchallenge/scripts/summarize_results.py @@ -2,7 +2,7 @@ """Aggregate a Harbor job directory into a Pass Rate + Score summary. Reads each trial's ``config.json`` (to recover which task it ran) and -``verifier/reward.json`` (``task_score``, ``passed``, ``evaluation_complete``) +``verifier/reward.json`` (``task_score``, ``evaluation_complete``) under a Harbor jobs-dir job, and writes a per-task CSV plus an overall JSON summary next to it. @@ -34,7 +34,7 @@ def valid_score(score: Any) -> bool: def official_pass(row: dict[str, Any]) -> bool: - """Exact full credit on a completed evaluation; native passed is diagnostic.""" + """The sole pass decision: exact full credit on a completed evaluation.""" return ( row.get("evaluation_complete") == 1 and valid_score(row.get("task_score")) @@ -118,7 +118,6 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: "model": agent.get("model_name"), "task_score": None, "passed": None, - "native_passed": None, "evaluation_complete": None, "error": None, "scored_zero_missing_artifact": False, @@ -126,7 +125,6 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: reward = load_json(trial_dir / "verifier" / "reward.json") if reward is not None: row["task_score"] = reward.get("task_score") - row["native_passed"] = reward.get("passed") row["evaluation_complete"] = reward.get("evaluation_complete") if not valid_score(row["task_score"]): row["task_score"] = None @@ -190,7 +188,6 @@ def write_outputs(job_dir: Path, rows: list[dict[str, Any]], summary: dict[str, "model", "task_score", "passed", - "native_passed", "evaluation_complete", "scored_zero_missing_artifact", "error", diff --git a/benchmarks/frontierchallenge/tests/test_apply_score_policy.py b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py new file mode 100644 index 0000000..b8ae249 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +import pytest +from apply_score_policy import OLD_RULE, apply_policy + + +@pytest.mark.parametrize("score,native,complete,expected", [ + (100, False, True, 1), (80, True, True, 0), (99.999, True, True, 0), + (100, True, False, 0), (0, True, True, 0), (100, None, True, 1), +]) +def test_staged_reward_uses_only_full_score(tmp_path, score, native, complete, expected): + (tmp_path / "tests").mkdir() + adapter = tmp_path / "tests/run_frontier_verifier.py" + adapter.write_text('reward = {' + OLD_RULE + '}\n') + apply_policy(tmp_path) + namespace = {"score": score, "passed": native, "complete": complete} + exec(compile(adapter.read_text(), str(adapter), "exec"), namespace) + assert namespace["reward"] == {"passed": float(expected)} + + +def test_unknown_adapter_fails_closed(tmp_path): + (tmp_path / "tests").mkdir() + adapter = tmp_path / "tests/run_frontier_verifier.py" + adapter.write_text("reward = {}\n") + with pytest.raises(ValueError, match="unsupported"): + apply_policy(tmp_path) + assert adapter.read_text() == "reward = {}\n" diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py index c5187cf..c205a3b 100644 --- a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py +++ b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py @@ -40,6 +40,9 @@ def runtime(tmp_path): verifier = reference / "tasks" / name (verifier / "tests").mkdir(parents=True) (verifier / "tests/test.sh").write_text("#!/bin/sh\nexit 0\n") + (verifier / "tests/run_frontier_verifier.py").write_text( + 'reward = {"passed": 1.0 if passed is True else 0.0,}\n' + ) reference_archive.pack(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"], "frontier-challenge-reference", force=True) reference_archive.strip(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"]) @@ -86,6 +89,8 @@ def test_open_selection_migrates_legacy_cache_and_ignores_stale_orca(runtime): assert (stale / "environment/data/output.txt").read_text() == "precomputed output" assert not (stale / "environment/data/output.txt").is_symlink() assert (stale / "tests/test.sh").is_file() + adapter = (stale / "tests/run_frontier_verifier.py").read_text() + assert '"passed": 1.0 if complete and float(score or 0.0) / 100.0 == 1.0 else 0.0' in adapter assert "orca-user-local" not in (root / "docker.log").read_text() args = (root / "harbor.log").read_text().splitlines() assert args[args.index("--include-task-name") + 1] == OPEN diff --git a/benchmarks/frontierchallenge/tests/test_summarize_results.py b/benchmarks/frontierchallenge/tests/test_summarize_results.py index 0d43912..42ce6bc 100644 --- a/benchmarks/frontierchallenge/tests/test_summarize_results.py +++ b/benchmarks/frontierchallenge/tests/test_summarize_results.py @@ -30,7 +30,7 @@ def test_official_pass_ignores_native_threshold(tmp_path, score, native, complet before = (directory / "verifier/reward.json").read_bytes() row = metrics.collect_rows(tmp_path)[0] assert row["passed"] == passed - assert row["native_passed"] == native + assert "native_passed" not in row assert (directory / "verifier/reward.json").read_bytes() == before assert metrics.summarize([row], 97)["pass_rate"] == passed / 97 @@ -87,7 +87,7 @@ def test_missing_submission_still_counts_as_zero(tmp_path): assert summary["pass_rate"] == 0 -def test_cli_outputs_official_and_native_flags_separately(tmp_path): +def test_cli_outputs_only_one_pass_flag(tmp_path): trial(tmp_path, "task_one", 0.9, 1) trial(tmp_path, "task_two", 1, 0) subprocess.run([sys.executable, str(Path(metrics.__file__)), str(tmp_path), @@ -98,7 +98,9 @@ def test_cli_outputs_official_and_native_flags_separately(tmp_path): assert summary["complete"] is True with (tmp_path / "summary.csv").open() as handle: rows = list(csv.DictReader(handle)) - assert [(r["passed"], r["native_passed"]) for r in rows] == [("0.0", "1"), ("1.0", "0")] + assert [r["passed"] for r in rows] == ["0.0", "1.0"] + assert all("native_passed" not in r for r in rows) + assert "native_passed" not in (tmp_path / "summary.json").read_text() def test_cli_excludes_stale_trials_from_reused_job(tmp_path): From 7f924eb7d0b0ce21543fd0d2f18c4e5be2970085 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 14:23:31 +0800 Subject: [PATCH 04/10] fix(frontierchallenge): pass scores strictly above 0.999 --- benchmarks/frontierchallenge/README.md | 4 ++-- .../frontierchallenge/docs/quickstart.md | 6 +++--- benchmarks/frontierchallenge/docs/running.md | 2 +- benchmarks/frontierchallenge/docs/scoring.md | 21 +++++++++++-------- .../frontierchallenge/docs/submitting.md | 4 ++-- .../frontierchallenge/docs/task-format.md | 4 ++-- .../frontierchallenge/release/datasets.json | 4 ++-- .../scripts/apply_score_policy.py | 8 +++---- .../frontierchallenge/scripts/run_eval.sh | 2 +- .../scripts/summarize_results.py | 10 ++++----- benchmarks/frontierchallenge/site/index.html | 2 +- .../tests/test_apply_score_policy.py | 8 +++++-- .../tests/test_run_eval_integration.py | 2 +- .../tests/test_summarize_results.py | 6 ++++-- 14 files changed, 46 insertions(+), 37 deletions(-) diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index aca47fc..009e388 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -105,11 +105,11 @@ cat results/harbor///verifier/reward.json cat results/harbor//summary.json ``` -Official **Pass Rate** counts completed evaluations with **`task_score == 1.0`** +Official **Pass Rate** counts completed evaluations with **`task_score > 0.999`** over all 97 tasks. **Score** is the mean `task_score` over 97, multiplied by 100. Missing or failed evaluations contribute zero. `passed` has this single meaning in both `reward.json` and summaries; no alternate pass field is emitted. -No rounding or `>= 0.999` tolerance is applied. +The comparison is strict and uses unrounded scores: exactly `0.999` does not pass. `task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms that grading finished. See [Quickstart](docs/quickstart.md) for credentials and expected output, and [Scoring](docs/scoring.md) for aggregate reporting. diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index e9c91ca..3584fc9 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -157,12 +157,12 @@ cat results/harbor///verifier/reward.json ``` - `evaluation_complete = 1` means the verifier finished; -- official Pass Rate counts completed `task_score == 1.0` evaluations over 97; -- `passed` uses this same full-score rule in both rewards and summaries; +- official Pass Rate counts completed `task_score > 0.999` evaluations over 97; +- `passed` uses this same strict threshold in both rewards and summaries; - `task_score` is a continuous score in `[0, 1]`. Score is the mean `task_score` over 97, times 100. Missing and failed evaluations -contribute zero. No `>= 0.999` tolerance or rounding is used for Pass Rate. See +contribute zero. Use unrounded scores: exactly `0.999` does not pass. See [Scoring](scoring.md) for subsets, repeated attempts, and historical results. The job aggregate is: diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index 874c394..e52f1fb 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -88,7 +88,7 @@ python3 scripts/summarize_results.py results/harbor/ cat results/harbor//summary.json ``` -Official Pass Rate requires completed `task_score == 1.0`; `passed` means the +Official Pass Rate requires completed `task_score > 0.999`; `passed` means the same thing in rewards and summaries. The default denominator is 97, including missing tasks. See [Scoring](scoring.md) for partial scores and explicitly labeled subsets. Automatic summaries use the runner's current selection. If a job directory diff --git a/benchmarks/frontierchallenge/docs/scoring.md b/benchmarks/frontierchallenge/docs/scoring.md index c746581..530a6f0 100644 --- a/benchmarks/frontierchallenge/docs/scoring.md +++ b/benchmarks/frontierchallenge/docs/scoring.md @@ -2,15 +2,18 @@ FrontierChallenge reports two numbers over a fixed denominator of 97 tasks: -- **Pass Rate:** completed evaluations with `task_score == 1.0`, divided by 97. +- **Pass Rate:** completed evaluations with `task_score > 0.999`, divided by 97. - **Score:** the mean of `task_score` across all 97, usually reported times 100. Unrun tasks and harness failures count as zero in the fixed denominator. The summarizer marks an incomplete run as partial while retaining the denominator 97. It never drops missing or failed tasks from the headline metrics. -Equality is exact: `0.999` and `0.999999` do not pass. No rounding, epsilon, -per-task pass threshold, or native `passed` decision enters this calculation. +The comparison is strict: `0.999` does not pass; `0.9991` and `1.0` pass. +Use unrounded scores without an additional epsilon or per-task threshold. +Compare the stored `task_score`, not a rounded display: for example, +`0.9990000000000001` passes even if displayed as `0.999`. Score normalization +and partial-credit arithmetic are unchanged. `evaluation_complete == 1` is required for a valid score. Invalid scores (non-numeric, non-finite, or outside `[0, 1]`) contribute zero and are flagged. @@ -20,23 +23,23 @@ Each trial writes `verifier/reward.json`: | Field | Meaning | |---|---| -| `passed` | 1 only when evaluation completed and `task_score == 1.0`; otherwise 0 | +| `passed` | 1 only when evaluation completed and valid `task_score > 0.999`; otherwise 0 | | `task_score` | score from 0 to 1 | | `evaluation_complete` | whether verification completed | There is only one pass field, `passed`, with the same meaning in `verifier/reward.json`, `summary.csv`, and `summary.json`. A completed score -of 1 passes; a score of 0.8 does not. No alternate pass field is emitted. +of 0.9991 passes; a score of 0.999 does not. No alternate pass field is emitted. After authenticating and unsealing the reference, the runtime applies the -full-score rule to the staged reward adapter before Harbor runs it. The +strict score threshold to the staged reward adapter before Harbor runs it. The encrypted reference archives and partial-credit rubrics remain unchanged. The summarizer also derives `passed` from score and completion when processing older results, discarding their old pass decision rather than copying it. -This policy is identified by `metric_definition: exact-full-score` in the +This policy is identified by `metric_definition: score-gt-0.999` in the summary. Recompute historical results from raw rewards before comparing them; -results computed with native thresholds or `>= 0.999` are not interchangeable. +results computed with per-task thresholds, `== 1.0`, or `>= 0.999` are not interchangeable. Summarize a Harbor job directory with: @@ -93,7 +96,7 @@ solve-side hash and an encrypted-verifier hash. Setup refuses mixed releases. Report: - denominator 97, with missing tasks counted as zero; -- Pass Rate from completed `task_score == 1.0`; +- Pass Rate from completed `task_score > 0.999`; - mean `task_score` times 100; - agent, model, judge model, and judge repetitions; - pinned Docker image identity and ORCA version for full-track runs; diff --git a/benchmarks/frontierchallenge/docs/submitting.md b/benchmarks/frontierchallenge/docs/submitting.md index d048c8b..22e8510 100644 --- a/benchmarks/frontierchallenge/docs/submitting.md +++ b/benchmarks/frontierchallenge/docs/submitting.md @@ -31,8 +31,8 @@ ORCA version; see [Scoring](scoring.md). listed if it is labelled as one, with the count of attempted tasks. It cannot be listed as a score over a smaller denominator. -**Full-score Pass Rate.** Count only completed evaluations with -`task_score == 1.0`; do not apply per-task thresholds or round scores. +**Pass Rate.** Count only completed evaluations with +`task_score > 0.999`; do not apply per-task thresholds or round scores. See [Scoring](scoring.md). **Judge configuration stated.** `gpt-5.6-sol`, `reasoning_effort=high`, diff --git a/benchmarks/frontierchallenge/docs/task-format.md b/benchmarks/frontierchallenge/docs/task-format.md index 8cca06c..88b099d 100644 --- a/benchmarks/frontierchallenge/docs/task-format.md +++ b/benchmarks/frontierchallenge/docs/task-format.md @@ -97,11 +97,11 @@ the task's source/provenance definition, and should be described that way: 1. runs the LLM judge `JUDGE_REPEATS` times, if the task has one (77 do), 2. combines them with `statistics.fmean`, 3. calls the task's own grader with that value as the rubric component, -4. emits `task_score` and `passed`, which is 1 only for a completed full score. +4. emits `task_score` and `passed`, which is 1 only for a completed valid score above 0.999. The runtime applies this single pass rule to the staged reward adapter after unsealing. Official Pass Rate is computed by the summarizer from completed -`task_score == 1.0` evaluations, independently of per-task thresholds. +`task_score > 0.999` evaluations, independently of per-task thresholds. The full tree is stored in the gated dataset's `verifier.fcref`, including the solved reference run, reference fixtures, grader source, judge prompt, and diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json index 80bedf4..2452b23 100644 --- a/benchmarks/frontierchallenge/release/datasets.json +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -2,10 +2,10 @@ "schema_version": 1, "solve": { "repo": "apodex/FrontierChallenge", - "revision": "8d20e59f504b8c988825b4f0d7b91cf0d71b1444" + "revision": "9ab8b7f6a19b2995ad43ad944af94b6d1e70b2b9" }, "reference": { "repo": "apodex/FrontierChallenge-reference", - "revision": "c66054ef3e76eec2b53367448aa81429ee8bb209" + "revision": "aaeb6d84d26f8c4174bf88ef4fc74e03013021f8" } } diff --git a/benchmarks/frontierchallenge/scripts/apply_score_policy.py b/benchmarks/frontierchallenge/scripts/apply_score_policy.py index 115ac01..171be65 100644 --- a/benchmarks/frontierchallenge/scripts/apply_score_policy.py +++ b/benchmarks/frontierchallenge/scripts/apply_score_policy.py @@ -10,17 +10,17 @@ from pathlib import Path OLD_RULE = '"passed": 1.0 if passed is True else 0.0,' -FULL_SCORE_RULE = ( - '"passed": 1.0 if complete and float(score or 0.0) / 100.0 == 1.0 else 0.0,' +PASS_RULE = ( + '"passed": 1.0 if complete and 0.999 < float(score or 0.0) / 100.0 <= 1.0 else 0.0,' ) def apply_policy(task_dir: Path) -> None: adapter = task_dir / "tests" / "run_frontier_verifier.py" text = adapter.read_text(encoding="utf-8") - if text.count(OLD_RULE) != 1 or FULL_SCORE_RULE in text: + if text.count(OLD_RULE) != 1 or PASS_RULE in text: raise ValueError(f"unsupported or already modified reward adapter: {adapter}") - adapter.write_text(text.replace(OLD_RULE, FULL_SCORE_RULE), encoding="utf-8") + adapter.write_text(text.replace(OLD_RULE, PASS_RULE), encoding="utf-8") def main() -> None: diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index 62e8d88..3238ffb 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -533,7 +533,7 @@ if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "FATAL: $(basename "$task_dir") has no verifier entrypoint after unsealing." >&2 exit 1 fi - # Emit only the benchmark-wide full-score pass decision, before Harbor + # Emit only the benchmark-wide task_score > 0.999 decision, before Harbor # reads reward.json. No alternate native pass metric is retained. python3 "$ROOT/scripts/apply_score_policy.py" "$task_dir" unsealed=$((unsealed + 1)) diff --git a/benchmarks/frontierchallenge/scripts/summarize_results.py b/benchmarks/frontierchallenge/scripts/summarize_results.py index c1cf1a4..e1221a2 100755 --- a/benchmarks/frontierchallenge/scripts/summarize_results.py +++ b/benchmarks/frontierchallenge/scripts/summarize_results.py @@ -21,7 +21,7 @@ #: 100 tasks minus task_065 (needs a GPU) and task_047 / task_049 (their #: deterministic grader only runs in a nested container, unavailable here). EXPECTED_TOTAL_TASKS = 97 -METRIC_DEFINITION = "exact-full-score" +METRIC_DEFINITION = "score-gt-0.999" def valid_score(score: Any) -> bool: @@ -34,11 +34,11 @@ def valid_score(score: Any) -> bool: def official_pass(row: dict[str, Any]) -> bool: - """The sole pass decision: exact full credit on a completed evaluation.""" + """The sole pass decision: a completed valid score strictly above 0.999.""" return ( row.get("evaluation_complete") == 1 and valid_score(row.get("task_score")) - and row["task_score"] == 1.0 + and row["task_score"] > 0.999 ) #: Judge stderr wording (both casings occur across the frozen graders) for the @@ -161,7 +161,7 @@ def summarize(rows: list[dict[str, Any]], expected_total: int) -> dict[str, Any] pass_rate = n_passed / expected_total return { "metric_definition": METRIC_DEFINITION, - "pass_rule": "evaluation_complete == 1 and task_score == 1.0", + "pass_rule": "evaluation_complete == 1 and 0.999 < task_score <= 1.0", "n_tasks_expected": expected_total, "n_trials_found": n, "n_graded": len(graded), @@ -222,7 +222,7 @@ def print_report(job_dir: Path, summary: dict[str, Any]) -> None: ) if summary["pass_rate"] is not None: print( - f"Pass Rate (task_score == 1): {summary['n_passed']}/{summary['n_tasks_expected']} " + f"Pass Rate (task_score > 0.999): {summary['n_passed']}/{summary['n_tasks_expected']} " f"= {summary['pass_rate'] * 100:.1f}%" ) else: diff --git a/benchmarks/frontierchallenge/site/index.html b/benchmarks/frontierchallenge/site/index.html index 277793a..10a1284 100644 --- a/benchmarks/frontierchallenge/site/index.html +++ b/benchmarks/frontierchallenge/site/index.html @@ -60,7 +60,7 @@

FrontierChallenge

-

13 model–scaffold configurations · 97 tasks · Historical table: Pass Rate used native Score ≥ 99.9. The current evaluation policy requires completed task_score == 1.0 (no tolerance); these historical numbers have not been recomputed.

+

13 model–scaffold configurations · 97 tasks · Historical table: Pass Rate used native Score ≥ 99.9. The current evaluation policy requires completed task_score > 0.999 (strictly greater, without rounding); these historical numbers have not been recomputed.

diff --git a/benchmarks/frontierchallenge/tests/test_apply_score_policy.py b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py index b8ae249..20dfd2a 100644 --- a/benchmarks/frontierchallenge/tests/test_apply_score_policy.py +++ b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py @@ -5,10 +5,14 @@ @pytest.mark.parametrize("score,native,complete,expected", [ - (100, False, True, 1), (80, True, True, 0), (99.999, True, True, 0), + (100, False, True, 1), (80, True, True, 0), (99.999, True, True, 1), + # The frozen normalization emits 99.9 / 100 as 0.9990000000000001. + # Compare that actual task_score, not a rounded display of 0.999. + (99.9, True, True, 1), (99.91, False, True, 1), (99.899, True, True, 0), + (99.91, True, False, 0), (100.1, True, True, 0), (float("inf"), True, True, 0), (100, True, False, 0), (0, True, True, 0), (100, None, True, 1), ]) -def test_staged_reward_uses_only_full_score(tmp_path, score, native, complete, expected): +def test_staged_reward_uses_strict_score_threshold(tmp_path, score, native, complete, expected): (tmp_path / "tests").mkdir() adapter = tmp_path / "tests/run_frontier_verifier.py" adapter.write_text('reward = {' + OLD_RULE + '}\n') diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py index c205a3b..a132eef 100644 --- a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py +++ b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py @@ -90,7 +90,7 @@ def test_open_selection_migrates_legacy_cache_and_ignores_stale_orca(runtime): assert not (stale / "environment/data/output.txt").is_symlink() assert (stale / "tests/test.sh").is_file() adapter = (stale / "tests/run_frontier_verifier.py").read_text() - assert '"passed": 1.0 if complete and float(score or 0.0) / 100.0 == 1.0 else 0.0' in adapter + assert '"passed": 1.0 if complete and 0.999 < float(score or 0.0) / 100.0 <= 1.0 else 0.0' in adapter assert "orca-user-local" not in (root / "docker.log").read_text() args = (root / "harbor.log").read_text().splitlines() assert args[args.index("--include-task-name") + 1] == OPEN diff --git a/benchmarks/frontierchallenge/tests/test_summarize_results.py b/benchmarks/frontierchallenge/tests/test_summarize_results.py index 42ce6bc..e29d91e 100644 --- a/benchmarks/frontierchallenge/tests/test_summarize_results.py +++ b/benchmarks/frontierchallenge/tests/test_summarize_results.py @@ -21,7 +21,8 @@ def trial(root, name, score, native_passed=1, complete=1): @pytest.mark.parametrize("score,native,complete,passed", [ - (1.0, 0, 1, 1), (1, 1, 1, 1), (0.999999999, 1, 1, 0), + (1.0, 0, 1, 1), (1, 1, 1, 1), (0.999999999, 1, 1, 1), + (0.9991, 0, 1, 1), (0.998999999, 1, 1, 0), (0.9991, 1, 0, 0), (0.999, 1, 1, 0), (0.8, 1, 1, 0), (0.0, 1, 1, 0), (1.0, 1, 0, 0), (1.0, 1, None, 0), ]) @@ -56,7 +57,8 @@ def test_fixed_denominator_and_partial_credit(tmp_path): assert summary["mean_task_score_100"] == pytest.approx(170 / 97) assert summary["n_missing_or_errored"] == 94 assert summary["complete"] is False - assert summary["metric_definition"] == "exact-full-score" + assert summary["metric_definition"] == "score-gt-0.999" + assert summary["pass_rule"] == "evaluation_complete == 1 and 0.999 < task_score <= 1.0" def test_empty_run_is_zero_and_incomplete(): From f63aa1f94f8f8801ad4dfed564d95e430e176a6d Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 15:09:48 +0800 Subject: [PATCH 05/10] fix(frontierchallenge): sanitize verifier logs and guard legacy job resumes --- benchmarks/frontierchallenge/docs/running.md | 11 ++- benchmarks/frontierchallenge/docs/scoring.md | 10 +++ .../frontierchallenge/docs/troubleshooting.md | 7 ++ .../frontierchallenge/release/datasets.json | 4 +- .../scripts/apply_score_policy.py | 31 +++++++- .../frontierchallenge/scripts/job_policy.py | 71 +++++++++++++++++++ .../frontierchallenge/scripts/run_eval.sh | 37 +++------- .../scripts/verifier_log_policy.py | 53 ++++++++++++++ .../tests/test_apply_score_policy.py | 45 +++++++++++- .../tests/test_job_policy.py | 45 ++++++++++++ .../tests/test_run_eval_integration.py | 32 ++++++++- .../tests/test_verifier_log_policy.py | 30 ++++++++ 12 files changed, 341 insertions(+), 35 deletions(-) create mode 100644 benchmarks/frontierchallenge/scripts/job_policy.py create mode 100644 benchmarks/frontierchallenge/scripts/verifier_log_policy.py create mode 100644 benchmarks/frontierchallenge/tests/test_job_policy.py create mode 100644 benchmarks/frontierchallenge/tests/test_verifier_log_policy.py diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index e52f1fb..8c3524a 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -75,12 +75,21 @@ for ORCA, unsealed, and passed to Harbor; leftover stage directories are ignored ## Resume and results -Reusing a job name resumes completed work when the requested task set matches: +Reusing a job name resumes completed work only when the task selection and +scoring/log policy match the recorded job: ```bash ./scripts/run_eval.sh --agent claude-code --model --job-name ``` +The runner records its policy in +`/.frontierchallenge-policies/.json`. Keep that sidecar +alongside jobs when moving them. A missing/old policy marker, changed task +selection, or invalid job metadata is refused before staging. Use a fresh +`--job-name` or `--jobs-dir`; old rewards and logs are not silently rewritten +or mixed into a new-policy job. Existing results can still be summarized +separately from their scores, but their original rewards remain historical. + Results are written under `results/harbor//`. Read the aggregate with: ```bash diff --git a/benchmarks/frontierchallenge/docs/scoring.md b/benchmarks/frontierchallenge/docs/scoring.md index 530a6f0..bfceb96 100644 --- a/benchmarks/frontierchallenge/docs/scoring.md +++ b/benchmarks/frontierchallenge/docs/scoring.md @@ -37,6 +37,16 @@ encrypted reference archives and partial-credit rubrics remain unchanged. The summarizer also derives `passed` from score and completion when processing older results, discarding their old pass decision rather than copying it. +Native grader diagnostics are generated in a temporary verifier directory. +Before publication, pass decisions are removed from diagnostic JSON and text; +only `reward.json` retains the benchmark's pass field. Partial scores and +ordinary error diagnostics are preserved. Raw grader logs are not published. + +The single-field guarantee applies to new-policy jobs. Legacy jobs are refused +on resume, rather than mixing their old rewards with new results. Summarizing +an old job does not rewrite its original rewards or logs; see +[resume and upgrade rules](running.md#resume-and-results). + This policy is identified by `metric_definition: score-gt-0.999` in the summary. Recompute historical results from raw rewards before comparing them; results computed with per-task thresholds, `== 1.0`, or `>= 0.999` are not interchangeable. diff --git a/benchmarks/frontierchallenge/docs/troubleshooting.md b/benchmarks/frontierchallenge/docs/troubleshooting.md index 1782740..9b1dadf 100644 --- a/benchmarks/frontierchallenge/docs/troubleshooting.md +++ b/benchmarks/frontierchallenge/docs/troubleshooting.md @@ -73,6 +73,13 @@ Inspect `verifier/reward.json`: The aggregate summarizer reports incomplete and missing-artifact counts instead of silently treating every zero as the same failure mode. +## An old job is refused after upgrading + +The runner refuses populated jobs without a matching scoring/log policy marker +or with a changed task selection. Start a fresh `--job-name` (or `--jobs-dir`). +Do not manufacture a marker for old results: their raw rewards may use the old +per-task pass rule. Old results remain untouched and may be summarized separately. + ## Inspect a trial ```bash diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json index 2452b23..2ffa9d0 100644 --- a/benchmarks/frontierchallenge/release/datasets.json +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -2,10 +2,10 @@ "schema_version": 1, "solve": { "repo": "apodex/FrontierChallenge", - "revision": "9ab8b7f6a19b2995ad43ad944af94b6d1e70b2b9" + "revision": "7fbd391bf3e69d7e3c7de1bcb145619dcc5f2f7c" }, "reference": { "repo": "apodex/FrontierChallenge-reference", - "revision": "aaeb6d84d26f8c4174bf88ef4fc74e03013021f8" + "revision": "43367f32c7785a37f0c84ee64426a6f4854b1348" } } diff --git a/benchmarks/frontierchallenge/scripts/apply_score_policy.py b/benchmarks/frontierchallenge/scripts/apply_score_policy.py index 171be65..442d646 100644 --- a/benchmarks/frontierchallenge/scripts/apply_score_policy.py +++ b/benchmarks/frontierchallenge/scripts/apply_score_policy.py @@ -7,20 +7,47 @@ from __future__ import annotations import argparse +import shutil from pathlib import Path OLD_RULE = '"passed": 1.0 if passed is True else 0.0,' PASS_RULE = ( '"passed": 1.0 if complete and 0.999 < float(score or 0.0) / 100.0 <= 1.0 else 0.0,' ) +ENTRYPOINT = '\nif __name__ == "__main__":' +LOG_WRAPPER = ''' +def main(): + # Native decisions exist only in a private temporary directory. Publish + # sanitized diagnostics on success and failure, never a second pass metric. + import tempfile + from verifier_log_policy import publish_logs + global LOGS + public_logs = LOGS + with tempfile.TemporaryDirectory(prefix="frontier-native-verifier-") as scratch: + LOGS = Path(scratch) + try: + _run_native_verifier() + finally: + try: + publish_logs(LOGS, public_logs) + finally: + LOGS = public_logs + +''' def apply_policy(task_dir: Path) -> None: adapter = task_dir / "tests" / "run_frontier_verifier.py" text = adapter.read_text(encoding="utf-8") - if text.count(OLD_RULE) != 1 or PASS_RULE in text: + if (text.count(OLD_RULE) != 1 or PASS_RULE in text + or text.count("def main():") != 1 or text.count(ENTRYPOINT) != 1): raise ValueError(f"unsupported or already modified reward adapter: {adapter}") - adapter.write_text(text.replace(OLD_RULE, PASS_RULE), encoding="utf-8") + updated = text.replace(OLD_RULE, PASS_RULE).replace("def main():", "def _run_native_verifier():") + updated = updated.replace(ENTRYPOINT, LOG_WRAPPER + ENTRYPOINT) + compile(updated, str(adapter), "exec") + shutil.copyfile(Path(__file__).with_name("verifier_log_policy.py"), + adapter.with_name("verifier_log_policy.py")) + adapter.write_text(updated, encoding="utf-8") def main() -> None: diff --git a/benchmarks/frontierchallenge/scripts/job_policy.py b/benchmarks/frontierchallenge/scripts/job_policy.py new file mode 100644 index 0000000..9a1d69b --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/job_policy.py @@ -0,0 +1,71 @@ +"""Prevent resuming or mixing jobs produced by incompatible scoring policies.""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +POLICY = {"schema": 1, "metric_definition": "score-gt-0.999", "diagnostics": "no-native-pass"} + + +def marker_path(job: Path) -> Path: + if not job.name: + raise ValueError("job must be a named directory") + return job.parent / ".frontierchallenge-policies" / f"{job.name}.json" + + +def expected_policy(task_ids: list[str]) -> dict: + return {**POLICY, "task_ids": sorted(set(task_ids))} + + +def check_job(job: Path, task_ids: list[str]) -> str: + if not job.exists() or (job.is_dir() and not any(job.iterdir())): + return "new" + try: + stored = json.loads(marker_path(job).read_text()) + if stored != expected_policy(task_ids): + raise ValueError("scoring/log policy or task selection changed") + if not (job / "config.json").is_file(): + raise ValueError("job config is missing") + lock = json.loads((job / "lock.json").read_text()) + recorded = {trial["task"]["name"] for trial in lock["trials"]} + if recorded != set(task_ids): + raise ValueError("recorded task selection differs") + except (OSError, ValueError, KeyError, TypeError) as exc: + raise ValueError( + f"cannot resume {job}: missing/incompatible FrontierChallenge policy or job metadata. " + "Use a fresh --job-name (or --jobs-dir); existing results were not modified. " + "Old results may be summarized separately, but must not be mixed into a new-policy job." + ) from exc + return "resume" + + +def record_policy(job: Path, task_ids: list[str]) -> None: + check_job(job, task_ids) + target = marker_path(job) + target.parent.mkdir(parents=True, exist_ok=True) + document = expected_policy(task_ids) + if target.exists() and json.loads(target.read_text()) == document: + return + # Only absent/empty jobs can acquire a new marker; populated old jobs fail + # the check above. No old result or old log is rewritten during an upgrade. + target.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=("check", "record")) + parser.add_argument("job", type=Path) + parser.add_argument("--task-id", action="append", required=True) + args = parser.parse_args() + try: + if args.action == "check": + print(check_job(args.job.absolute(), args.task_id)) + else: + record_policy(args.job.absolute(), args.task_id) + except (OSError, ValueError) as exc: + parser.error(str(exc)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index 3238ffb..c6e1985 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -278,6 +278,14 @@ if [[ ${#EFFECTIVE_TASK_IDS[@]} -eq 0 ]]; then exit 1 fi +POLICY_TASK_ARGS=() +for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + POLICY_TASK_ARGS+=(--task-id "$task_id") +done +# Fail before modifying staging when an old job would mix scoring/log policies. +JOB_ACTION="$(python3 "$ROOT/scripts/job_policy.py" check \ + "$JOBS_DIR/$JOB_NAME" "${POLICY_TASK_ARGS[@]}")" + echo "== Staging $TRACK-track tasks from $SOLVE_TASKS into $STAGE_DIR ==" staged=0 skipped=0 @@ -454,31 +462,6 @@ if [[ -n "$N_CONCURRENT_AGENTS" ]]; then CONCURRENT_AGENTS_ARGS+=(--n-concurrent-agents "$N_CONCURRENT_AGENTS") fi -# Resume only when the recorded and requested task sets match. Otherwise keep -# completed trial directories but archive stale job-level metadata. -RESUME_JOB=0 -if [[ -f "$JOBS_DIR/$JOB_NAME/config.json" ]]; then - if REQUESTED="${EFFECTIVE_TASK_IDS[*]}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY' -import json, os, sys -requested = set(os.environ.get("REQUESTED", "").split()) -try: - recorded = {t["task"]["name"] for t in json.load(open(sys.argv[1]))["trials"]} -except Exception: - sys.exit(1) # unreadable lock -> treat as new work -sys.exit(0 if requested == recorded else 1) -PY - then - RESUME_JOB=1 - else - echo "== Task set changed - archiving stale job files in $JOBS_DIR/$JOB_NAME ==" - stamp=$(date +%Y%m%d-%H%M%S) - for f in config.json lock.json result.json job.log; do - [[ -e "$JOBS_DIR/$JOB_NAME/$f" ]] && \ - mv "$JOBS_DIR/$JOB_NAME/$f" "$JOBS_DIR/$JOB_NAME/.prev-$stamp-$f" - done - fi -fi - # The real verifier is distributed only through the separate encrypted # reference dataset. Download and verify that package first, then point this # runner at the resulting directory. The archive password is intentionally @@ -541,7 +524,9 @@ if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "Unsealed $unsealed task(s)." fi -if [[ "$RESUME_JOB" -eq 1 ]]; then +# Record outside Harbor's job directory so a fresh job remains fresh to Harbor. +python3 "$ROOT/scripts/job_policy.py" record "$JOBS_DIR/$JOB_NAME" "${POLICY_TASK_ARGS[@]}" +if [[ "$JOB_ACTION" == "resume" ]]; then echo "== Resuming existing job dir: $JOBS_DIR/$JOB_NAME ==" PYTHONPATH="$ROOT" HARBOR_TELEMETRY=off harbor job resume \ --job-path "$JOBS_DIR/$JOB_NAME" diff --git a/benchmarks/frontierchallenge/scripts/verifier_log_policy.py b/benchmarks/frontierchallenge/scripts/verifier_log_policy.py new file mode 100644 index 0000000..7fb0e94 --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/verifier_log_policy.py @@ -0,0 +1,53 @@ +"""Publish grader diagnostics without retaining alternate pass decisions.""" +from __future__ import annotations + +import json +import re +from pathlib import Path + +PASS_KEYS = {"pass", "passed", "native_passed", "pass_threshold"} +DECISION_WORD = re.compile(r"\b(?:pass|passed|fail|native_passed|pass_threshold)\b", re.IGNORECASE) + + +def clean_json(value): + if isinstance(value, dict): + return {key: clean_json(item) for key, item in value.items() + if key.casefold() not in PASS_KEYS} + if isinstance(value, list): + return [clean_json(item) for item in value] + if isinstance(value, str) and value.strip().casefold() in {"pass", "passed", "fail"}: + return "[decision omitted]" + return value + + +def clean_text(text: str) -> str: + # Structured grader stdout retains scores and diagnostics, minus decisions. + try: + return json.dumps(clean_json(json.loads(text)), indent=2) + "\n" + except json.JSONDecodeError: + return "".join( + "[decision omitted]\n" if DECISION_WORD.search(line) else line + for line in text.splitlines(keepends=True) + ) + + +def publish_logs(private: Path, public: Path) -> None: + """Raw grader files never enter the Harbor log directory. + + reward.json already uses the sole benchmark pass rule and is preserved. + Other JSON/text files are diagnostic, not an additional scoring interface. + """ + public.mkdir(parents=True, exist_ok=True) + for source in sorted(private.iterdir()): + if not source.is_file() or source.is_symlink(): + raise ValueError(f"unexpected verifier log entry: {source.name}") + text = source.read_text(encoding="utf-8") + if source.name == "reward.json": + cleaned = text + elif source.suffix == ".json": + cleaned = json.dumps(clean_json(json.loads(text)), indent=2) + "\n" + elif source.suffix == ".txt": + cleaned = clean_text(text) + else: + raise ValueError(f"unsupported verifier log type: {source.name}") + (public / source.name).write_text(cleaned, encoding="utf-8") diff --git a/benchmarks/frontierchallenge/tests/test_apply_score_policy.py b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py index 20dfd2a..49ed79e 100644 --- a/benchmarks/frontierchallenge/tests/test_apply_score_policy.py +++ b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py @@ -15,10 +15,12 @@ def test_staged_reward_uses_strict_score_threshold(tmp_path, score, native, complete, expected): (tmp_path / "tests").mkdir() adapter = tmp_path / "tests/run_frontier_verifier.py" - adapter.write_text('reward = {' + OLD_RULE + '}\n') + adapter.write_text('def main():\n global reward\n reward = {' + OLD_RULE + + '}\n\nif __name__ == "__main__":\n main()\n') apply_policy(tmp_path) - namespace = {"score": score, "passed": native, "complete": complete} + namespace = {"score": score, "passed": native, "complete": complete, "__name__": "test_adapter"} exec(compile(adapter.read_text(), str(adapter), "exec"), namespace) + namespace["_run_native_verifier"]() assert namespace["reward"] == {"passed": float(expected)} @@ -29,3 +31,42 @@ def test_unknown_adapter_fails_closed(tmp_path): with pytest.raises(ValueError, match="unsupported"): apply_policy(tmp_path) assert adapter.read_text() == "reward = {}\n" + + +@pytest.mark.parametrize("failure", [False, True]) +def test_adapter_publishes_clean_logs_on_success_and_failure(tmp_path, failure): + import json + adapter_dir = tmp_path / "task/tests" + adapter_dir.mkdir(parents=True) + adapter = adapter_dir / "run_frontier_verifier.py" + adapter.write_text('''from pathlib import Path +import json +def main(): + global private_path + private_path = LOGS + LOGS.mkdir(parents=True, exist_ok=True) + (LOGS / "native_grader_result.json").write_text(json.dumps({"score": 80, "passed": True})) + (LOGS / "native_grader.stdout.txt").write_text("PASSED=true\\nscore:80\\n") + if failure: + raise RuntimeError("fixture failure") + reward = {"task_score": 0.8, ''' + OLD_RULE + ''' "evaluation_complete": 1} + (LOGS / "reward.json").write_text(json.dumps(reward)) + +if __name__ == "__main__": + main() +''') + apply_policy(tmp_path / "task") + public = tmp_path / "public" + ns = {"__name__": "fixture", "LOGS": public, "failure": failure, + "score": 80, "passed": True, "complete": True} + exec(compile(adapter.read_text(), str(adapter), "exec"), ns) + if failure: + with pytest.raises(RuntimeError, match="fixture failure"): + ns["main"]() + else: + ns["main"]() + assert json.loads((public / "reward.json").read_text())["passed"] == 0 + assert json.loads((public / "native_grader_result.json").read_text()) == {"score": 80} + assert "PASSED" not in (public / "native_grader.stdout.txt").read_text() + assert ns["LOGS"] == public + assert not ns["private_path"].exists() diff --git a/benchmarks/frontierchallenge/tests/test_job_policy.py b/benchmarks/frontierchallenge/tests/test_job_policy.py new file mode 100644 index 0000000..16cbf57 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_job_policy.py @@ -0,0 +1,45 @@ +import json + +import pytest +from job_policy import check_job, marker_path, record_policy + + +def existing(job): + job.mkdir(parents=True) + (job / "config.json").write_text("{}") + (job / "lock.json").write_text(json.dumps({"trials": [{"task": {"name": "task_one"}}]})) + + +def test_new_job_marker_does_not_create_harbor_job(tmp_path): + job = tmp_path / "new" + assert check_job(job, ["task_one"]) == "new" + record_policy(job, ["task_one"]) + assert not job.exists() + assert marker_path(job).is_file() + existing(job) + assert check_job(job, ["task_one"]) == "resume" + + +@pytest.mark.parametrize("case", ["missing", "old-metric", "old-logs", "different-tasks", "invalid-lock"]) +def test_unsafe_resume_does_not_mutate_results(tmp_path, case): + job = tmp_path / "old" + record_policy(job, ["task_one"]) + existing(job) + marker = marker_path(job) + doc = json.loads(marker.read_text()) + if case == "missing": + marker.unlink() + elif case == "old-metric": + doc["metric_definition"] = "exact-full-score" + marker.write_text(json.dumps(doc)) + elif case == "old-logs": + doc.pop("diagnostics") + marker.write_text(json.dumps(doc)) + elif case == "invalid-lock": + (job / "lock.json").write_text("broken") + tasks = ["other"] if case == "different-tasks" else ["task_one"] + before = {p.name: p.read_bytes() for p in job.iterdir()} + for operation in (check_job, record_policy): + with pytest.raises(ValueError, match="fresh --job-name"): + operation(job, tasks) + assert before == {p.name: p.read_bytes() for p in job.iterdir()} diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py index a132eef..4755c97 100644 --- a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py +++ b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py @@ -41,7 +41,8 @@ def runtime(tmp_path): (verifier / "tests").mkdir(parents=True) (verifier / "tests/test.sh").write_text("#!/bin/sh\nexit 0\n") (verifier / "tests/run_frontier_verifier.py").write_text( - 'reward = {"passed": 1.0 if passed is True else 0.0,}\n' + 'def main():\n reward = {"passed": 1.0 if passed is True else 0.0,}\n' + '\nif __name__ == "__main__":\n main()\n' ) reference_archive.pack(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"], "frontier-challenge-reference", force=True) @@ -69,7 +70,8 @@ def runtime(tmp_path): stage = tmp_path / "stage" cmd = ["bash", str(runtime / "scripts/run_eval.sh"), "--agent", "claude-code", "--model", "fixture", "--solve-dir", str(solve), "--reference-dir", str(reference), - "--stage-dir", str(stage), "--env-file", str(envfile), "--no-judge-override", "--no-summary"] + "--stage-dir", str(stage), "--env-file", str(envfile), "--no-judge-override", "--no-summary", + "--jobs-dir", str(tmp_path / "jobs"), "--job-name", "fixture"] return tmp_path, solve, stage, cmd, env @@ -115,3 +117,29 @@ def test_selected_licensed_task_still_requires_orca(runtime): assert result.returncode != 0 assert "1 task(s) require ORCA" in result.stderr assert not (root / "harbor.log").exists() + + +def test_legacy_job_is_refused_before_staging(runtime): + root, _, stage, cmd, env = runtime + job = root / "jobs/fixture" + job.mkdir(parents=True) + (job / "config.json").write_text("{}") + result = subprocess.run(cmd, env=env, capture_output=True, text=True, check=False) + assert result.returncode != 0 + assert "Use a fresh --job-name" in result.stderr + assert not (root / "harbor.log").exists() + assert not (stage / OPEN).exists() + assert (job / "config.json").read_text() == "{}" + + +def test_current_policy_job_resumes(runtime): + import job_policy + root, _, _, cmd, env = runtime + job = root / "jobs/fixture" + job_policy.record_policy(job, [OPEN]) + job.mkdir(parents=True) + (job / "config.json").write_text("{}") + (job / "lock.json").write_text(json.dumps({"trials": [{"task": {"name": OPEN}}]})) + result = subprocess.run(cmd, env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert (root / "harbor.log").read_text().splitlines()[:2] == ["job", "resume"] diff --git a/benchmarks/frontierchallenge/tests/test_verifier_log_policy.py b/benchmarks/frontierchallenge/tests/test_verifier_log_policy.py new file mode 100644 index 0000000..b8ded81 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_verifier_log_policy.py @@ -0,0 +1,30 @@ +import json + +from verifier_log_policy import clean_json, clean_text, publish_logs + + +def test_nested_decisions_are_removed_but_scores_remain(): + source = {"score": 80, "passed": True, "detail": [{"PASSED": True, "score": 5}], + "evaluation_complete": True, "pass_threshold": 70, "verdict": "PASS"} + assert clean_json(source) == {"score": 80, "detail": [{"score": 5}], + "evaluation_complete": True, "verdict": "[decision omitted]"} + assert source["passed"] is True # The adapter can still parse its in-memory payload. + + +def test_text_decisions_are_removed_without_losing_ordinary_errors(): + text = 'score: 80\nPASSED = true\nscore 80 -> PASS\nfailed to open input.csv\n' + assert clean_text(text) == 'score: 80\n[decision omitted]\n[decision omitted]\nfailed to open input.csv\n' + assert json.loads(clean_text('{"score": 80, "passed": true}')) == {"score": 80} + + +def test_only_reward_keeps_canonical_pass(tmp_path): + private, public = tmp_path / "private", tmp_path / "public" + private.mkdir() + (private / "native_grader_result.json").write_text('{"score":80,"passed":true}') + (private / "native_grader.stdout.txt").write_text('PASSED=true\nscore: 80\n') + reward = '{"task_score":0.8,"passed":0,"evaluation_complete":1}' + (private / "reward.json").write_text(reward) + publish_logs(private, public) + assert (public / "reward.json").read_text() == reward + assert json.loads((public / "native_grader_result.json").read_text()) == {"score": 80} + assert "PASSED" not in (public / "native_grader.stdout.txt").read_text() From 19159d1ef6bbc28f4acb05fc6eb9fe5fd1d44820 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 15:48:05 +0800 Subject: [PATCH 06/10] fix(frontierchallenge): verify OCI image identities on containerd --- .../frontierchallenge/docs/quickstart.md | 6 ++ .../frontierchallenge/docs/troubleshooting.md | 9 +++ benchmarks/frontierchallenge/pyproject.toml | 1 + .../scripts/setup_release.py | 57 ++++++++++++++++- .../tests/test_image_identity.py | 63 +++++++++++++++++++ 5 files changed, 134 insertions(+), 2 deletions(-) create mode 100644 benchmarks/frontierchallenge/tests/test_image_identity.py diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 3584fc9..0e20a68 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -69,6 +69,12 @@ solve dataset. It checks the declared size, SHA-256 and image ID before loading the `linux/amd64` image into Docker. No container registry is used. Evaluator- local paths are written to `.frontierchallenge/config.env`. +Docker's classic and containerd image stores expose different image IDs. Setup +accepts the published config digest directly, or verifies that the loaded OCI +manifest digest links to that exact config inside the SHA-256-verified archive. +Keep runtime dependencies current with `python -m pip install -e .`; do not +disable identity checks or change Docker's storage backend to work around this. + For release development only, `--revision main` overrides both pins; `--reference-revision` can override the reference revision independently. Normal evaluation should keep the checkout's pins so later dataset changes cannot alter diff --git a/benchmarks/frontierchallenge/docs/troubleshooting.md b/benchmarks/frontierchallenge/docs/troubleshooting.md index 9b1dadf..668f0ec 100644 --- a/benchmarks/frontierchallenge/docs/troubleshooting.md +++ b/benchmarks/frontierchallenge/docs/troubleshooting.md @@ -80,6 +80,15 @@ or with a changed task selection. Start a fresh `--job-name` (or `--jobs-dir`). Do not manufacture a marker for old results: their raw rewards may use the old per-task pass rule. Old results remain untouched and may be summarized separately. +## Loaded image identity differs on Docker's containerd store + +Update the runtime and reinstall its dependencies (`python -m pip install -e .`). +Older setup code compares Docker's OCI manifest ID directly with the release's +config digest, which can reject a valid image on Docker 29/containerd. Current +setup verifies the manifest-to-config digest link inside the verified archive. +Archive size/SHA-256 and platform checks remain mandatory; no daemon restart, +storage-driver change, registry pull, or HF payload update is needed. + ## Inspect a trial ```bash diff --git a/benchmarks/frontierchallenge/pyproject.toml b/benchmarks/frontierchallenge/pyproject.toml index eaa966e..84376b4 100644 --- a/benchmarks/frontierchallenge/pyproject.toml +++ b/benchmarks/frontierchallenge/pyproject.toml @@ -23,6 +23,7 @@ license-files = ["LICENSE"] dependencies = [ "harbor==0.20.0", "huggingface_hub>=0.33,<2", + "zstandard>=0.23,<1", ] [project.optional-dependencies] diff --git a/benchmarks/frontierchallenge/scripts/setup_release.py b/benchmarks/frontierchallenge/scripts/setup_release.py index 48ea539..0f93031 100755 --- a/benchmarks/frontierchallenge/scripts/setup_release.py +++ b/benchmarks/frontierchallenge/scripts/setup_release.py @@ -7,10 +7,12 @@ import hashlib import json import os +import re import shlex import shutil import subprocess import sys +import tarfile from pathlib import Path ROOT = Path(__file__).resolve().parents[1] @@ -240,6 +242,58 @@ def validate_orca_runtime(image: str) -> None: ) +def verify_oci_image_identity(stream, image_id: str, config_id: str) -> None: + """Bind a containerd manifest ID to the release's frozen config digest. + + The caller has already verified the entire archive's SHA-256. Read only + metadata, never extract files or change Docker's storage configuration. + """ + for value in (image_id, config_id): + if not isinstance(value, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", value): + raise SystemExit("invalid image identity digest") + member_name = "blobs/sha256/" + image_id.removeprefix("sha256:") + with tarfile.open(fileobj=stream, mode="r|") as archive: + for member in archive: + if member.name != member_name: + continue + if not member.isfile() or member.size > 1024 * 1024: + raise SystemExit("invalid OCI image manifest entry") + raw = archive.extractfile(member).read() + if "sha256:" + hashlib.sha256(raw).hexdigest() != image_id: + raise SystemExit("OCI image manifest digest mismatch") + manifest = json.loads(raw) + if not isinstance(manifest, dict): + raise SystemExit("invalid OCI image manifest") + config = manifest.get("config") + if ( + manifest.get("schemaVersion") != 2 + or manifest.get("mediaType") not in { + "application/vnd.oci.image.manifest.v1+json", + "application/vnd.docker.distribution.manifest.v2+json", + } + or not isinstance(config, dict) + or config.get("digest") != config_id + ): + raise SystemExit("loaded HF image configuration does not match its manifest") + return + raise SystemExit("loaded HF image identity has no matching manifest in the verified archive") + + +def verify_loaded_image_identity(archive: Path, image_id: str, config_id: str) -> None: + # Classic Docker exposes the config digest directly. Containerd exposes an + # OCI target digest; verify its content-addressed manifest and config link. + if image_id == config_id: + return + try: + import zstandard + except ImportError as exc: + raise SystemExit("install runtime dependencies with `python -m pip install -e .`") from exc + with archive.open("rb") as source: + with zstandard.ZstdDecompressor().stream_reader(source) as stream: + verify_oci_image_identity(stream, image_id, config_id) + print("verified OCI manifest → published image config digest") + + def load_hf_image_archive( *, solve: Path, @@ -283,8 +337,7 @@ def load_hf_image_archive( if loaded_ref != archive_config["loaded_ref"]: raise SystemExit("HF image loaded_ref does not match release/images.json") image_id = ensure_image(loaded_ref) - if manifest.get("image_id") != image_id: - raise SystemExit("loaded HF image identity does not match its manifest") + verify_loaded_image_identity(archive, image_id, manifest.get("image_id")) return image_id diff --git a/benchmarks/frontierchallenge/tests/test_image_identity.py b/benchmarks/frontierchallenge/tests/test_image_identity.py new file mode 100644 index 0000000..f412208 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_image_identity.py @@ -0,0 +1,63 @@ +import hashlib +import io +import json +import tarfile + +import pytest +from setup_release import verify_loaded_image_identity, verify_oci_image_identity + +CONFIG = "sha256:" + "a" * 64 + + +def fixture(*, config=CONFIG, media_type="application/vnd.oci.image.manifest.v1+json", + corrupt=False, missing=False, symlink=False, oversized=False): + raw = json.dumps({"schemaVersion": 2, "mediaType": media_type, + "config": {"digest": config}}).encode() + identity = "sha256:" + hashlib.sha256(raw).hexdigest() + if corrupt: + raw += b" " + if oversized: + raw += b" " * (1024 * 1024) + identity = "sha256:" + hashlib.sha256(raw).hexdigest() + stream = io.BytesIO() + with tarfile.open(fileobj=stream, mode="w") as archive: + member = tarfile.TarInfo("other" if missing else "blobs/sha256/" + identity[7:]) + if symlink: + member.type = tarfile.SYMTYPE + member.linkname = "/should-not-be-read" + else: + member.size = len(raw) + archive.addfile(member, None if symlink else io.BytesIO(raw)) + stream.seek(0) + return stream, identity + + +def test_classic_identity_needs_no_decompression(tmp_path): + verify_loaded_image_identity(tmp_path / "not-opened", CONFIG, CONFIG) + + +@pytest.mark.parametrize("media_type", ["application/vnd.oci.image.manifest.v1+json", + "application/vnd.docker.distribution.manifest.v2+json"]) +def test_containerd_identity_is_bound_to_published_config(media_type): + stream, identity = fixture(media_type=media_type) + assert identity != CONFIG + verify_oci_image_identity(stream, identity, CONFIG) + + +@pytest.mark.parametrize("kwargs,expected", [ + ({"config": "sha256:" + "b" * 64}, "configuration does not match"), + ({"corrupt": True}, "digest mismatch"), + ({"missing": True}, "no matching manifest"), + ({"symlink": True}, "invalid OCI image manifest entry"), + ({"oversized": True}, "invalid OCI image manifest entry"), + ({"media_type": "application/vnd.oci.image.index.v1+json"}, "configuration does not match"), +]) +def test_identity_mismatch_still_fails_closed(kwargs, expected): + stream, identity = fixture(**kwargs) + with pytest.raises(SystemExit, match=expected): + verify_oci_image_identity(stream, identity, CONFIG) + + +def test_invalid_digest_rejected_before_reading(): + with pytest.raises(SystemExit, match="invalid image identity"): + verify_oci_image_identity(io.BytesIO(), "../../wrong", CONFIG) From c3af95cd7195945d64aa83f9d74b1fe7fd21c8d6 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 16:04:59 +0800 Subject: [PATCH 07/10] fix(docker): exclude FrontierChallenge evaluator state from builds --- .dockerignore | 13 +++++++++++++ apodex/tests/test_deployment_config.py | 11 +++++++++++ docs/install/docker.md | 7 +++++++ 3 files changed, 31 insertions(+) diff --git a/.dockerignore b/.dockerignore index 5532047..94a3219 100644 --- a/.dockerignore +++ b/.dockerignore @@ -5,6 +5,19 @@ benchmarks/public/datasets benchmarks/public/results benchmarks/public/tasks-generated +# FrontierChallenge's evaluator state is never part of a runtime image. +# Root-only .env/results patterns do not cover this nested benchmark. +benchmarks/frontierchallenge/.env +benchmarks/frontierchallenge/.env.* +!benchmarks/frontierchallenge/.env.example +benchmarks/frontierchallenge/.frontierchallenge +benchmarks/frontierchallenge/tasks +benchmarks/frontierchallenge/results +benchmarks/frontierchallenge/dist +benchmarks/frontierchallenge/.venv +benchmarks/frontierchallenge/venv +benchmarks/frontierchallenge/.pytest_cache +benchmarks/frontierchallenge/*.log # Legacy pre-rename paths may still contain local corpora, generated ground # truth, and run artifacts in existing checkouts. Keep them out of remote build # contexts and images even though the current runtime no longer reads them. diff --git a/apodex/tests/test_deployment_config.py b/apodex/tests/test_deployment_config.py index f9ebf3e..9fca2c2 100644 --- a/apodex/tests/test_deployment_config.py +++ b/apodex/tests/test_deployment_config.py @@ -76,6 +76,17 @@ def test_development_compose_forces_rebuild_of_public_local_image() -> None: assert development["services"]["agent"]["pull_policy"] == "build" +def test_docker_context_excludes_frontierchallenge_evaluator_state() -> None: + patterns = (ROOT / ".dockerignore").read_text().splitlines() + prefix = "benchmarks/frontierchallenge/" + for path in (".env", ".env.*", ".frontierchallenge", "tasks", "results", + "dist", ".venv", "venv", "*.log"): + assert prefix + path in patterns + example = "!" + prefix + ".env.example" + assert example in patterns + assert patterns.index(example) > patterns.index(prefix + ".env.*") + + @pytest.mark.parametrize("arguments,service", [(["-p", "hello"], "agent"), (["eval", "--limit", "5"], "eval")]) def test_docker_helper_reuses_image_from_repository_directory(tmp_path, arguments, service): repo = tmp_path / "repo" diff --git a/docs/install/docker.md b/docs/install/docker.md index 6041a6e..ed098c5 100644 --- a/docs/install/docker.md +++ b/docs/install/docker.md @@ -44,6 +44,13 @@ After building, the convenience helper reuses that local image: ./docker/run.sh eval --limit 5 ``` +The build excludes FrontierChallenge's nested credentials, default evaluator +staging, task caches, and results. Keep any custom dataset/staging directories +outside the checkout as well: build contexts must never contain private grader +material or local secrets. `.env.example` and public runtime source remain in +the image. The FrontierChallenge task image itself is downloaded from HF via +its [separate Quickstart](../../benchmarks/frontierchallenge/docs/quickstart.md). + ## Pin a release or another image Users with access to the private GHCR package can explicitly log in and pull From 2b1bd9d2b596c2ac217c50ec21df98e81b15cf33 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 16:14:04 +0800 Subject: [PATCH 08/10] docs(frontierchallenge): clarify disk budget and image verification order --- benchmarks/frontierchallenge/docs/quickstart.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 0e20a68..ba7eeac 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -6,7 +6,8 @@ Hugging Face datasets, a real Harbor + Claude Code run, and the final score. ## Requirements - Linux x86-64 with Docker and Compose v2; -- Python 3.11+ and about 20 GB for the open image; +- Python 3.11+; allow at least 40 GB of free disk for the downloaded archive, + Docker image, and working data (more for concurrent runs and results); - a model API key and a judge API key; - `HF_TOKEN` while either dataset is private or gated; - for the full track only, an official ORCA 6.0.1 download and permission to @@ -65,8 +66,9 @@ Setup downloads the solve and reference revisions pinned by this Git checkout in `release/datasets.json`, verifies both packages, binds them to this checkout's `registry.json`, then downloads `images/frontierchallenge-cpu-open-2026.08.docker.tar.zst` from the -solve dataset. It checks the declared size, SHA-256 and image ID before loading -the `linux/amd64` image into Docker. No container registry is used. Evaluator- +solve dataset. It checks the archive's declared size and SHA-256 before loading +the `linux/amd64` image into Docker, then verifies the loaded image identity. +No container registry is used. Evaluator- local paths are written to `.frontierchallenge/config.env`. Docker's classic and containerd image stores expose different image IDs. Setup From 3fcfe87cfa07d5898fffa9b95e5ba713ab9b914d Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 16:26:17 +0800 Subject: [PATCH 09/10] fix(frontierchallenge): align evaluator Python requirement with Harbor --- benchmarks/frontierchallenge/README.md | 4 +++- benchmarks/frontierchallenge/docs/quickstart.md | 5 ++++- .../frontierchallenge/docs/troubleshooting.md | 10 +++++++++- benchmarks/frontierchallenge/pyproject.toml | 4 +++- benchmarks/frontierchallenge/release/datasets.json | 4 ++-- .../tests/test_run_eval_contract.py | 13 +++++++++++++ 6 files changed, 34 insertions(+), 6 deletions(-) diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index 009e388..8ff8966 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -38,13 +38,15 @@ simulation, electrochemistry, quantitative imaging, and molecular biology. ## End-to-end workflow -Requirements: Linux x86-64, Python 3.11+, Docker with Compose, model +Requirements: Linux x86-64, Python 3.12+ (Harbor 0.20.0), Docker with Compose, model and judge credentials, and a Hugging Face token while either dataset is private or gated. ```bash git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent/benchmarks/frontierchallenge +python3.12 -m venv .venv +source .venv/bin/activate python -m pip install -e . cp .env.example .env ``` diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index ba7eeac..f918596 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -6,7 +6,8 @@ Hugging Face datasets, a real Harbor + Claude Code run, and the final score. ## Requirements - Linux x86-64 with Docker and Compose v2; -- Python 3.11+; allow at least 40 GB of free disk for the downloaded archive, +- Python 3.12+ on the evaluator host (required by Harbor 0.20.0); + allow at least 40 GB of free disk for the downloaded archive, Docker image, and working data (more for concurrent runs and results); - a model API key and a judge API key; - `HF_TOKEN` while either dataset is private or gated; @@ -25,6 +26,8 @@ docker compose version ```bash git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent/benchmarks/frontierchallenge +python3.12 -m venv .venv +source .venv/bin/activate python -m pip install -e . cp .env.example .env ``` diff --git a/benchmarks/frontierchallenge/docs/troubleshooting.md b/benchmarks/frontierchallenge/docs/troubleshooting.md index 668f0ec..71f1a48 100644 --- a/benchmarks/frontierchallenge/docs/troubleshooting.md +++ b/benchmarks/frontierchallenge/docs/troubleshooting.md @@ -1,5 +1,13 @@ # Troubleshooting +## Editable install cannot resolve Harbor + +The pinned Harbor 0.20.0 requires Python 3.12 or newer on the evaluator host. +Use a Python 3.12+ virtual environment as shown in [Quickstart](quickstart.md), +then rerun `python -m pip install -e .`. Do not downgrade Harbor to work around +an older system Python. The Python versions inside the frozen scientific task +images are separate and do not need to change. + ## Setup cannot find the datasets While either HF repository is private or gated, pass an authorized token only @@ -11,7 +19,7 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open Do not place the HF token in `.env`, which is used to configure model and judge credentials. For offline use, pass local solve and reference directories as -shown in [Quickstart](quickstart.md). +shown in [Hugging Face layout](huggingface-release.md#download-and-verify). ## Docker is installed but runs do not start diff --git a/benchmarks/frontierchallenge/pyproject.toml b/benchmarks/frontierchallenge/pyproject.toml index 84376b4..0f1bb24 100644 --- a/benchmarks/frontierchallenge/pyproject.toml +++ b/benchmarks/frontierchallenge/pyproject.toml @@ -12,7 +12,9 @@ name = "frontier-challenge" version = "0" description = "FrontierChallenge: evaluating AI systems on real scientific workflows" readme = "README.md" -requires-python = ">=3.11" +# Harbor 0.20.0 requires Python 3.12 on the evaluator host. Scientific task +# containers have their own, separately frozen interpreter versions. +requires-python = ">=3.12" authors = [{ name = "Apodex AI" }, { name = "GADE Union" }] keywords = ["benchmark", "agents", "evaluation", "scientific-computing"] license = "CC-BY-4.0" diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json index 2ffa9d0..ed1717e 100644 --- a/benchmarks/frontierchallenge/release/datasets.json +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -2,10 +2,10 @@ "schema_version": 1, "solve": { "repo": "apodex/FrontierChallenge", - "revision": "7fbd391bf3e69d7e3c7de1bcb145619dcc5f2f7c" + "revision": "6306f5d07fcd23c911242a36c6c923e9b2ad19bf" }, "reference": { "repo": "apodex/FrontierChallenge-reference", - "revision": "43367f32c7785a37f0c84ee64426a6f4854b1348" + "revision": "e3e6719c43de28cc0f0116cf1f1cd95e05825aab" } } diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_contract.py b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py index 96fc7db..ed47971 100644 --- a/benchmarks/frontierchallenge/tests/test_run_eval_contract.py +++ b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py @@ -1,8 +1,21 @@ +import tomllib from pathlib import Path SCRIPT = Path(__file__).parents[1] / "scripts" / "run_eval.sh" +def test_runtime_python_floor_matches_pinned_harbor_and_docs(): + root = SCRIPT.parents[1] + project = tomllib.loads((root / "pyproject.toml").read_text())["project"] + assert "harbor==0.20.0" in project["dependencies"] + assert project["requires-python"] == ">=3.12" + for relative in ("README.md", "docs/quickstart.md"): + text = (root / relative).read_text() + assert "Python 3.12+" in text + assert "Python 3.11+" not in text + assert "python3.12 -m venv .venv" in text + + def test_orca_preflight_uses_declared_environment_not_instruction_text(): text = SCRIPT.read_text(encoding="utf-8") From c481b67bcddbf8593bd7640e5a4d99ffc61b73e3 Mon Sep 17 00:00:00 2001 From: Zhaopeng Feng Date: Wed, 23 Sep 2026 16:50:48 +0800 Subject: [PATCH 10/10] chore: keep PR30 scoped to FrontierChallenge fixes --- .dockerignore | 13 ------- README.md | 6 +-- apodex/tests/test_deployment_config.py | 52 ++++---------------------- compose.yaml | 13 ++----- docker/run.sh | 7 +--- docs/install/docker.md | 42 +++++++-------------- 6 files changed, 29 insertions(+), 104 deletions(-) diff --git a/.dockerignore b/.dockerignore index 94a3219..5532047 100644 --- a/.dockerignore +++ b/.dockerignore @@ -5,19 +5,6 @@ benchmarks/public/datasets benchmarks/public/results benchmarks/public/tasks-generated -# FrontierChallenge's evaluator state is never part of a runtime image. -# Root-only .env/results patterns do not cover this nested benchmark. -benchmarks/frontierchallenge/.env -benchmarks/frontierchallenge/.env.* -!benchmarks/frontierchallenge/.env.example -benchmarks/frontierchallenge/.frontierchallenge -benchmarks/frontierchallenge/tasks -benchmarks/frontierchallenge/results -benchmarks/frontierchallenge/dist -benchmarks/frontierchallenge/.venv -benchmarks/frontierchallenge/venv -benchmarks/frontierchallenge/.pytest_cache -benchmarks/frontierchallenge/*.log # Legacy pre-rename paths may still contain local corpora, generated ground # truth, and run artifacts in existing checkouts. Keep them out of remote build # contexts and images even though the current runtime no longer reads them. diff --git a/README.md b/README.md index 359dcf7..77273a9 100644 --- a/README.md +++ b/README.md @@ -199,13 +199,11 @@ Chinese-speaking macOS users can use the ## Containers and local models -Build the Docker image from this checkout; no local Python environment is -needed. The organization's GHCR image is private, so the public quickstart -uses a local image and does not require registry credentials: +Pre-built `linux/amd64` and `linux/arm64` images are published to the GitHub +Container Registry, so no local Python environment is needed: ```bash cp .env.example .env -docker compose build docker compose run --rm agent ``` diff --git a/apodex/tests/test_deployment_config.py b/apodex/tests/test_deployment_config.py index 9fca2c2..b412924 100644 --- a/apodex/tests/test_deployment_config.py +++ b/apodex/tests/test_deployment_config.py @@ -1,12 +1,8 @@ from __future__ import annotations -import os import re -import shutil -import subprocess from pathlib import Path -import pytest import yaml ROOT = Path(__file__).resolve().parents[2] @@ -34,13 +30,13 @@ def _dotenv(name: str) -> dict[str, str]: return values -def test_default_compose_uses_public_local_build_and_preserves_cli_state() -> None: +def test_default_compose_pulls_release_image_and_preserves_cli_state() -> None: compose = _yaml("compose.yaml") agent = compose["services"]["agent"] - assert agent["build"]["context"] == "." - assert _fallback(agent["image"]) == "frontieragent:local" - assert agent["pull_policy"] == "never" + assert "build" not in agent + assert IMAGE in agent["image"] + assert agent["pull_policy"] == "always" assert agent["environment"]["APODEX_IN_CONTAINER"] == "1" assert agent["environment"]["SANDBOX_BACKEND"] == "container" assert "security_opt" not in agent @@ -63,48 +59,16 @@ def test_default_compose_uses_public_local_build_and_preserves_cli_state() -> No assert agent["environment"]["APODEX_WORKSPACE_LINK"] == "/workspace" -def test_development_compose_forces_rebuild_of_public_local_image() -> None: +def test_development_compose_is_the_only_compose_file_that_builds() -> None: compose = _yaml("compose.yaml") development = _yaml("compose.dev.yaml") - for service in compose["services"].values(): - assert service["build"]["context"] == "." - assert service["pull_policy"] == "never" - assert _fallback(service["image"]) == "frontieragent:local" + assert all("build" not in service for service in compose["services"].values()) assert development["services"]["agent"]["build"]["context"] == "." assert development["services"]["eval"]["build"]["context"] == "." assert development["services"]["agent"]["pull_policy"] == "build" -def test_docker_context_excludes_frontierchallenge_evaluator_state() -> None: - patterns = (ROOT / ".dockerignore").read_text().splitlines() - prefix = "benchmarks/frontierchallenge/" - for path in (".env", ".env.*", ".frontierchallenge", "tasks", "results", - "dist", ".venv", "venv", "*.log"): - assert prefix + path in patterns - example = "!" + prefix + ".env.example" - assert example in patterns - assert patterns.index(example) > patterns.index(prefix + ".env.*") - - -@pytest.mark.parametrize("arguments,service", [(["-p", "hello"], "agent"), (["eval", "--limit", "5"], "eval")]) -def test_docker_helper_reuses_image_from_repository_directory(tmp_path, arguments, service): - repo = tmp_path / "repo" - (repo / "docker").mkdir(parents=True) - shutil.copy2(ROOT / "docker/run.sh", repo / "docker/run.sh") - binaries = tmp_path / "bin" - binaries.mkdir() - fake = binaries / "docker" - fake.write_text('#!/bin/sh\npwd\nprintf "%s\\n" "$@"\n') - fake.chmod(0o755) - result = subprocess.run(["bash", str(repo / "docker/run.sh"), *arguments], - cwd=tmp_path, capture_output=True, text=True, check=True, - env={**os.environ, "PATH": f"{binaries}:{os.environ['PATH']}"}) - lines = result.stdout.splitlines() - assert Path(lines[0]).resolve() == repo.resolve() - assert lines[1:7] == ["compose", "run", "--pull", "never", "--rm", service] - - def test_sglang_compose_mounts_an_optional_local_checkpoint_read_only() -> None: compose = _yaml("compose.sglang.yaml") model = compose["services"]["model"] @@ -347,9 +311,7 @@ def test_user_docs_use_the_published_registry_name() -> None: # rather than repeating them. paths = [ROOT / "docs/install/docker.md", ROOT / "compose.yaml"] - assert IMAGE in paths[0].read_text(encoding="utf-8") - assert "frontieragent:local" in paths[1].read_text(encoding="utf-8") - assert "docker compose build" in paths[0].read_text(encoding="utf-8") + assert all(IMAGE in path.read_text(encoding="utf-8") for path in paths) assert "docs/install/docker.md" in (ROOT / "README.md").read_text(encoding="utf-8") # The hyphenated spelling is not the published name and must appear nowhere. diff --git a/compose.yaml b/compose.yaml index 226840b..d8b0644 100644 --- a/compose.yaml +++ b/compose.yaml @@ -1,10 +1,7 @@ services: agent: - image: ${FRONTIER_AGENT_IMAGE:-frontieragent:local} - # Public users build locally; the organization's GHCR image is private. - pull_policy: never - build: - context: . + image: ${FRONTIER_AGENT_IMAGE:-ghcr.io/apodexai/frontieragent:latest} + pull_policy: always env_file: - path: .env required: false @@ -44,10 +41,8 @@ services: tty: true eval: - image: ${FRONTIER_AGENT_IMAGE:-frontieragent:local} - pull_policy: never - build: - context: . + image: ${FRONTIER_AGENT_IMAGE:-ghcr.io/apodexai/frontieragent:latest} + pull_policy: always env_file: - path: .env required: false diff --git a/docker/run.sh b/docker/run.sh index 53286cc..c355f3b 100755 --- a/docker/run.sh +++ b/docker/run.sh @@ -8,9 +8,6 @@ set -euo pipefail # ./docker/run.sh eval --limit 1 # Run benchmark evaluation repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -cd "$repo_root" -# Run `docker compose build` once and after source updates. The default local -# image is reused without requiring access to the organization's private GHCR. export APODEX_HOST_UID="$(id -u)" export APODEX_HOST_GID="$(id -g)" export APODEX_LOCAL_UTC_OFFSET="$(date +%z)" @@ -21,10 +18,10 @@ if [ "${1:-}" = "eval" ]; then shift # `docker compose run SERVICE ARGS...` replaces the service command, so # include the required benchmark defaults before forwarding overrides. - exec docker compose run --pull never --rm eval \ + exec docker compose run --rm eval \ --benchmark browsecomp \ --out /app/results/smoke \ "$@" else - exec docker compose run --pull never --rm agent "$@" + exec docker compose run --rm agent "$@" fi diff --git a/docs/install/docker.md b/docs/install/docker.md index ed098c5..4a8f433 100644 --- a/docs/install/docker.md +++ b/docs/install/docker.md @@ -1,17 +1,16 @@ # Run FrontierAgent in Docker -The public Docker workflow builds FrontierAgent from this checkout for your -host architecture (`linux/amd64` or `linux/arm64`). It requires Docker and -network access to download build dependencies, but no local Python environment. -The organization's GHCR package is private. The default `compose.yaml` builds -`frontieragent:local` and reuses it with `pull_policy: never`. +FrontierAgent publishes pre-built `linux/amd64` and `linux/arm64` images to the +GitHub Container Registry. Using them requires no local Python environment and +no system dependencies beyond Docker itself. The default `compose.yaml` pulls +that published image; it does not build the repository locally. This page covers the CPU agent container. For a **local NVIDIA model server**, the GPU belongs to a separate SGLang container or process — use [Docker SGLang on a Linux NVIDIA host](linux-nvidia.md) or [Native SGLang without nested Docker](linux-nvidia-native.md) instead. -## Build and run with Compose +## One-click Compose run `compose.yaml` marks `.env` as optional, which requires Docker Compose 2.24 or newer; older versions reject the file outright. @@ -20,7 +19,6 @@ newer; older versions reject the file outright. git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent cp .env.example .env -docker compose build # Interactive CLI docker compose run --rm agent @@ -37,37 +35,25 @@ Its named state volume is retained for legacy sessions. Attached inputs are copied into a separate volume that tools can only read. See [run artifacts and timestamps](../run-artifacts.md) for the on-disk layout. -After building, the convenience helper reuses that local image: +The convenience helper wraps the same thing: ```bash ./docker/run.sh -p "analyze repository structure" ./docker/run.sh eval --limit 5 ``` -The build excludes FrontierChallenge's nested credentials, default evaluator -staging, task caches, and results. Keep any custom dataset/staging directories -outside the checkout as well: build contexts must never contain private grader -material or local secrets. `.env.example` and public runtime source remain in -the image. The FrontierChallenge task image itself is downloaded from HF via -its [separate Quickstart](../../benchmarks/frontierchallenge/docs/quickstart.md). - ## Pin a release or another image -Users with access to the private GHCR package can explicitly log in and pull -an image. Set the same `FRONTIER_AGENT_IMAGE` when running Compose, which then -uses the downloaded image without pulling or building it. Do not run -`docker compose build` with this override: that would replace the local tag. +Set `FRONTIER_AGENT_IMAGE` before running Compose: ```bash -docker login ghcr.io -docker pull ghcr.io/apodexai/frontieragent:latest FRONTIER_AGENT_IMAGE=ghcr.io/apodexai/frontieragent:latest \ docker compose run --rm agent -p "explain pyproject.toml" ``` ## Direct `docker run` -First run `docker compose build`. Compose is the supported path; this is the equivalent for environments that +Compose is the supported path; this is the equivalent for environments that cannot use it. The environment variables and mounts are not optional — they are what tells the runtime it is inside a container and where the three sandbox roots live. @@ -91,7 +77,7 @@ docker run --rm -it \ -v frontier-agent-state:/root/.apodex \ -v frontier-agent-config:/root/.config/apodex \ -w /workspace \ - frontieragent:local \ + ghcr.io/apodexai/frontieragent:latest \ -p "explain main workflow" ``` @@ -101,26 +87,26 @@ For a terminal deployment accessed over SSH: 1. Provision an EC2 or ECS Linux instance with Docker and the Compose plugin. 2. Clone this repository and create `.env` from `.env.example`. -3. Build and launch the local container: +3. Pull and launch the pre-built container: ```bash git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent cp .env.example .env # Edit .env, then: -docker compose build +docker compose pull agent docker compose run --rm agent ``` The container itself is disposable; Compose persists sessions, configuration, attachments, and deliverables in volumes or the checked-out workspace. Pull the -source updates and rebuild with `docker compose build` to upgrade. This is an interactive SSH/TUI deployment, not a +image again to upgrade. This is an interactive SSH/TUI deployment, not a long-running HTTP service. ## Build from the current checkout -The default build already uses the checkout. For development, the override -forces a rebuild on each launch: +To run your own changes instead of the published image, add the development +override: ```bash cp .env.example .env