diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index a4a654c..8ff8966 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -29,7 +29,7 @@ simulation, electrochemistry, quantitative imaging, and molecular biology. Tasks97 (74 hard, 23 medium) Taxonomy6 domains, 21 subdomains - Runtime81 open-image tasks, 16 user-supplied ORCA tasks + Runtime81 open-image tasks, 16 tasks executing user-supplied ORCA Gradingdeterministic checks; 77 tasks also judge the report HarnessHarbor 0.20.0 Outputnamed files under /app/output @@ -38,13 +38,15 @@ simulation, electrochemistry, quantitative imaging, and molecular biology. ## End-to-end workflow -Requirements: Linux x86-64, Python 3.11+, Docker with Compose, model +Requirements: Linux x86-64, Python 3.12+ (Harbor 0.20.0), Docker with Compose, model and judge credentials, and a Hugging Face token while either dataset is private or gated. ```bash git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent/benchmarks/frontierchallenge +python3.12 -m venv .venv +source .venv/bin/activate python -m pip install -e . cp .env.example .env ``` @@ -59,10 +61,10 @@ its SHA-256, and load it into Docker: HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -The full track adds 16 normally released ORCA tasks. FrontierChallenge does -not distribute ORCA or an image containing it. After obtaining ORCA 6.0.1 from -its official provider, build and smoke-test the private local runtime, then -validate the full track: +The full track adds 16 normally released tasks that execute ORCA. +FrontierChallenge does not distribute ORCA or an image containing it. After +obtaining ORCA 6.0.1 from its official provider, build and smoke-test the +private local runtime, then validate the full track: ```bash ./scripts/build_orca_runtime.sh \ @@ -74,6 +76,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track full Do not push, export, publish, or share the resulting ORCA image. See the [ORCA setup tutorial](docs/providers/orca.md). +Track membership describes what a task executes, not where its input files +came from. For example, `task_098_orca_claisen_thermochemistry` reads supplied +ORCA output but does not run ORCA, so it belongs to the open track. + ### 2. Run a real task Fill `.env`, then run Harbor with the Claude Code agent: @@ -101,8 +107,12 @@ cat results/harbor///verifier/reward.json cat results/harbor//summary.json ``` -`passed` is the task's own pass decision; do not derive it from a global score -threshold. `task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms +Official **Pass Rate** counts completed evaluations with **`task_score > 0.999`** +over all 97 tasks. **Score** is the mean `task_score` over 97, multiplied by 100. +Missing or failed evaluations contribute zero. `passed` has this single meaning +in both `reward.json` and summaries; no alternate pass field is emitted. +The comparison is strict and uses unrounded scores: exactly `0.999` does not pass. +`task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms that grading finished. See [Quickstart](docs/quickstart.md) for credentials and expected output, and [Scoring](docs/scoring.md) for aggregate reporting. diff --git a/benchmarks/frontierchallenge/TASKS.md b/benchmarks/frontierchallenge/TASKS.md index 6fa43fd..527bcb6 100644 --- a/benchmarks/frontierchallenge/TASKS.md +++ b/benchmarks/frontierchallenge/TASKS.md @@ -14,7 +14,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` The table below previews benchmark coverage without exposing evaluator data. -Keywords come from each task and describe technique rather than answers. +Keywords come from each task and describe technique rather than answers. The +`Image` column records what the task executes, not software named in supplied +files: `task_098_orca_claisen_thermochemistry`, for example, reads precomputed +ORCA output and therefore uses the open image. | Task | Difficulty | Image | Judge | Agent budget | Techniques | |---|---|---|---|---|---| diff --git a/benchmarks/frontierchallenge/docs/huggingface-release.md b/benchmarks/frontierchallenge/docs/huggingface-release.md index 5b83e3a..583bfee 100644 --- a/benchmarks/frontierchallenge/docs/huggingface-release.md +++ b/benchmarks/frontierchallenge/docs/huggingface-release.md @@ -19,9 +19,12 @@ workspace. HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -Setup downloads the current `main` branches, runs the verification tool bundled -with each dataset, and requires both `source_registry.json` files to equal this -checkout's `registry.json`. A mixed or incomplete dataset is refused. +Setup downloads the exact solve and reference commits declared in +`release/datasets.json`, runs the verification tool bundled with each dataset, +and requires both `source_registry.json` files to equal this checkout's +`registry.json`. A mixed or incomplete dataset is refused. Release maintainers +may test newer snapshots with `--revision` and `--reference-revision`; published +runtime changes should update both pins together. Use local directories instead of HF repository IDs for an offline handoff: @@ -44,3 +47,14 @@ the encrypted verifier hash. GitHub contains neither payload. The solve dataset must contain no `tests/`, verifier archive, rubric, fixture, or reference output; the reference dataset must contain no instruction, input, or runtime environment. + +The top-level Hugging Face `README.md` is intentionally outside +`checksums.sha256`: it is a mutable dataset card whose citation and links may be +edited without changing the benchmark payload. Task files, task-level READMEs, +registries, manifests, image artifacts, and verifier archives remain covered by +the checksum manifests and registry commitments. + +After payload edits, regenerate the affected checksum entries and run both +bundled verification tools before publishing. Dataset-card-only edits require +no payload checksum change. Update the runtime's pinned HF revisions after +publishing; see [Scoring](scoring.md) for the metric contract shared by both cards. diff --git a/benchmarks/frontierchallenge/docs/providers/docker.md b/benchmarks/frontierchallenge/docs/providers/docker.md index 34370d8..c791448 100644 --- a/benchmarks/frontierchallenge/docs/providers/docker.md +++ b/benchmarks/frontierchallenge/docs/providers/docker.md @@ -92,7 +92,7 @@ path. ## Two things that bite **ORCA is user-supplied and writes beside its input.** Before selecting one of -the 16 ORCA tasks, create the licensed local runtime described in +the 16 tasks that execute ORCA, create the licensed local runtime described in [orca.md](orca.md). Copy ORCA inputs into a writable directory (`/app/data`, `/tmp`) before running; invoking ORCA directly on a read-only bind-mounted file fails. diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 8a11fe9..f918596 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -6,7 +6,9 @@ Hugging Face datasets, a real Harbor + Claude Code run, and the final score. ## Requirements - Linux x86-64 with Docker and Compose v2; -- Python 3.11+ and about 20 GB for the open image; +- Python 3.12+ on the evaluator host (required by Harbor 0.20.0); + allow at least 40 GB of free disk for the downloaded archive, + Docker image, and working data (more for concurrent runs and results); - a model API key and a judge API key; - `HF_TOKEN` while either dataset is private or gated; - for the full track only, an official ORCA 6.0.1 download and permission to @@ -24,6 +26,8 @@ docker compose version ```bash git clone https://github.com/ApodexAI/FrontierAgent.git cd FrontierAgent/benchmarks/frontierchallenge +python3.12 -m venv .venv +source .venv/bin/activate python -m pip install -e . cp .env.example .env ``` @@ -61,19 +65,32 @@ This is the shortest path for most evaluators: HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -Setup downloads the solve and reference datasets from their current `main` branches, -verifies both packages, binds them to this checkout's `registry.json`, then +Setup downloads the solve and reference revisions pinned by this Git checkout in +`release/datasets.json`, verifies both packages, binds them to this checkout's +`registry.json`, then downloads `images/frontierchallenge-cpu-open-2026.08.docker.tar.zst` from the -solve dataset. It checks the declared size, SHA-256 and image ID before loading -the `linux/amd64` image into Docker. No container registry is used. Evaluator- +solve dataset. It checks the archive's declared size and SHA-256 before loading +the `linux/amd64` image into Docker, then verifies the loaded image identity. +No container registry is used. Evaluator- local paths are written to `.frontierchallenge/config.env`. +Docker's classic and containerd image stores expose different image IDs. Setup +accepts the published config digest directly, or verifies that the loaded OCI +manifest digest links to that exact config inside the SHA-256-verified archive. +Keep runtime dependencies current with `python -m pip install -e .`; do not +disable identity checks or change Docker's storage backend to work around this. + +For release development only, `--revision main` overrides both pins; +`--reference-revision` can override the reference revision independently. Normal +evaluation should keep the checkout's pins so later dataset changes cannot alter +an otherwise identical run. + ### Full track: build the private ORCA runtime -All 16 ORCA task statements and inputs are released normally. Only ORCA and a -configured ORCA image are absent. Obtain ORCA 6.0.1 from its official provider, -install it outside this checkout, and keep the complete directory together. -Then run: +All statements and inputs for the 16 tasks that execute ORCA are released +normally. Only ORCA and a configured ORCA image are absent. Obtain ORCA 6.0.1 +from its official provider, install it outside this checkout, and keep the +complete directory together. Then run: ```bash ./scripts/build_orca_runtime.sh \ @@ -118,6 +135,11 @@ tasks into evaluator staging, decrypts the matching verifier there, starts the agent, and invokes Harbor's verifier after the agent exits. By default Claude Code's `WebSearch` and `WebFetch` tools are disabled. +Selection comes from each task's declared `task.json.environment`, validated +against the registry. Include/exclude filters are applied before image preflight, +staging, verifier decryption, resume, and Harbor invocation; stale directories +from an older run cannot add tasks to the effective run. + A healthy run reaches messages like: ```text @@ -146,10 +168,14 @@ cat results/harbor///verifier/reward.json ``` - `evaluation_complete = 1` means the verifier finished; -- `passed` is the task's own pass decision and must not be recomputed from a - global threshold; +- official Pass Rate counts completed `task_score > 0.999` evaluations over 97; +- `passed` uses this same strict threshold in both rewards and summaries; - `task_score` is a continuous score in `[0, 1]`. +Score is the mean `task_score` over 97, times 100. Missing and failed evaluations +contribute zero. Use unrounded scores: exactly `0.999` does not pass. See +[Scoring](scoring.md) for subsets, repeated attempts, and historical results. + The job aggregate is: ```bash diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index faff611..8c3524a 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -12,7 +12,8 @@ The open track contains the 81 tasks that use the redistributable image: ./scripts/run_eval.sh --agent claude-code --model ``` -The full track adds 16 ORCA tasks. Prepare the licensed local runtime first: +The full track adds 16 tasks that execute ORCA. Prepare the licensed local +runtime first: ```bash ./scripts/build_orca_runtime.sh --orca-root /path/to/orca-6.0.1 @@ -22,6 +23,10 @@ The full track adds 16 ORCA tasks. Prepare the licensed local runtime first: Setup writes verified local paths under `.frontierchallenge/`. The runner validates the GitHub/solve/reference registries again before staging anything. +Track membership follows each task's declared execution environment, not +software names in its instruction or supplied files. Thus +`task_098_orca_claisen_thermochemistry`, which only reads precomputed ORCA +output, remains an open-track task. ## Runtime @@ -44,7 +49,9 @@ missing. See [Docker](providers/docker.md). ``` Use disjoint include lists and distinct job names to shard across machines. -`summarize_results.py` accepts multiple job directories and merges them. +`summarize_results.py` accepts one job directory at a time; it does not merge +shards. A shard's report is partial, not a separate full-benchmark result. +Do not average shard Pass Rates as though they were full runs. Concurrency must fit both machine resources and model-provider rate limits. Start with one task, then increase gradually. @@ -62,14 +69,27 @@ lowering longer ones: Use a separate stage directory for each concurrent run that changes timeouts. Verifier timeouts use `--verifier-timeout-multiplier` (default 40). +Legacy staging caches containing Hugging Face symlinks are automatically +rebuilt once. Only the effective include/exclude selection is staged, checked +for ORCA, unsealed, and passed to Harbor; leftover stage directories are ignored. + ## Resume and results -Reusing a job name resumes completed work when the requested task set matches: +Reusing a job name resumes completed work only when the task selection and +scoring/log policy match the recorded job: ```bash ./scripts/run_eval.sh --agent claude-code --model --job-name ``` +The runner records its policy in +`/.frontierchallenge-policies/.json`. Keep that sidecar +alongside jobs when moving them. A missing/old policy marker, changed task +selection, or invalid job metadata is refused before staging. Use a fresh +`--job-name` or `--jobs-dir`; old rewards and logs are not silently rewritten +or mixed into a new-policy job. Existing results can still be summarized +separately from their scores, but their original rewards remain historical. + Results are written under `results/harbor//`. Read the aggregate with: ```bash @@ -77,6 +97,13 @@ python3 scripts/summarize_results.py results/harbor/ cat results/harbor//summary.json ``` +Official Pass Rate requires completed `task_score > 0.999`; `passed` means the +same thing in rewards and summaries. The default denominator is 97, including missing tasks. +See [Scoring](scoring.md) for partial scores and explicitly labeled subsets. +Automatic summaries use the runner's current selection. If a job directory +contains trials from an older selection, use repeatable `--task-id ` with +the standalone summarizer, or start a fresh job directory. + Before a long run, verify one task reaches `evaluation_complete = 1`, confirm the selected backend in the startup banner, and confirm the local ORCA runtime before selecting the full track. diff --git a/benchmarks/frontierchallenge/docs/scoring.md b/benchmarks/frontierchallenge/docs/scoring.md index 229fd4a..bfceb96 100644 --- a/benchmarks/frontierchallenge/docs/scoring.md +++ b/benchmarks/frontierchallenge/docs/scoring.md @@ -2,12 +2,20 @@ FrontierChallenge reports two numbers over a fixed denominator of 97 tasks: -- **Pass Rate:** tasks whose verifier writes `passed = 1`, divided by 97. +- **Pass Rate:** completed evaluations with `task_score > 0.999`, divided by 97. - **Score:** the mean of `task_score` across all 97, usually reported times 100. Unrun tasks and harness failures count as zero in the fixed denominator. The -summarizer marks an incomplete run as partial instead of averaging only the -tasks that happened to finish. +summarizer marks an incomplete run as partial while retaining the denominator +97. It never drops missing or failed tasks from the headline metrics. + +The comparison is strict: `0.999` does not pass; `0.9991` and `1.0` pass. +Use unrounded scores without an additional epsilon or per-task threshold. +Compare the stored `task_score`, not a rounded display: for example, +`0.9990000000000001` passes even if displayed as `0.999`. Score normalization +and partial-credit arithmetic are unchanged. +`evaluation_complete == 1` is required for a valid score. Invalid scores +(non-numeric, non-finite, or outside `[0, 1]`) contribute zero and are flagged. ## Authoritative fields @@ -15,19 +23,51 @@ Each trial writes `verifier/reward.json`: | Field | Meaning | |---|---| -| `passed` | the task's own pass decision; do not derive it from a global threshold | +| `passed` | 1 only when evaluation completed and valid `task_score > 0.999`; otherwise 0 | | `task_score` | score from 0 to 1 | | `evaluation_complete` | whether verification completed | -The 97 verifiers do not share one pass threshold. For full-mark counts, use -`task_score >= 0.999`; judge averaging can produce a value just below 1. +There is only one pass field, `passed`, with the same meaning in +`verifier/reward.json`, `summary.csv`, and `summary.json`. A completed score +of 0.9991 passes; a score of 0.999 does not. No alternate pass field is emitted. + +After authenticating and unsealing the reference, the runtime applies the +strict score threshold to the staged reward adapter before Harbor runs it. The +encrypted reference archives and partial-credit rubrics remain unchanged. +The summarizer also derives `passed` from score and completion when processing +older results, discarding their old pass decision rather than copying it. + +Native grader diagnostics are generated in a temporary verifier directory. +Before publication, pass decisions are removed from diagnostic JSON and text; +only `reward.json` retains the benchmark's pass field. Partial scores and +ordinary error diagnostics are preserved. Raw grader logs are not published. + +The single-field guarantee applies to new-policy jobs. Legacy jobs are refused +on resume, rather than mixing their old rewards with new results. Summarizing +an old job does not rewrite its original rewards or logs; see +[resume and upgrade rules](running.md#resume-and-results). -Summarize one or more Harbor job directories with: +This policy is identified by `metric_definition: score-gt-0.999` in the +summary. Recompute historical results from raw rewards before comparing them; +results computed with per-task thresholds, `== 1.0`, or `>= 0.999` are not interchangeable. + +Summarize a Harbor job directory with: ```bash python3 scripts/summarize_results.py results/harbor/ ``` +Use one predeclared attempt per task. Duplicate task trials are rejected rather +than silently counted twice or selected by their score. For a deliberately +separate subset report (for example, the 81-task open track), specify +`--expected-total 81` and report that denominator explicitly; it is not the +97-task headline metric. `--expected-total` must not be reduced to the number +of tasks that happened to succeed. Missing tasks still contribute zero. + +For repeated trials, use `run_eval.sh --n-attempts N --no-summary` and summarize +each predeclared attempt separately. Use this runtime's summary for the +fixed-denominator headline metric; Harbor may aggregate only attempted trials. + ## Verifiers and judges Each task has a frozen verifier. Deterministic checks validate submitted files, @@ -66,7 +106,7 @@ solve-side hash and an encrypted-verifier hash. Setup refuses mixed releases. Report: - denominator 97, with missing tasks counted as zero; -- Pass Rate from `passed`, not from a new threshold; +- Pass Rate from completed `task_score > 0.999`; - mean `task_score` times 100; - agent, model, judge model, and judge repetitions; - pinned Docker image identity and ORCA version for full-track runs; diff --git a/benchmarks/frontierchallenge/docs/submitting.md b/benchmarks/frontierchallenge/docs/submitting.md index c893cea..22e8510 100644 --- a/benchmarks/frontierchallenge/docs/submitting.md +++ b/benchmarks/frontierchallenge/docs/submitting.md @@ -31,8 +31,9 @@ ORCA version; see [Scoring](scoring.md). listed if it is labelled as one, with the count of attempted tasks. It cannot be listed as a score over a smaller denominator. -**The `passed` field, not a threshold.** See -[Scoring](scoring.md#the-two-headline-numbers). +**Pass Rate.** Count only completed evaluations with +`task_score > 0.999`; do not apply per-task thresholds or round scores. +See [Scoring](scoring.md). **Judge configuration stated.** `gpt-5.6-sol`, `reasoning_effort=high`, `JUDGE_REPEATS=3` with `--no-judge-override` is the definitional setting. Any diff --git a/benchmarks/frontierchallenge/docs/task-format.md b/benchmarks/frontierchallenge/docs/task-format.md index ca35ea7..88b099d 100644 --- a/benchmarks/frontierchallenge/docs/task-format.md +++ b/benchmarks/frontierchallenge/docs/task-format.md @@ -97,7 +97,11 @@ the task's source/provenance definition, and should be described that way: 1. runs the LLM judge `JUDGE_REPEATS` times, if the task has one (77 do), 2. combines them with `statistics.fmean`, 3. calls the task's own grader with that value as the rubric component, -4. emits `task_score` and `passed`. +4. emits `task_score` and `passed`, which is 1 only for a completed valid score above 0.999. + +The runtime applies this single pass rule to the staged reward adapter after +unsealing. Official Pass Rate is computed by the summarizer from completed +`task_score > 0.999` evaluations, independently of per-task thresholds. The full tree is stored in the gated dataset's `verifier.fcref`, including the solved reference run, reference fixtures, grader source, judge prompt, and diff --git a/benchmarks/frontierchallenge/docs/troubleshooting.md b/benchmarks/frontierchallenge/docs/troubleshooting.md index 9bece3f..71f1a48 100644 --- a/benchmarks/frontierchallenge/docs/troubleshooting.md +++ b/benchmarks/frontierchallenge/docs/troubleshooting.md @@ -1,5 +1,13 @@ # Troubleshooting +## Editable install cannot resolve Harbor + +The pinned Harbor 0.20.0 requires Python 3.12 or newer on the evaluator host. +Use a Python 3.12+ virtual environment as shown in [Quickstart](quickstart.md), +then rerun `python -m pip install -e .`. Do not downgrade Harbor to work around +an older system Python. The Python versions inside the frozen scientific task +images are separate and do not need to change. + ## Setup cannot find the datasets While either HF repository is private or gated, pass an authorized token only @@ -11,7 +19,7 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open Do not place the HF token in `.env`, which is used to configure model and judge credentials. For offline use, pass local solve and reference directories as -shown in [Quickstart](quickstart.md). +shown in [Hugging Face layout](huggingface-release.md#download-and-verify). ## Docker is installed but runs do not start @@ -40,6 +48,11 @@ or distribute it. After obtaining and installing ORCA officially, run: The build helper runs a real ORCA calculation, and the runner checks the local image again before any selected ORCA task starts. +An open task may still mention ORCA because it reads supplied output files. +That does not require the licensed runtime: preflight follows the task's +registry-backed execution environment, not instruction text. For example, +`task_098_orca_claisen_thermochemistry` belongs to the open track. + ## Every task fails during agent setup Check that `.env` contains the key required by the selected agent and that the @@ -68,6 +81,22 @@ Inspect `verifier/reward.json`: The aggregate summarizer reports incomplete and missing-artifact counts instead of silently treating every zero as the same failure mode. +## An old job is refused after upgrading + +The runner refuses populated jobs without a matching scoring/log policy marker +or with a changed task selection. Start a fresh `--job-name` (or `--jobs-dir`). +Do not manufacture a marker for old results: their raw rewards may use the old +per-task pass rule. Old results remain untouched and may be summarized separately. + +## Loaded image identity differs on Docker's containerd store + +Update the runtime and reinstall its dependencies (`python -m pip install -e .`). +Older setup code compares Docker's OCI manifest ID directly with the release's +config digest, which can reject a valid image on Docker 29/containerd. Current +setup verifies the manifest-to-config digest link inside the verified archive. +Archive size/SHA-256 and platform checks remain mandatory; no daemon restart, +storage-driver change, registry pull, or HF payload update is needed. + ## Inspect a trial ```bash diff --git a/benchmarks/frontierchallenge/pyproject.toml b/benchmarks/frontierchallenge/pyproject.toml index eaa966e..0f1bb24 100644 --- a/benchmarks/frontierchallenge/pyproject.toml +++ b/benchmarks/frontierchallenge/pyproject.toml @@ -12,7 +12,9 @@ name = "frontier-challenge" version = "0" description = "FrontierChallenge: evaluating AI systems on real scientific workflows" readme = "README.md" -requires-python = ">=3.11" +# Harbor 0.20.0 requires Python 3.12 on the evaluator host. Scientific task +# containers have their own, separately frozen interpreter versions. +requires-python = ">=3.12" authors = [{ name = "Apodex AI" }, { name = "GADE Union" }] keywords = ["benchmark", "agents", "evaluation", "scientific-computing"] license = "CC-BY-4.0" @@ -23,6 +25,7 @@ license-files = ["LICENSE"] dependencies = [ "harbor==0.20.0", "huggingface_hub>=0.33,<2", + "zstandard>=0.23,<1", ] [project.optional-dependencies] diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json new file mode 100644 index 0000000..ed1717e --- /dev/null +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -0,0 +1,11 @@ +{ + "schema_version": 1, + "solve": { + "repo": "apodex/FrontierChallenge", + "revision": "6306f5d07fcd23c911242a36c6c923e9b2ad19bf" + }, + "reference": { + "repo": "apodex/FrontierChallenge-reference", + "revision": "e3e6719c43de28cc0f0116cf1f1cd95e05825aab" + } +} diff --git a/benchmarks/frontierchallenge/scripts/apply_score_policy.py b/benchmarks/frontierchallenge/scripts/apply_score_policy.py new file mode 100644 index 0000000..442d646 --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/apply_score_policy.py @@ -0,0 +1,64 @@ +"""Set the sole pass rule in an evaluator-owned, freshly unsealed task. + +The encrypted reference and partial-credit rubric are not modified. Apply this +after archive verification/unsealing and before Harbor reads the verifier. +Fail closed if a future reference changes the known reward adapter contract. +""" +from __future__ import annotations + +import argparse +import shutil +from pathlib import Path + +OLD_RULE = '"passed": 1.0 if passed is True else 0.0,' +PASS_RULE = ( + '"passed": 1.0 if complete and 0.999 < float(score or 0.0) / 100.0 <= 1.0 else 0.0,' +) +ENTRYPOINT = '\nif __name__ == "__main__":' +LOG_WRAPPER = ''' +def main(): + # Native decisions exist only in a private temporary directory. Publish + # sanitized diagnostics on success and failure, never a second pass metric. + import tempfile + from verifier_log_policy import publish_logs + global LOGS + public_logs = LOGS + with tempfile.TemporaryDirectory(prefix="frontier-native-verifier-") as scratch: + LOGS = Path(scratch) + try: + _run_native_verifier() + finally: + try: + publish_logs(LOGS, public_logs) + finally: + LOGS = public_logs + +''' + + +def apply_policy(task_dir: Path) -> None: + adapter = task_dir / "tests" / "run_frontier_verifier.py" + text = adapter.read_text(encoding="utf-8") + if (text.count(OLD_RULE) != 1 or PASS_RULE in text + or text.count("def main():") != 1 or text.count(ENTRYPOINT) != 1): + raise ValueError(f"unsupported or already modified reward adapter: {adapter}") + updated = text.replace(OLD_RULE, PASS_RULE).replace("def main():", "def _run_native_verifier():") + updated = updated.replace(ENTRYPOINT, LOG_WRAPPER + ENTRYPOINT) + compile(updated, str(adapter), "exec") + shutil.copyfile(Path(__file__).with_name("verifier_log_policy.py"), + adapter.with_name("verifier_log_policy.py")) + adapter.write_text(updated, encoding="utf-8") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("task_dir", type=Path) + args = parser.parse_args() + try: + apply_policy(args.task_dir) + except (OSError, ValueError) as exc: + parser.error(str(exc)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/frontierchallenge/scripts/job_policy.py b/benchmarks/frontierchallenge/scripts/job_policy.py new file mode 100644 index 0000000..9a1d69b --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/job_policy.py @@ -0,0 +1,71 @@ +"""Prevent resuming or mixing jobs produced by incompatible scoring policies.""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +POLICY = {"schema": 1, "metric_definition": "score-gt-0.999", "diagnostics": "no-native-pass"} + + +def marker_path(job: Path) -> Path: + if not job.name: + raise ValueError("job must be a named directory") + return job.parent / ".frontierchallenge-policies" / f"{job.name}.json" + + +def expected_policy(task_ids: list[str]) -> dict: + return {**POLICY, "task_ids": sorted(set(task_ids))} + + +def check_job(job: Path, task_ids: list[str]) -> str: + if not job.exists() or (job.is_dir() and not any(job.iterdir())): + return "new" + try: + stored = json.loads(marker_path(job).read_text()) + if stored != expected_policy(task_ids): + raise ValueError("scoring/log policy or task selection changed") + if not (job / "config.json").is_file(): + raise ValueError("job config is missing") + lock = json.loads((job / "lock.json").read_text()) + recorded = {trial["task"]["name"] for trial in lock["trials"]} + if recorded != set(task_ids): + raise ValueError("recorded task selection differs") + except (OSError, ValueError, KeyError, TypeError) as exc: + raise ValueError( + f"cannot resume {job}: missing/incompatible FrontierChallenge policy or job metadata. " + "Use a fresh --job-name (or --jobs-dir); existing results were not modified. " + "Old results may be summarized separately, but must not be mixed into a new-policy job." + ) from exc + return "resume" + + +def record_policy(job: Path, task_ids: list[str]) -> None: + check_job(job, task_ids) + target = marker_path(job) + target.parent.mkdir(parents=True, exist_ok=True) + document = expected_policy(task_ids) + if target.exists() and json.loads(target.read_text()) == document: + return + # Only absent/empty jobs can acquire a new marker; populated old jobs fail + # the check above. No old result or old log is rewritten during an upgrade. + target.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=("check", "record")) + parser.add_argument("job", type=Path) + parser.add_argument("--task-id", action="append", required=True) + args = parser.parse_args() + try: + if args.action == "check": + print(check_job(args.job.absolute(), args.task_id)) + else: + record_policy(args.job.absolute(), args.task_id) + except (OSError, ValueError) as exc: + parser.error(str(exc)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index b522b63..c6e1985 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -72,7 +72,8 @@ Options: (default: unset, follows --n-concurrent). A lower cap here than --n-concurrent adds headroom against agent-setup timeouts at high concurrency. - --n-attempts N Attempts per task (default: 1) + --n-attempts N Attempts per task (default: 1); use --no-summary + for repeated trials, then report each attempt separately --job-name NAME Harbor job name (default: derived) --jobs-dir PATH Where Harbor writes results (default: results/harbor) --stage-dir PATH Scratch dir for evaluator-owned task copies @@ -161,6 +162,11 @@ if [[ "$TRACK" != "open" && "$TRACK" != "full" ]]; then exit 1 fi +if [[ "$N_ATTEMPTS" != "1" && "$NO_SUMMARY" -eq 0 ]]; then + echo "error: automatic metrics require one attempt per task; use --no-summary for repeated trials" >&2 + exit 1 +fi + if [[ -z "$SOLVE_DIR" || ! -d "$SOLVE_DIR" ]]; then echo "error: a downloaded solve-side HF package is required." >&2 echo " Run ./scripts/setup.sh, pass --solve-dir PATH, or set FRONTIER_SOLVE_DIR." >&2 @@ -233,34 +239,66 @@ fi mkdir -p "$STAGE_DIR" "$JOBS_DIR" -# Mirror Harbor's include/exclude matching so only selected tasks are staged. -task_selected() { - local task_id="$1" - if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then - local matched=0 - for pattern in "${INCLUDE_PATTERNS[@]}"; do - [[ "$task_id" == $pattern ]] && { matched=1; break; } - done - [[ "$matched" -eq 1 ]] || return 1 - fi +# Resolve selection once from the verified solve package. Persistent staging may +# contain tasks from older runs, so it must never define preflight or grading. +SELECTION_ARGS=( + --tasks-root "$SOLVE_TASKS" + --registry "$SOLVE_DIR/source_registry.json" + --track "$TRACK" +) +if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then + for pattern in "${INCLUDE_PATTERNS[@]}"; do + SELECTION_ARGS+=(--include "$pattern") + done +fi +if [[ ${#EXCLUDE_PATTERNS[@]} -gt 0 ]]; then for pattern in "${EXCLUDE_PATTERNS[@]}"; do - [[ "$task_id" == $pattern ]] && return 1 + SELECTION_ARGS+=(--exclude "$pattern") done - return 0 -} +fi + +if ! SELECTION_OUTPUT="$( + python3 "$ROOT/scripts/task_selection.py" "${SELECTION_ARGS[@]}" +)"; then + echo "FATAL: task selection is invalid." >&2 + exit 1 +fi +EFFECTIVE_TASK_IDS=() +EFFECTIVE_TASK_ENVS=() +EFFECTIVE_TASK_SOURCES=() +while IFS=$'\t' read -r task_id task_environment task_source; do + [[ -n "$task_id" ]] || continue + EFFECTIVE_TASK_IDS+=("$task_id") + EFFECTIVE_TASK_ENVS+=("$task_environment") + EFFECTIVE_TASK_SOURCES+=("$task_source") +done <<< "$SELECTION_OUTPUT" +unset SELECTION_OUTPUT +if [[ ${#EFFECTIVE_TASK_IDS[@]} -eq 0 ]]; then + echo "FATAL: task selection is empty." >&2 + exit 1 +fi + +POLICY_TASK_ARGS=() +for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + POLICY_TASK_ARGS+=(--task-id "$task_id") +done +# Fail before modifying staging when an old job would mix scoring/log policies. +JOB_ACTION="$(python3 "$ROOT/scripts/job_policy.py" check \ + "$JOBS_DIR/$JOB_NAME" "${POLICY_TASK_ARGS[@]}")" echo "== Staging $TRACK-track tasks from $SOLVE_TASKS into $STAGE_DIR ==" staged=0 skipped=0 -for task_dir in "$SOLVE_TASKS"/*/; do - task_id="$(basename "$task_dir")" - [[ -f "$task_dir/task.toml" ]] || continue - if [[ "$TRACK" == "open" ]] && ! grep -q '"environment": "open"' "$task_dir/task.json"; then - continue - fi - task_selected "$task_id" || continue +EFFECTIVE_TASK_DIRS=() +for index in "${!EFFECTIVE_TASK_IDS[@]}"; do + task_id="${EFFECTIVE_TASK_IDS[$index]}" + task_environment="${EFFECTIVE_TASK_ENVS[$index]}" + task_dir="${EFFECTIVE_TASK_SOURCES[$index]}" dest="$STAGE_DIR/$task_id" - source_identity="$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE" + EFFECTIVE_TASK_DIRS+=("$dest") + # Invalidate stages produced by the legacy cp -a runner. Their nested HF + # cache-relative symlinks can be broken even when task.toml remains readable. + source_identity="dereferenced-v2|$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE" if [[ "$FORCE_RESTAGE" -eq 0 && -f "$dest/task.toml" \ && -f "$dest/instruction.md" && ! -e "$dest/statement.fcref" \ && -f "$dest/.frontier-source" ]] \ @@ -271,9 +309,12 @@ for task_dir in "$SOLVE_TASKS"/*/; do # Copy the verified solve task and pin its open-image Dockerfile to setup's # selected reference. The HF source is immutable; only the staged copy changes. rm -rf "$dest" - cp -a "$task_dir" "$dest" + # Hugging Face snapshots may expose files as cache-relative symlinks. Copy + # their contents so the evaluator stage cannot contain broken links after it + # leaves the snapshot directory hierarchy. + cp -aL "$task_dir" "$dest" printf '%s\n' "$source_identity" > "$dest/.frontier-source" - if grep -q '"environment": "open"' "$dest/task.json"; then + if [[ "$task_environment" == "open" ]]; then OPEN_IMAGE="$OPEN_IMAGE" python3 - "$dest/environment/Dockerfile" <<'PIN_OPEN_IMAGE' import os import pathlib @@ -316,15 +357,15 @@ if [[ "$AGENT" == "claude-code" && ${#AGENT_KWARGS[@]} -eq 0 ]]; then # Prevent direct web lookup unless the evaluator deliberately overrides this. AGENT_KWARG_ARGS+=(--agent-kwarg "disallowed_tools=WebSearch WebFetch") fi -AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}") +if [[ ${#AGENT_KWARGS[@]} -gt 0 ]]; then + AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}") +fi INCLUDE_ARGS=() -for pattern in "${INCLUDE_PATTERNS[@]}"; do - INCLUDE_ARGS+=("--include-task-name" "$pattern") -done -EXCLUDE_ARGS=() -for pattern in "${EXCLUDE_PATTERNS[@]}"; do - EXCLUDE_ARGS+=("--exclude-task-name" "$pattern") +# Give Harbor the exact resolved IDs. This makes stale directories in a reused +# stage invisible even when the user supplied no include/exclude flags. +for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + INCLUDE_ARGS+=("--include-task-name" "$task_id") done # A command-line verifier env takes precedence over the task declaration. @@ -388,18 +429,9 @@ fi # Fail before evaluation if a selected task needs ORCA but the evaluator-local # licensed runtime is unavailable. orca_tasks=() -for task_dir in "$STAGE_DIR"/*/; do - [[ -d "$task_dir" ]] || continue - task_name="$(basename "$task_dir")" - if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then - matched=0 - for pattern in "${INCLUDE_PATTERNS[@]}"; do - [[ "$task_name" == *"$pattern"* ]] && { matched=1; break; } - done - [[ "$matched" -eq 1 ]] || continue - fi - if grep -qil 'orca' "$task_dir/task.toml" "$task_dir/instruction.md" "$task_dir/environment/Dockerfile" 2>/dev/null; then - orca_tasks+=("$task_name") +for index in "${!EFFECTIVE_TASK_IDS[@]}"; do + if [[ "${EFFECTIVE_TASK_ENVS[$index]}" == "licensed-orca" ]]; then + orca_tasks+=("${EFFECTIVE_TASK_IDS[$index]}") fi done if [[ ${#orca_tasks[@]} -gt 0 ]]; then @@ -430,32 +462,6 @@ if [[ -n "$N_CONCURRENT_AGENTS" ]]; then CONCURRENT_AGENTS_ARGS+=(--n-concurrent-agents "$N_CONCURRENT_AGENTS") fi -# Resume only when the recorded and requested task sets match. Otherwise keep -# completed trial directories but archive stale job-level metadata. -RESUME_JOB=0 -if [[ -f "$JOBS_DIR/$JOB_NAME/config.json" ]]; then - if REQUESTED="${INCLUDE_PATTERNS[*]-}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY' -import json, os, sys -requested = set(os.environ.get("REQUESTED", "").split()) -try: - recorded = {t["task"]["name"] for t in json.load(open(sys.argv[1]))["trials"]} -except Exception: - sys.exit(1) # unreadable lock -> treat as new work -# No --include means "the whole staged set", which resume also covers. -sys.exit(0 if not requested or requested == recorded else 1) -PY - then - RESUME_JOB=1 - else - echo "== Task set changed - archiving stale job files in $JOBS_DIR/$JOB_NAME ==" - stamp=$(date +%Y%m%d-%H%M%S) - for f in config.json lock.json result.json job.log; do - [[ -e "$JOBS_DIR/$JOB_NAME/$f" ]] && \ - mv "$JOBS_DIR/$JOB_NAME/$f" "$JOBS_DIR/$JOB_NAME/.prev-$stamp-$f" - done - fi -fi - # The real verifier is distributed only through the separate encrypted # reference dataset. Download and verify that package first, then point this # runner at the resulting directory. The archive password is intentionally @@ -469,10 +475,8 @@ if [[ -f "$REFERENCE_DIR/tools/verify_reference_dataset.py" ]]; then fi echo "== Injecting encrypted verifier archives from $REFERENCE_DIR ==" injected=0 -for task_dir in "$STAGE_DIR"/*/; do - [[ -f "$task_dir/task.toml" ]] || continue +for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do task_id="$(basename "$task_dir")" - task_selected "$task_id" || continue source_verifier="$REFERENCE_TASKS/$task_id/verifier.fcref" if [[ ! -f "$source_verifier" ]]; then echo "FATAL: encrypted verifier missing for $task_id: $source_verifier" >&2 @@ -497,8 +501,7 @@ echo "Injected $injected encrypted verifier archive(s)." if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "== Unsealing encrypted verifiers with the published archive password ==" unsealed=0 - for task_dir in "$STAGE_DIR"/*/; do - [[ -f "$task_dir/task.toml" ]] || continue + for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do if [[ ! -f "$task_dir/instruction.md" || ! -f "$task_dir/verifier.fcref" ]]; then echo "FATAL: $(basename "$task_dir") lacks plaintext instruction or verifier archive." >&2 exit 1 @@ -513,12 +516,17 @@ if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "FATAL: $(basename "$task_dir") has no verifier entrypoint after unsealing." >&2 exit 1 fi + # Emit only the benchmark-wide task_score > 0.999 decision, before Harbor + # reads reward.json. No alternate native pass metric is retained. + python3 "$ROOT/scripts/apply_score_policy.py" "$task_dir" unsealed=$((unsealed + 1)) done echo "Unsealed $unsealed task(s)." fi -if [[ "$RESUME_JOB" -eq 1 ]]; then +# Record outside Harbor's job directory so a fresh job remains fresh to Harbor. +python3 "$ROOT/scripts/job_policy.py" record "$JOBS_DIR/$JOB_NAME" "${POLICY_TASK_ARGS[@]}" +if [[ "$JOB_ACTION" == "resume" ]]; then echo "== Resuming existing job dir: $JOBS_DIR/$JOB_NAME ==" PYTHONPATH="$ROOT" HARBOR_TELEMETRY=off harbor job resume \ --job-path "$JOBS_DIR/$JOB_NAME" @@ -528,17 +536,21 @@ else --path "$STAGE_DIR" \ --env docker \ --agent "$AGENT" --model "$MODEL" \ - --n-attempts "$N_ATTEMPTS" --n-concurrent "$N_CONCURRENT" "${CONCURRENT_AGENTS_ARGS[@]}" \ + --n-attempts "$N_ATTEMPTS" --n-concurrent "$N_CONCURRENT" ${CONCURRENT_AGENTS_ARGS[@]+"${CONCURRENT_AGENTS_ARGS[@]}"} \ --verifier-timeout-multiplier "$VERIFIER_TIMEOUT_MULTIPLIER" \ --agent-setup-timeout-multiplier "$AGENT_SETUP_TIMEOUT_MULTIPLIER" \ --artifact /app/output \ --jobs-dir "$JOBS_DIR" --job-name "$JOB_NAME" \ --env-file "$ENV_FILE" \ - "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${EXCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \ + ${AGENT_KWARG_ARGS[@]+"${AGENT_KWARG_ARGS[@]}"} "${INCLUDE_ARGS[@]}" ${VERIFIER_ENV_ARGS[@]+"${VERIFIER_ENV_ARGS[@]}"} \ --yes fi if [[ "$NO_SUMMARY" -eq 0 ]]; then echo "== Summarizing $JOBS_DIR/$JOB_NAME ==" - python3 scripts/summarize_results.py "$JOBS_DIR/$JOB_NAME" + SUMMARY_TASK_ARGS=() + for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + SUMMARY_TASK_ARGS+=(--task-id "$task_id") + done + python3 scripts/summarize_results.py "$JOBS_DIR/$JOB_NAME" "${SUMMARY_TASK_ARGS[@]}" fi diff --git a/benchmarks/frontierchallenge/scripts/setup_release.py b/benchmarks/frontierchallenge/scripts/setup_release.py index f8cc997..0f93031 100755 --- a/benchmarks/frontierchallenge/scripts/setup_release.py +++ b/benchmarks/frontierchallenge/scripts/setup_release.py @@ -7,17 +7,30 @@ import hashlib import json import os +import re import shlex import shutil import subprocess import sys +import tarfile from pathlib import Path - ROOT = Path(__file__).resolve().parents[1] -DEFAULT_SOLVE_REPO = "apodex/FrontierChallenge" -DEFAULT_REFERENCE_REPO = "apodex/FrontierChallenge-reference" -DEFAULT_REVISION = "main" + + +def select_revisions( + release: dict, + revision_override: str | None, + reference_revision_override: str | None, +) -> tuple[str, str]: + """Return pinned defaults, preserving the legacy --revision override.""" + solve_revision = revision_override or release["solve"]["revision"] + reference_revision = ( + reference_revision_override + or revision_override + or release["reference"]["revision"] + ) + return solve_revision, reference_revision def digest(path: Path) -> str: @@ -229,6 +242,58 @@ def validate_orca_runtime(image: str) -> None: ) +def verify_oci_image_identity(stream, image_id: str, config_id: str) -> None: + """Bind a containerd manifest ID to the release's frozen config digest. + + The caller has already verified the entire archive's SHA-256. Read only + metadata, never extract files or change Docker's storage configuration. + """ + for value in (image_id, config_id): + if not isinstance(value, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", value): + raise SystemExit("invalid image identity digest") + member_name = "blobs/sha256/" + image_id.removeprefix("sha256:") + with tarfile.open(fileobj=stream, mode="r|") as archive: + for member in archive: + if member.name != member_name: + continue + if not member.isfile() or member.size > 1024 * 1024: + raise SystemExit("invalid OCI image manifest entry") + raw = archive.extractfile(member).read() + if "sha256:" + hashlib.sha256(raw).hexdigest() != image_id: + raise SystemExit("OCI image manifest digest mismatch") + manifest = json.loads(raw) + if not isinstance(manifest, dict): + raise SystemExit("invalid OCI image manifest") + config = manifest.get("config") + if ( + manifest.get("schemaVersion") != 2 + or manifest.get("mediaType") not in { + "application/vnd.oci.image.manifest.v1+json", + "application/vnd.docker.distribution.manifest.v2+json", + } + or not isinstance(config, dict) + or config.get("digest") != config_id + ): + raise SystemExit("loaded HF image configuration does not match its manifest") + return + raise SystemExit("loaded HF image identity has no matching manifest in the verified archive") + + +def verify_loaded_image_identity(archive: Path, image_id: str, config_id: str) -> None: + # Classic Docker exposes the config digest directly. Containerd exposes an + # OCI target digest; verify its content-addressed manifest and config link. + if image_id == config_id: + return + try: + import zstandard + except ImportError as exc: + raise SystemExit("install runtime dependencies with `python -m pip install -e .`") from exc + with archive.open("rb") as source: + with zstandard.ZstdDecompressor().stream_reader(source) as stream: + verify_oci_image_identity(stream, image_id, config_id) + print("verified OCI manifest → published image config digest") + + def load_hf_image_archive( *, solve: Path, @@ -272,8 +337,7 @@ def load_hf_image_archive( if loaded_ref != archive_config["loaded_ref"]: raise SystemExit("HF image loaded_ref does not match release/images.json") image_id = ensure_image(loaded_ref) - if manifest.get("image_id") != image_id: - raise SystemExit("loaded HF image identity does not match its manifest") + verify_loaded_image_identity(archive, image_id, manifest.get("image_id")) return image_id @@ -284,14 +348,16 @@ def write_config( reference: Path, track: str, open_image: str, - revision: str, + solve_revision: str, + reference_revision: str, ) -> None: values = { "FRONTIER_SOLVE_DIR": str(solve), "FRONTIER_REFERENCE_DIR": str(reference), "FRONTIER_TRACK": track, "FRONTIER_OPEN_IMAGE": open_image, - "FRONTIER_DATASET_REVISION": revision, + "FRONTIER_SOLVE_REVISION": solve_revision, + "FRONTIER_REFERENCE_REVISION": reference_revision, } path.parent.mkdir(parents=True, exist_ok=True) path.write_text( @@ -303,11 +369,16 @@ def write_config( def main() -> int: image_manifest = json.loads((ROOT / "release" / "images.json").read_text()) + dataset_release = json.loads((ROOT / "release" / "datasets.json").read_text()) default_image = image_manifest["images"]["open"]["ref"] parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--solve-source", default=DEFAULT_SOLVE_REPO) - parser.add_argument("--reference-source", default=DEFAULT_REFERENCE_REPO) - parser.add_argument("--revision", default=DEFAULT_REVISION) + parser.add_argument("--solve-source", default=dataset_release["solve"]["repo"]) + parser.add_argument("--reference-source", default=dataset_release["reference"]["repo"]) + parser.add_argument( + "--revision", + default=None, + help="override both pinned dataset revisions (for example, main)", + ) parser.add_argument("--reference-revision", default=None) parser.add_argument("--track", choices=("open", "full"), default="open") parser.add_argument("--cache-dir", type=Path, default=Path.home() / ".cache/frontierchallenge") @@ -316,18 +387,21 @@ def main() -> int: ) parser.add_argument("--skip-image", action="store_true", help="verify datasets only") args = parser.parse_args() + solve_revision, reference_revision = select_revisions( + dataset_release, args.revision, args.reference_revision + ) token = os.environ.get("HF_TOKEN") solve = resolve_dataset( args.solve_source, - args.revision, + solve_revision, args.cache_dir, token=token, ignore_patterns=["images/*.tar.zst"], ) reference = resolve_dataset( args.reference_source, - args.reference_revision or args.revision, + reference_revision, args.cache_dir, token=token, ) @@ -340,7 +414,7 @@ def main() -> int: load_hf_image_archive( solve=solve, solve_source=args.solve_source, - revision=args.revision, + revision=solve_revision, cache_dir=args.cache_dir, archive_config=image_manifest["images"]["open"]["hf_archive"], token=token, @@ -357,7 +431,8 @@ def main() -> int: reference=reference, track=args.track, open_image=default_image, - revision=args.revision, + solve_revision=solve_revision, + reference_revision=reference_revision, ) print(f"ready: {len(selected)} tasks; configuration written to {args.config.resolve()}") return 0 diff --git a/benchmarks/frontierchallenge/scripts/summarize_results.py b/benchmarks/frontierchallenge/scripts/summarize_results.py old mode 100644 new mode 100755 index 24e8a84..e1221a2 --- a/benchmarks/frontierchallenge/scripts/summarize_results.py +++ b/benchmarks/frontierchallenge/scripts/summarize_results.py @@ -2,7 +2,7 @@ """Aggregate a Harbor job directory into a Pass Rate + Score summary. Reads each trial's ``config.json`` (to recover which task it ran) and -``verifier/reward.json`` (``task_score``, ``passed``, ``evaluation_complete``) +``verifier/reward.json`` (``task_score``, ``evaluation_complete``) under a Harbor jobs-dir job, and writes a per-task CSV plus an overall JSON summary next to it. @@ -14,12 +14,32 @@ import argparse import csv import json +import math from pathlib import Path from typing import Any #: 100 tasks minus task_065 (needs a GPU) and task_047 / task_049 (their #: deterministic grader only runs in a nested container, unavailable here). EXPECTED_TOTAL_TASKS = 97 +METRIC_DEFINITION = "score-gt-0.999" + + +def valid_score(score: Any) -> bool: + return ( + isinstance(score, (int, float)) + and not isinstance(score, bool) + and 0.0 <= score <= 1.0 + and math.isfinite(score) + ) + + +def official_pass(row: dict[str, Any]) -> bool: + """The sole pass decision: a completed valid score strictly above 0.999.""" + return ( + row.get("evaluation_complete") == 1 + and valid_score(row.get("task_score")) + and row["task_score"] > 0.999 + ) #: Judge stderr wording (both casings occur across the frozen graders) for the #: one "verifier failed" cause that is not an infrastructure fault at all: the @@ -38,7 +58,8 @@ def load_json(path: Path) -> dict[str, Any] | None: try: - return json.loads(path.read_text(encoding="utf-8")) + value = json.loads(path.read_text(encoding="utf-8")) + return value if isinstance(value, dict) else None except (OSError, json.JSONDecodeError): return None @@ -104,8 +125,11 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: reward = load_json(trial_dir / "verifier" / "reward.json") if reward is not None: row["task_score"] = reward.get("task_score") - row["passed"] = reward.get("passed") row["evaluation_complete"] = reward.get("evaluation_complete") + if not valid_score(row["task_score"]): + row["task_score"] = None + row["evaluation_complete"] = 0.0 + row["error"] = "invalid task_score: expected a finite number in [0, 1]" else: row["error"] = "no reward.json (trial errored or is still running)" if row["evaluation_complete"] != 1.0 and missing_required_artifact(trial_dir): @@ -114,35 +138,38 @@ def collect_rows(job_dir: Path) -> list[dict[str, Any]]: row["evaluation_complete"] = 1.0 row["scored_zero_missing_artifact"] = True row["error"] = "required artifact missing from submission - genuine 0" + row["passed"] = float(official_pass(row)) rows.append(row) return rows def summarize(rows: list[dict[str, Any]], expected_total: int) -> dict[str, Any]: - # A trial can report task_score=0.0/passed=0.0 alongside - # evaluation_complete=0.0 - that 0 is the verifier crashing (e.g. a Judge - # request the endpoint rejected, a missing-artifact hard error), not a - # real graded attempt. Averaging it in as if it were a genuine 0/100 - # silently deflates the score. Only evaluation_complete=1 rows are a - # trustworthy signal; everything else is reported separately so a run's - # real grading coverage is visible instead of hidden inside the average. - graded = [r for r in rows if r["evaluation_complete"] == 1.0 and r["task_score"] is not None] - ungraded = [ - r for r in rows - if r["task_score"] is not None and r["evaluation_complete"] != 1.0 - ] + if expected_total <= 0: + raise ValueError("expected_total must be positive") + ids = [r["task_id"] for r in rows] + if len(set(ids)) != len(ids): + raise ValueError("duplicate task trials: select one predeclared attempt per task; do not select by score") + if len(rows) > expected_total: + raise ValueError("more tasks than expected_total; check the evaluation scope") + # Incomplete/invalid evaluations contribute zero to the fixed denominator, + # but remain visible as failures rather than successful zero-score grading. + graded = [r for r in rows if r["evaluation_complete"] == 1.0 and valid_score(r["task_score"])] + ungraded = [r for r in rows if r["evaluation_complete"] == 0.0] n = len(rows) - mean_score = (sum(r["task_score"] for r in graded) / len(graded)) if graded else None - pass_rate = (sum(r["passed"] for r in graded) / len(graded)) if graded else None + mean_score = sum(r["task_score"] for r in graded) / expected_total + n_passed = sum(official_pass(r) for r in rows) + pass_rate = n_passed / expected_total return { + "metric_definition": METRIC_DEFINITION, + "pass_rule": "evaluation_complete == 1 and 0.999 < task_score <= 1.0", "n_tasks_expected": expected_total, "n_trials_found": n, "n_graded": len(graded), "n_zero_missing_artifact": sum(1 for r in graded if r["scored_zero_missing_artifact"]), "n_verifier_failed": len(ungraded), - "n_missing_or_errored": n - len(graded) - len(ungraded), + "n_missing_or_errored": expected_total - len(graded) - len(ungraded), "complete": n == expected_total and len(graded) == expected_total, - "n_passed": sum(1 for r in graded if r["passed"]) if graded else 0, + "n_passed": n_passed, "pass_rate": pass_rate, "mean_task_score": mean_score, "mean_task_score_100": (mean_score * 100) if mean_score is not None else None, @@ -191,17 +218,17 @@ def print_report(job_dir: Path, summary: dict[str, Any]) -> None: print( f"NOTE: {summary['n_verifier_failed']} trial(s) had the verifier itself " "fail (e.g. a Judge request the endpoint rejected, a missing-artifact " - "hard error) - excluded from Pass Rate / Score below, not counted as 0." + "hard error) - contribute 0 to the fixed-denominator metrics below." ) if summary["pass_rate"] is not None: print( - f"Pass Rate: {summary['n_passed']}/{summary['n_graded']} " + f"Pass Rate (task_score > 0.999): {summary['n_passed']}/{summary['n_tasks_expected']} " f"= {summary['pass_rate'] * 100:.1f}%" ) else: print("Pass Rate: n/a (no graded trials)") if summary["mean_task_score_100"] is not None: - print(f"Mean Score: {summary['mean_task_score_100']:.1f} / 100 (over {summary['n_graded']} graded trials)") + print(f"Mean Score: {summary['mean_task_score_100']:.1f} / 100 (denominator {summary['n_tasks_expected']})") else: print("Mean Score: n/a (no graded trials)") print(f"Wrote {job_dir / 'summary.csv'} and {job_dir / 'summary.json'}") @@ -210,6 +237,10 @@ def print_report(job_dir: Path, summary: dict[str, Any]) -> None: def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("job_dir", type=Path, help="Harbor jobs-dir job, e.g. results/harbor/") + parser.add_argument( + "--task-id", action="append", + help="only include this task ID (repeatable); excludes stale trials from a reused job", + ) parser.add_argument( "--expected-total", type=int, @@ -223,7 +254,13 @@ def main() -> None: raise SystemExit(f"not a directory: {job_dir}") rows = collect_rows(job_dir) - summary = summarize(rows, args.expected_total) + if args.task_id is not None: + selected = set(args.task_id) + rows = [row for row in rows if row["task_id"] in selected] + try: + summary = summarize(rows, args.expected_total) + except ValueError as exc: + parser.error(str(exc)) write_outputs(job_dir, rows, summary) print_report(job_dir, summary) diff --git a/benchmarks/frontierchallenge/scripts/task_selection.py b/benchmarks/frontierchallenge/scripts/task_selection.py new file mode 100755 index 0000000..aafa65c --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/task_selection.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Resolve the exact FrontierChallenge task set for an evaluation run.""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +from dataclasses import dataclass +from pathlib import Path + +VALID_ENVIRONMENTS = {"open", "licensed-orca"} + + +@dataclass(frozen=True) +class SelectedTask: + task_id: str + environment: str + path: Path + + +def select_tasks( + tasks_root: Path, + *, + track: str, + include: tuple[str, ...] = (), + exclude: tuple[str, ...] = (), + registry_path: Path | None = None, +) -> list[SelectedTask]: + """Select from verified solve tasks, never from persistent staging.""" + if track not in {"open", "full"}: + raise ValueError(f"unsupported track: {track}") + + registry_environments: dict[str, str] | None = None + if registry_path is not None: + registry = json.loads(registry_path.read_text(encoding="utf-8")) + registry_environments = {} + for record in registry.get("tasks", []): + task_id = record.get("id") + environment = record.get("image") + if not isinstance(task_id, str) or environment not in VALID_ENVIRONMENTS: + raise ValueError(f"invalid registry task record: {record!r}") + if task_id in registry_environments: + raise ValueError(f"duplicate registry task ID: {task_id}") + registry_environments[task_id] = environment + + selected: list[SelectedTask] = [] + for task_path in sorted(tasks_root.iterdir()): + if not task_path.is_dir() or not (task_path / "task.toml").is_file(): + continue + metadata_path = task_path / "task.json" + if not metadata_path.is_file(): + raise ValueError(f"task metadata missing: {metadata_path}") + metadata = json.loads(metadata_path.read_text(encoding="utf-8")) + task_id = metadata.get("task_id") + environment = metadata.get("environment") + if task_id != task_path.name: + raise ValueError(f"task ID/path mismatch: {task_path.name} != {task_id!r}") + if environment not in VALID_ENVIRONMENTS: + raise ValueError(f"invalid environment for {task_id}: {environment!r}") + if registry_environments is not None: + registered = registry_environments.get(task_id) + if registered != environment: + raise ValueError( + f"task/registry environment mismatch for {task_id}: " + f"{environment!r} != {registered!r}" + ) + if track == "open" and environment != "open": + continue + if include and not any(fnmatch.fnmatchcase(task_id, pattern) for pattern in include): + continue + if any(fnmatch.fnmatchcase(task_id, pattern) for pattern in exclude): + continue + selected.append(SelectedTask(task_id, environment, task_path.resolve())) + return selected + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tasks-root", type=Path, required=True) + parser.add_argument("--registry", type=Path, required=True) + parser.add_argument("--track", choices=("open", "full"), required=True) + parser.add_argument("--include", action="append", default=[]) + parser.add_argument("--exclude", action="append", default=[]) + args = parser.parse_args() + + try: + selected = select_tasks( + args.tasks_root, + track=args.track, + include=tuple(args.include), + exclude=tuple(args.exclude), + registry_path=args.registry, + ) + except (OSError, ValueError, json.JSONDecodeError) as exc: + parser.error(str(exc)) + if not selected: + parser.error("task selection is empty") + for task in selected: + path = str(task.path) + if any(character in path for character in ("\t", "\n", "\r")): + parser.error(f"task path contains a control character: {task.path}") + print(f"{task.task_id}\t{task.environment}\t{path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/frontierchallenge/scripts/verifier_log_policy.py b/benchmarks/frontierchallenge/scripts/verifier_log_policy.py new file mode 100644 index 0000000..7fb0e94 --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/verifier_log_policy.py @@ -0,0 +1,53 @@ +"""Publish grader diagnostics without retaining alternate pass decisions.""" +from __future__ import annotations + +import json +import re +from pathlib import Path + +PASS_KEYS = {"pass", "passed", "native_passed", "pass_threshold"} +DECISION_WORD = re.compile(r"\b(?:pass|passed|fail|native_passed|pass_threshold)\b", re.IGNORECASE) + + +def clean_json(value): + if isinstance(value, dict): + return {key: clean_json(item) for key, item in value.items() + if key.casefold() not in PASS_KEYS} + if isinstance(value, list): + return [clean_json(item) for item in value] + if isinstance(value, str) and value.strip().casefold() in {"pass", "passed", "fail"}: + return "[decision omitted]" + return value + + +def clean_text(text: str) -> str: + # Structured grader stdout retains scores and diagnostics, minus decisions. + try: + return json.dumps(clean_json(json.loads(text)), indent=2) + "\n" + except json.JSONDecodeError: + return "".join( + "[decision omitted]\n" if DECISION_WORD.search(line) else line + for line in text.splitlines(keepends=True) + ) + + +def publish_logs(private: Path, public: Path) -> None: + """Raw grader files never enter the Harbor log directory. + + reward.json already uses the sole benchmark pass rule and is preserved. + Other JSON/text files are diagnostic, not an additional scoring interface. + """ + public.mkdir(parents=True, exist_ok=True) + for source in sorted(private.iterdir()): + if not source.is_file() or source.is_symlink(): + raise ValueError(f"unexpected verifier log entry: {source.name}") + text = source.read_text(encoding="utf-8") + if source.name == "reward.json": + cleaned = text + elif source.suffix == ".json": + cleaned = json.dumps(clean_json(json.loads(text)), indent=2) + "\n" + elif source.suffix == ".txt": + cleaned = clean_text(text) + else: + raise ValueError(f"unsupported verifier log type: {source.name}") + (public / source.name).write_text(cleaned, encoding="utf-8") diff --git a/benchmarks/frontierchallenge/site/index.html b/benchmarks/frontierchallenge/site/index.html index 49789cc..10a1284 100644 --- a/benchmarks/frontierchallenge/site/index.html +++ b/benchmarks/frontierchallenge/site/index.html @@ -60,7 +60,7 @@

FrontierChallenge

-

13 model–scaffold configurations · 97 tasks · Pass Rate = native Score ≥ 99.9.

+

13 model–scaffold configurations · 97 tasks · Historical table: Pass Rate used native Score ≥ 99.9. The current evaluation policy requires completed task_score > 0.999 (strictly greater, without rounding); these historical numbers have not been recomputed.

diff --git a/benchmarks/frontierchallenge/tests/test_apply_score_policy.py b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py new file mode 100644 index 0000000..49ed79e --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_apply_score_policy.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +import pytest +from apply_score_policy import OLD_RULE, apply_policy + + +@pytest.mark.parametrize("score,native,complete,expected", [ + (100, False, True, 1), (80, True, True, 0), (99.999, True, True, 1), + # The frozen normalization emits 99.9 / 100 as 0.9990000000000001. + # Compare that actual task_score, not a rounded display of 0.999. + (99.9, True, True, 1), (99.91, False, True, 1), (99.899, True, True, 0), + (99.91, True, False, 0), (100.1, True, True, 0), (float("inf"), True, True, 0), + (100, True, False, 0), (0, True, True, 0), (100, None, True, 1), +]) +def test_staged_reward_uses_strict_score_threshold(tmp_path, score, native, complete, expected): + (tmp_path / "tests").mkdir() + adapter = tmp_path / "tests/run_frontier_verifier.py" + adapter.write_text('def main():\n global reward\n reward = {' + OLD_RULE + + '}\n\nif __name__ == "__main__":\n main()\n') + apply_policy(tmp_path) + namespace = {"score": score, "passed": native, "complete": complete, "__name__": "test_adapter"} + exec(compile(adapter.read_text(), str(adapter), "exec"), namespace) + namespace["_run_native_verifier"]() + assert namespace["reward"] == {"passed": float(expected)} + + +def test_unknown_adapter_fails_closed(tmp_path): + (tmp_path / "tests").mkdir() + adapter = tmp_path / "tests/run_frontier_verifier.py" + adapter.write_text("reward = {}\n") + with pytest.raises(ValueError, match="unsupported"): + apply_policy(tmp_path) + assert adapter.read_text() == "reward = {}\n" + + +@pytest.mark.parametrize("failure", [False, True]) +def test_adapter_publishes_clean_logs_on_success_and_failure(tmp_path, failure): + import json + adapter_dir = tmp_path / "task/tests" + adapter_dir.mkdir(parents=True) + adapter = adapter_dir / "run_frontier_verifier.py" + adapter.write_text('''from pathlib import Path +import json +def main(): + global private_path + private_path = LOGS + LOGS.mkdir(parents=True, exist_ok=True) + (LOGS / "native_grader_result.json").write_text(json.dumps({"score": 80, "passed": True})) + (LOGS / "native_grader.stdout.txt").write_text("PASSED=true\\nscore:80\\n") + if failure: + raise RuntimeError("fixture failure") + reward = {"task_score": 0.8, ''' + OLD_RULE + ''' "evaluation_complete": 1} + (LOGS / "reward.json").write_text(json.dumps(reward)) + +if __name__ == "__main__": + main() +''') + apply_policy(tmp_path / "task") + public = tmp_path / "public" + ns = {"__name__": "fixture", "LOGS": public, "failure": failure, + "score": 80, "passed": True, "complete": True} + exec(compile(adapter.read_text(), str(adapter), "exec"), ns) + if failure: + with pytest.raises(RuntimeError, match="fixture failure"): + ns["main"]() + else: + ns["main"]() + assert json.loads((public / "reward.json").read_text())["passed"] == 0 + assert json.loads((public / "native_grader_result.json").read_text()) == {"score": 80} + assert "PASSED" not in (public / "native_grader.stdout.txt").read_text() + assert ns["LOGS"] == public + assert not ns["private_path"].exists() diff --git a/benchmarks/frontierchallenge/tests/test_image_identity.py b/benchmarks/frontierchallenge/tests/test_image_identity.py new file mode 100644 index 0000000..f412208 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_image_identity.py @@ -0,0 +1,63 @@ +import hashlib +import io +import json +import tarfile + +import pytest +from setup_release import verify_loaded_image_identity, verify_oci_image_identity + +CONFIG = "sha256:" + "a" * 64 + + +def fixture(*, config=CONFIG, media_type="application/vnd.oci.image.manifest.v1+json", + corrupt=False, missing=False, symlink=False, oversized=False): + raw = json.dumps({"schemaVersion": 2, "mediaType": media_type, + "config": {"digest": config}}).encode() + identity = "sha256:" + hashlib.sha256(raw).hexdigest() + if corrupt: + raw += b" " + if oversized: + raw += b" " * (1024 * 1024) + identity = "sha256:" + hashlib.sha256(raw).hexdigest() + stream = io.BytesIO() + with tarfile.open(fileobj=stream, mode="w") as archive: + member = tarfile.TarInfo("other" if missing else "blobs/sha256/" + identity[7:]) + if symlink: + member.type = tarfile.SYMTYPE + member.linkname = "/should-not-be-read" + else: + member.size = len(raw) + archive.addfile(member, None if symlink else io.BytesIO(raw)) + stream.seek(0) + return stream, identity + + +def test_classic_identity_needs_no_decompression(tmp_path): + verify_loaded_image_identity(tmp_path / "not-opened", CONFIG, CONFIG) + + +@pytest.mark.parametrize("media_type", ["application/vnd.oci.image.manifest.v1+json", + "application/vnd.docker.distribution.manifest.v2+json"]) +def test_containerd_identity_is_bound_to_published_config(media_type): + stream, identity = fixture(media_type=media_type) + assert identity != CONFIG + verify_oci_image_identity(stream, identity, CONFIG) + + +@pytest.mark.parametrize("kwargs,expected", [ + ({"config": "sha256:" + "b" * 64}, "configuration does not match"), + ({"corrupt": True}, "digest mismatch"), + ({"missing": True}, "no matching manifest"), + ({"symlink": True}, "invalid OCI image manifest entry"), + ({"oversized": True}, "invalid OCI image manifest entry"), + ({"media_type": "application/vnd.oci.image.index.v1+json"}, "configuration does not match"), +]) +def test_identity_mismatch_still_fails_closed(kwargs, expected): + stream, identity = fixture(**kwargs) + with pytest.raises(SystemExit, match=expected): + verify_oci_image_identity(stream, identity, CONFIG) + + +def test_invalid_digest_rejected_before_reading(): + with pytest.raises(SystemExit, match="invalid image identity"): + verify_oci_image_identity(io.BytesIO(), "../../wrong", CONFIG) diff --git a/benchmarks/frontierchallenge/tests/test_job_policy.py b/benchmarks/frontierchallenge/tests/test_job_policy.py new file mode 100644 index 0000000..16cbf57 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_job_policy.py @@ -0,0 +1,45 @@ +import json + +import pytest +from job_policy import check_job, marker_path, record_policy + + +def existing(job): + job.mkdir(parents=True) + (job / "config.json").write_text("{}") + (job / "lock.json").write_text(json.dumps({"trials": [{"task": {"name": "task_one"}}]})) + + +def test_new_job_marker_does_not_create_harbor_job(tmp_path): + job = tmp_path / "new" + assert check_job(job, ["task_one"]) == "new" + record_policy(job, ["task_one"]) + assert not job.exists() + assert marker_path(job).is_file() + existing(job) + assert check_job(job, ["task_one"]) == "resume" + + +@pytest.mark.parametrize("case", ["missing", "old-metric", "old-logs", "different-tasks", "invalid-lock"]) +def test_unsafe_resume_does_not_mutate_results(tmp_path, case): + job = tmp_path / "old" + record_policy(job, ["task_one"]) + existing(job) + marker = marker_path(job) + doc = json.loads(marker.read_text()) + if case == "missing": + marker.unlink() + elif case == "old-metric": + doc["metric_definition"] = "exact-full-score" + marker.write_text(json.dumps(doc)) + elif case == "old-logs": + doc.pop("diagnostics") + marker.write_text(json.dumps(doc)) + elif case == "invalid-lock": + (job / "lock.json").write_text("broken") + tasks = ["other"] if case == "different-tasks" else ["task_one"] + before = {p.name: p.read_bytes() for p in job.iterdir()} + for operation in (check_job, record_policy): + with pytest.raises(ValueError, match="fresh --job-name"): + operation(job, tasks) + assert before == {p.name: p.read_bytes() for p in job.iterdir()} diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_contract.py b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py new file mode 100644 index 0000000..ed47971 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py @@ -0,0 +1,42 @@ +import tomllib +from pathlib import Path + +SCRIPT = Path(__file__).parents[1] / "scripts" / "run_eval.sh" + + +def test_runtime_python_floor_matches_pinned_harbor_and_docs(): + root = SCRIPT.parents[1] + project = tomllib.loads((root / "pyproject.toml").read_text())["project"] + assert "harbor==0.20.0" in project["dependencies"] + assert project["requires-python"] == ">=3.12" + for relative in ("README.md", "docs/quickstart.md"): + text = (root / relative).read_text() + assert "Python 3.12+" in text + assert "Python 3.11+" not in text + assert "python3.12 -m venv .venv" in text + + +def test_orca_preflight_uses_declared_environment_not_instruction_text(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'EFFECTIVE_TASK_ENVS[$index]' in text + assert "grep -qil 'orca'" not in text + + +def test_harbor_receives_exact_effective_task_ids(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'for task_id in "${EFFECTIVE_TASK_IDS[@]}"' in text + assert 'INCLUDE_ARGS+=("--include-task-name" "$task_id")' in text + + +def test_staging_dereferences_hugging_face_cache_symlinks(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'cp -aL "$task_dir" "$dest"' in text + + +def test_verifier_work_is_limited_to_effective_task_directories(): + text = SCRIPT.read_text(encoding="utf-8") + + assert text.count('for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"') == 2 diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_integration.py b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py new file mode 100644 index 0000000..4755c97 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_run_eval_integration.py @@ -0,0 +1,145 @@ +"""Exercise the shell runner with real archives and stubbed Docker/Harbor APIs. + +This verifies orchestration, not container isolation or real model execution. +""" +from __future__ import annotations + +import json +import os +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest +import reference_archive + +ROOT = Path(__file__).parents[1] +OPEN = "task_098_orca_claisen_thermochemistry" +LICENSED = "task_199_orca" + + +@pytest.fixture +def runtime(tmp_path): + runtime = tmp_path / "runtime" + shutil.copytree(ROOT / "scripts", runtime / "scripts") + solve, reference = tmp_path / "solve", tmp_path / "reference" + records = [] + for name, environment in [(OPEN, "open"), (LICENSED, "licensed-orca")]: + task = solve / "tasks" / name + (task / "environment/data").mkdir(parents=True) + (task / "task.toml").write_text('[agent]\ntimeout_sec = 600\n') + (task / "instruction.md").write_text("Analyze supplied ORCA output.") + (task / "task.json").write_text(json.dumps({ + "task_id": name, "environment": environment, "source_task_sha256": "fixture", + }, indent=2)) + (task / "environment/Dockerfile").write_text("FROM fixture/open\n") + blob = tmp_path / f"{name}.blob" + blob.write_text("precomputed output") + (task / "environment/data/output.txt").symlink_to(os.path.relpath(blob, task / "environment/data")) + verifier = reference / "tasks" / name + (verifier / "tests").mkdir(parents=True) + (verifier / "tests/test.sh").write_text("#!/bin/sh\nexit 0\n") + (verifier / "tests/run_frontier_verifier.py").write_text( + 'def main():\n reward = {"passed": 1.0 if passed is True else 0.0,}\n' + '\nif __name__ == "__main__":\n main()\n' + ) + reference_archive.pack(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"], + "frontier-challenge-reference", force=True) + reference_archive.strip(verifier, reference_archive.ARCHIVE_BY_KIND["verifier"]) + records.append({"id": name, "image": environment}) + registry = {"name": "fixture", "n_tasks": 2, "tasks": records} + for root, filename in [(runtime, "registry.json"), (solve, "source_registry.json"), + (reference, "source_registry.json")]: + (root / filename).write_text(json.dumps(registry)) + envfile = tmp_path / "credentials.env" + envfile.write_text("ANTHROPIC_API_KEY=fixture-not-a-key\n") + binaries = tmp_path / "bin" + binaries.mkdir() + (binaries / "python3").symlink_to(sys.executable) + docker = binaries / "docker" + docker.write_text('#!/bin/sh\nprintf "%s\\n" "$*" >> "$DOCKER_LOG"\n' + 'case "$*" in *orca-user-local*) exit 1;; esac\nexit 0\n') + docker.chmod(0o755) + harbor = binaries / "harbor" + harbor.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "$HARBOR_LOG"\n') + harbor.chmod(0o755) + env = {**os.environ, "PATH": f"{binaries}:{os.environ['PATH']}", + "DOCKER_LOG": str(tmp_path / "docker.log"), "HARBOR_LOG": str(tmp_path / "harbor.log"), + "FRONTIER_CONFIG_FILE": str(tmp_path / "no-config")} + stage = tmp_path / "stage" + cmd = ["bash", str(runtime / "scripts/run_eval.sh"), "--agent", "claude-code", + "--model", "fixture", "--solve-dir", str(solve), "--reference-dir", str(reference), + "--stage-dir", str(stage), "--env-file", str(envfile), "--no-judge-override", "--no-summary", + "--jobs-dir", str(tmp_path / "jobs"), "--job-name", "fixture"] + return tmp_path, solve, stage, cmd, env + + +def test_open_selection_migrates_legacy_cache_and_ignores_stale_orca(runtime): + root, solve, stage, cmd, env = runtime + stale = stage / OPEN + shutil.copytree(solve / "tasks" / OPEN, stale, symlinks=True) + # Exactly the identity the old runner wrote, but nested input symlink now broken. + (stale / ".frontier-source").write_text( + f"{solve}|source_task_sha256:fixture|frontierchallenge/cpu-open:2026.08\n") + assert (stale / "environment/data/output.txt").is_symlink() + assert not (stale / "environment/data/output.txt").exists() + shutil.copytree(solve / "tasks" / LICENSED, stage / LICENSED, symlinks=True) + result = subprocess.run(cmd + ["--track", "open"], env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert "Staged 1 task(s), reused 0" in result.stdout + assert (stale / "environment/data/output.txt").read_text() == "precomputed output" + assert not (stale / "environment/data/output.txt").is_symlink() + assert (stale / "tests/test.sh").is_file() + adapter = (stale / "tests/run_frontier_verifier.py").read_text() + assert '"passed": 1.0 if complete and 0.999 < float(score or 0.0) / 100.0 <= 1.0 else 0.0' in adapter + assert "orca-user-local" not in (root / "docker.log").read_text() + args = (root / "harbor.log").read_text().splitlines() + assert args[args.index("--include-task-name") + 1] == OPEN + assert LICENSED not in args + result = subprocess.run(cmd, env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert "Staged 0 task(s), reused 1" in result.stdout + + +def test_full_track_exclusion_prevents_orca_preflight(runtime): + root, _, _, cmd, env = runtime + result = subprocess.run(cmd + ["--track", "full", "--exclude-task-name", LICENSED], + env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert "orca-user-local" not in (root / "docker.log").read_text() + + +def test_selected_licensed_task_still_requires_orca(runtime): + root, _, _, cmd, env = runtime + result = subprocess.run(cmd + ["--track", "full", "--include-task-name", LICENSED], + env=env, capture_output=True, text=True, check=False) + assert result.returncode != 0 + assert "1 task(s) require ORCA" in result.stderr + assert not (root / "harbor.log").exists() + + +def test_legacy_job_is_refused_before_staging(runtime): + root, _, stage, cmd, env = runtime + job = root / "jobs/fixture" + job.mkdir(parents=True) + (job / "config.json").write_text("{}") + result = subprocess.run(cmd, env=env, capture_output=True, text=True, check=False) + assert result.returncode != 0 + assert "Use a fresh --job-name" in result.stderr + assert not (root / "harbor.log").exists() + assert not (stage / OPEN).exists() + assert (job / "config.json").read_text() == "{}" + + +def test_current_policy_job_resumes(runtime): + import job_policy + root, _, _, cmd, env = runtime + job = root / "jobs/fixture" + job_policy.record_policy(job, [OPEN]) + job.mkdir(parents=True) + (job / "config.json").write_text("{}") + (job / "lock.json").write_text(json.dumps({"trials": [{"task": {"name": OPEN}}]})) + result = subprocess.run(cmd, env=env, capture_output=True, text=True, check=False) + assert result.returncode == 0, result.stdout + result.stderr + assert (root / "harbor.log").read_text().splitlines()[:2] == ["job", "resume"] diff --git a/benchmarks/frontierchallenge/tests/test_setup_release.py b/benchmarks/frontierchallenge/tests/test_setup_release.py index 38bd35f..b1b2391 100644 --- a/benchmarks/frontierchallenge/tests/test_setup_release.py +++ b/benchmarks/frontierchallenge/tests/test_setup_release.py @@ -1,12 +1,11 @@ from __future__ import annotations import json -from pathlib import Path import sys +from pathlib import Path from types import SimpleNamespace import pytest - import setup_release @@ -91,16 +90,44 @@ def test_write_config_quotes_evaluator_paths(tmp_path): reference=tmp_path / "reference package", track="open", open_image="example/image@sha256:123", - revision="main", + solve_revision="solve-sha", + reference_revision="reference-sha", ) text = config.read_text() assert "FRONTIER_SOLVE_DIR=" in text assert "'" in text assert "example/image@sha256:123" in text + assert "FRONTIER_SOLVE_REVISION=solve-sha" in text + assert "FRONTIER_REFERENCE_REVISION=reference-sha" in text assert "FRONTIER_IMAGE_SOURCE" not in text +def test_select_revisions_uses_independent_pins_by_default(): + release = { + "solve": {"revision": "solve-sha"}, + "reference": {"revision": "reference-sha"}, + } + + assert setup_release.select_revisions(release, None, None) == ( + "solve-sha", + "reference-sha", + ) + + +def test_revision_override_preserves_legacy_both_dataset_behavior(): + release = { + "solve": {"revision": "solve-sha"}, + "reference": {"revision": "reference-sha"}, + } + + assert setup_release.select_revisions(release, "main", None) == ("main", "main") + assert setup_release.select_revisions(release, "solve-next", "ref-next") == ( + "solve-next", + "ref-next", + ) + + def test_validate_orca_runtime_accepts_wrapper_contract(monkeypatch): calls = [] diff --git a/benchmarks/frontierchallenge/tests/test_summarize_results.py b/benchmarks/frontierchallenge/tests/test_summarize_results.py new file mode 100644 index 0000000..e29d91e --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_summarize_results.py @@ -0,0 +1,125 @@ +from __future__ import annotations + +import csv +import json +import subprocess +import sys +from pathlib import Path + +import pytest +import summarize_results as metrics + + +def trial(root, name, score, native_passed=1, complete=1): + directory = root / name + (directory / "verifier").mkdir(parents=True) + (directory / "config.json").write_text(json.dumps({"task": {"path": f"/tasks/{name}"}})) + (directory / "verifier/reward.json").write_text(json.dumps({ + "task_score": score, "passed": native_passed, "evaluation_complete": complete, + })) + return directory + + +@pytest.mark.parametrize("score,native,complete,passed", [ + (1.0, 0, 1, 1), (1, 1, 1, 1), (0.999999999, 1, 1, 1), + (0.9991, 0, 1, 1), (0.998999999, 1, 1, 0), (0.9991, 1, 0, 0), + (0.999, 1, 1, 0), (0.8, 1, 1, 0), (0.0, 1, 1, 0), + (1.0, 1, 0, 0), (1.0, 1, None, 0), +]) +def test_official_pass_ignores_native_threshold(tmp_path, score, native, complete, passed): + directory = trial(tmp_path, "task_one", score, native, complete) + before = (directory / "verifier/reward.json").read_bytes() + row = metrics.collect_rows(tmp_path)[0] + assert row["passed"] == passed + assert "native_passed" not in row + assert (directory / "verifier/reward.json").read_bytes() == before + assert metrics.summarize([row], 97)["pass_rate"] == passed / 97 + + +@pytest.mark.parametrize("score", [None, "1", True, -0.1, 1.001, 10**400, float("inf"), float("nan")]) +def test_invalid_scores_are_not_passes_or_partial_credit(tmp_path, score): + trial(tmp_path, "task_bad", score) + rows = metrics.collect_rows(tmp_path) + assert rows[0]["task_score"] is None + assert rows[0]["evaluation_complete"] == 0 + summary = metrics.summarize(rows, 97) + assert summary["n_passed"] == summary["mean_task_score"] == 0 + assert summary["n_verifier_failed"] == 1 + + +def test_fixed_denominator_and_partial_credit(tmp_path): + trial(tmp_path, "task_one", 1, 0) + trial(tmp_path, "task_two", 0.7, 1) + trial(tmp_path, "task_failed", 1, 1, 0) + summary = metrics.summarize(metrics.collect_rows(tmp_path), 97) + assert summary["n_passed"] == 1 + assert summary["pass_rate"] == 1 / 97 + assert summary["mean_task_score_100"] == pytest.approx(170 / 97) + assert summary["n_missing_or_errored"] == 94 + assert summary["complete"] is False + assert summary["metric_definition"] == "score-gt-0.999" + assert summary["pass_rule"] == "evaluation_complete == 1 and 0.999 < task_score <= 1.0" + + +def test_empty_run_is_zero_and_incomplete(): + summary = metrics.summarize([], 97) + assert summary["n_missing_or_errored"] == 97 + assert summary["pass_rate"] == summary["mean_task_score"] == 0 + assert not summary["complete"] + + +def test_repeated_trials_and_wrong_denominator_are_rejected(tmp_path): + trial(tmp_path, "task_one", 1) + rows = metrics.collect_rows(tmp_path) + with pytest.raises(ValueError, match="duplicate"): + metrics.summarize(rows * 2, 97) + with pytest.raises(ValueError, match="positive"): + metrics.summarize(rows, 0) + trial(tmp_path, "task_two", 0) + with pytest.raises(ValueError, match="more tasks"): + metrics.summarize(metrics.collect_rows(tmp_path), 1) + + +def test_missing_submission_still_counts_as_zero(tmp_path): + directory = trial(tmp_path, "task_one", 0, 0, 0) + (directory / "verifier/native_grader.stderr.txt").write_text("submission directory not found") + summary = metrics.summarize(metrics.collect_rows(tmp_path), 1) + assert summary["complete"] is True + assert summary["n_zero_missing_artifact"] == 1 + assert summary["pass_rate"] == 0 + + +def test_cli_outputs_only_one_pass_flag(tmp_path): + trial(tmp_path, "task_one", 0.9, 1) + trial(tmp_path, "task_two", 1, 0) + subprocess.run([sys.executable, str(Path(metrics.__file__)), str(tmp_path), + "--expected-total", "2"], check=True, capture_output=True) + summary = json.loads((tmp_path / "summary.json").read_text()) + assert summary["pass_rate"] == 0.5 + assert summary["mean_task_score_100"] == 95 + assert summary["complete"] is True + with (tmp_path / "summary.csv").open() as handle: + rows = list(csv.DictReader(handle)) + assert [r["passed"] for r in rows] == ["0.0", "1.0"] + assert all("native_passed" not in r for r in rows) + assert "native_passed" not in (tmp_path / "summary.json").read_text() + + +def test_cli_excludes_stale_trials_from_reused_job(tmp_path): + trial(tmp_path, "task_selected", 0.5) + trial(tmp_path, "task_stale", 1) + subprocess.run([sys.executable, str(Path(metrics.__file__)), str(tmp_path), + "--task-id", "task_selected"], check=True, capture_output=True) + summary = json.loads((tmp_path / "summary.json").read_text()) + assert summary["n_trials_found"] == 1 + assert summary["n_passed"] == 0 + assert summary["mean_task_score"] == 0.5 / 97 + + +@pytest.mark.parametrize("value", [[], 1, "broken", None]) +def test_non_object_reward_is_reported_as_missing(tmp_path, value): + directory = trial(tmp_path, "task_bad", 1) + (directory / "verifier/reward.json").write_text(json.dumps(value)) + rows = metrics.collect_rows(tmp_path) + assert rows[0]["task_score"] is None + assert metrics.summarize(rows, 97)["n_passed"] == 0 diff --git a/benchmarks/frontierchallenge/tests/test_task_selection.py b/benchmarks/frontierchallenge/tests/test_task_selection.py new file mode 100644 index 0000000..6f089b0 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_task_selection.py @@ -0,0 +1,103 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import task_selection + + +def make_task(root: Path, task_id: str, environment: str, instruction: str = "") -> Path: + task = root / task_id + (task / "environment").mkdir(parents=True) + (task / "task.toml").write_text('schema_version = "1.1"\n') + (task / "task.json").write_text( + json.dumps({"task_id": task_id, "environment": environment}) + ) + (task / "instruction.md").write_text(instruction) + (task / "environment" / "Dockerfile").write_text("FROM example/open\n") + return task + + +def test_open_task_can_mention_orca_without_requiring_licensed_runtime(tmp_path): + make_task( + tmp_path, + "task_098_orca_claisen_thermochemistry", + "open", + "Read the supplied ORCA output files; do not execute ORCA.", + ) + + selected = task_selection.select_tasks(tmp_path, track="open") + + assert [(task.task_id, task.environment) for task in selected] == [ + ("task_098_orca_claisen_thermochemistry", "open") + ] + + +def test_open_track_excludes_declared_licensed_tasks(tmp_path): + make_task(tmp_path, "task_011_open", "open") + make_task(tmp_path, "task_199_orca", "licensed-orca") + + assert [task.task_id for task in task_selection.select_tasks(tmp_path, track="open")] == [ + "task_011_open" + ] + assert [task.task_id for task in task_selection.select_tasks(tmp_path, track="full")] == [ + "task_011_open", + "task_199_orca", + ] + + +def test_include_and_exclude_use_glob_semantics(tmp_path): + for task_id in ("task_011_alpha", "task_012_beta", "task_199_orca"): + make_task(tmp_path, task_id, "open") + + selected = task_selection.select_tasks( + tmp_path, + track="open", + include=("task_0*",), + exclude=("*_beta",), + ) + + assert [task.task_id for task in selected] == ["task_011_alpha"] + + +def test_selection_reads_solve_source_not_stale_stage(tmp_path): + solve = tmp_path / "solve" + stale_stage = tmp_path / "stage" + make_task(solve, "task_011_open", "open") + make_task(stale_stage, "task_199_orca", "licensed-orca") + + selected = task_selection.select_tasks(solve, track="open") + + assert [task.task_id for task in selected] == ["task_011_open"] + + +def test_invalid_environment_is_rejected(tmp_path): + make_task(tmp_path, "task_011_bad", "orca-by-text-search") + + with pytest.raises(ValueError, match="invalid environment"): + task_selection.select_tasks(tmp_path, track="full") + + +def test_task_environment_must_match_registry_commitment(tmp_path): + make_task(tmp_path, "task_098_orca_claisen_thermochemistry", "open") + registry = tmp_path / "source_registry.json" + registry.write_text( + json.dumps( + { + "tasks": [ + { + "id": "task_098_orca_claisen_thermochemistry", + "image": "licensed-orca", + } + ] + } + ) + ) + + with pytest.raises(ValueError, match="task/registry environment mismatch"): + task_selection.select_tasks( + tmp_path, + track="full", + registry_path=registry, + ) diff --git a/benchmarks/frontierchallenge/tests/test_verifier_log_policy.py b/benchmarks/frontierchallenge/tests/test_verifier_log_policy.py new file mode 100644 index 0000000..b8ded81 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_verifier_log_policy.py @@ -0,0 +1,30 @@ +import json + +from verifier_log_policy import clean_json, clean_text, publish_logs + + +def test_nested_decisions_are_removed_but_scores_remain(): + source = {"score": 80, "passed": True, "detail": [{"PASSED": True, "score": 5}], + "evaluation_complete": True, "pass_threshold": 70, "verdict": "PASS"} + assert clean_json(source) == {"score": 80, "detail": [{"score": 5}], + "evaluation_complete": True, "verdict": "[decision omitted]"} + assert source["passed"] is True # The adapter can still parse its in-memory payload. + + +def test_text_decisions_are_removed_without_losing_ordinary_errors(): + text = 'score: 80\nPASSED = true\nscore 80 -> PASS\nfailed to open input.csv\n' + assert clean_text(text) == 'score: 80\n[decision omitted]\n[decision omitted]\nfailed to open input.csv\n' + assert json.loads(clean_text('{"score": 80, "passed": true}')) == {"score": 80} + + +def test_only_reward_keeps_canonical_pass(tmp_path): + private, public = tmp_path / "private", tmp_path / "public" + private.mkdir() + (private / "native_grader_result.json").write_text('{"score":80,"passed":true}') + (private / "native_grader.stdout.txt").write_text('PASSED=true\nscore: 80\n') + reward = '{"task_score":0.8,"passed":0,"evaluation_complete":1}' + (private / "reward.json").write_text(reward) + publish_logs(private, public) + assert (public / "reward.json").read_text() == reward + assert json.loads((public / "native_grader_result.json").read_text()) == {"score": 80} + assert "PASSED" not in (public / "native_grader.stdout.txt").read_text()