fix(frontierchallenge): unify scoring and repair task-cache upgrades

benchmarks/frontierchallenge/README.md

@@ -29,7 +29,7 @@ simulation, electrochemistry, quantitative imaging, and molecular biology.
   <tbody>
     <tr><td>Tasks</td><td>97 (74 hard, 23 medium)</td></tr>
     <tr><td>Taxonomy</td><td>6 domains, 21 subdomains</td></tr>
-    <tr><td>Runtime</td><td>81 open-image tasks, 16 user-supplied ORCA tasks</td></tr>
+    <tr><td>Runtime</td><td>81 open-image tasks, 16 tasks executing user-supplied ORCA</td></tr>
     <tr><td>Grading</td><td>deterministic checks; 77 tasks also judge the report</td></tr>
     <tr><td>Harness</td><td>Harbor 0.20.0</td></tr>
     <tr><td>Output</td><td>named files under <code>/app/output</code></td></tr>
@@ -38,13 +38,15 @@ simulation, electrochemistry, quantitative imaging, and molecular biology.
 
 ## End-to-end workflow
 
-Requirements: Linux x86-64, Python 3.11+, Docker with Compose, model
+Requirements: Linux x86-64, Python 3.12+ (Harbor 0.20.0), Docker with Compose, model
 and judge credentials, and a Hugging Face token while either dataset is private
 or gated.
 
 ```bash
 git clone https://github.com/ApodexAI/FrontierAgent.git
 cd FrontierAgent/benchmarks/frontierchallenge
+python3.12 -m venv .venv
+source .venv/bin/activate
 python -m pip install -e .
 cp .env.example .env
 ```
@@ -59,10 +61,10 @@ its SHA-256, and load it into Docker:
 HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
-The full track adds 16 normally released ORCA tasks. FrontierChallenge does
-not distribute ORCA or an image containing it. After obtaining ORCA 6.0.1 from
-its official provider, build and smoke-test the private local runtime, then
-validate the full track:
+The full track adds 16 normally released tasks that execute ORCA.
+FrontierChallenge does not distribute ORCA or an image containing it. After
+obtaining ORCA 6.0.1 from its official provider, build and smoke-test the
+private local runtime, then validate the full track:
 
 ```bash
 ./scripts/build_orca_runtime.sh \
@@ -74,6 +76,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track full
 Do not push, export, publish, or share the resulting ORCA image. See the
 [ORCA setup tutorial](docs/providers/orca.md).
 
+Track membership describes what a task executes, not where its input files
+came from. For example, `task_098_orca_claisen_thermochemistry` reads supplied
+ORCA output but does not run ORCA, so it belongs to the open track.
+
 ### 2. Run a real task
 
 Fill `.env`, then run Harbor with the Claude Code agent:
@@ -101,8 +107,12 @@ cat results/harbor/<job>/<trial>/verifier/reward.json
 cat results/harbor/<job>/summary.json
 ```
 
-`passed` is the task's own pass decision; do not derive it from a global score
-threshold. `task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms
+Official **Pass Rate** counts completed evaluations with **`task_score > 0.999`**
+over all 97 tasks. **Score** is the mean `task_score` over 97, multiplied by 100.
+Missing or failed evaluations contribute zero. `passed` has this single meaning
+in both `reward.json` and summaries; no alternate pass field is emitted.
+The comparison is strict and uses unrounded scores: exactly `0.999` does not pass.
+`task_score` is in `[0, 1]`, and `evaluation_complete = 1` confirms
 that grading finished. See [Quickstart](docs/quickstart.md) for credentials and
 expected output, and [Scoring](docs/scoring.md) for aggregate reporting.
 

benchmarks/frontierchallenge/TASKS.md

@@ -14,7 +14,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
 The table below previews benchmark coverage without exposing evaluator data.
-Keywords come from each task and describe technique rather than answers.
+Keywords come from each task and describe technique rather than answers. The
+`Image` column records what the task executes, not software named in supplied
+files: `task_098_orca_claisen_thermochemistry`, for example, reads precomputed
+ORCA output and therefore uses the open image.
 
 | Task | Difficulty | Image | Judge | Agent budget | Techniques |
 |---|---|---|---|---|---|

benchmarks/frontierchallenge/docs/huggingface-release.md

@@ -19,9 +19,12 @@ workspace.
 HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
-Setup downloads the current `main` branches, runs the verification tool bundled
-with each dataset, and requires both `source_registry.json` files to equal this
-checkout's `registry.json`. A mixed or incomplete dataset is refused.
+Setup downloads the exact solve and reference commits declared in
+`release/datasets.json`, runs the verification tool bundled with each dataset,
+and requires both `source_registry.json` files to equal this checkout's
+`registry.json`. A mixed or incomplete dataset is refused. Release maintainers
+may test newer snapshots with `--revision` and `--reference-revision`; published
+runtime changes should update both pins together.
 
 Use local directories instead of HF repository IDs for an offline handoff:
 
@@ -44,3 +47,14 @@ the encrypted verifier hash. GitHub contains neither payload. The solve dataset
 must contain no `tests/`, verifier archive, rubric, fixture, or reference
 output; the reference dataset must contain no instruction, input, or runtime
 environment.
+
+The top-level Hugging Face `README.md` is intentionally outside
+`checksums.sha256`: it is a mutable dataset card whose citation and links may be
+edited without changing the benchmark payload. Task files, task-level READMEs,
+registries, manifests, image artifacts, and verifier archives remain covered by
+the checksum manifests and registry commitments.
+
+After payload edits, regenerate the affected checksum entries and run both
+bundled verification tools before publishing. Dataset-card-only edits require
+no payload checksum change. Update the runtime's pinned HF revisions after
+publishing; see [Scoring](scoring.md) for the metric contract shared by both cards.

benchmarks/frontierchallenge/docs/providers/docker.md

@@ -92,7 +92,7 @@ path.
 ## Two things that bite
 
 **ORCA is user-supplied and writes beside its input.** Before selecting one of
-the 16 ORCA tasks, create the licensed local runtime described in
+the 16 tasks that execute ORCA, create the licensed local runtime described in
 [orca.md](orca.md). Copy ORCA inputs into a writable directory (`/app/data`,
 `/tmp`) before running; invoking ORCA directly on a read-only bind-mounted file
 fails.

benchmarks/frontierchallenge/docs/quickstart.md

@@ -6,7 +6,9 @@ Hugging Face datasets, a real Harbor + Claude Code run, and the final score.
 ## Requirements
 
 - Linux x86-64 with Docker and Compose v2;
-- Python 3.11+ and about 20 GB for the open image;
+- Python 3.12+ on the evaluator host (required by Harbor 0.20.0);
+  allow at least 40 GB of free disk for the downloaded archive,
+  Docker image, and working data (more for concurrent runs and results);
 - a model API key and a judge API key;
 - `HF_TOKEN` while either dataset is private or gated;
 - for the full track only, an official ORCA 6.0.1 download and permission to
@@ -24,6 +26,8 @@ docker compose version
 ```bash
 git clone https://github.com/ApodexAI/FrontierAgent.git
 cd FrontierAgent/benchmarks/frontierchallenge
+python3.12 -m venv .venv
+source .venv/bin/activate
 python -m pip install -e .
 cp .env.example .env
 ```
@@ -61,19 +65,32 @@ This is the shortest path for most evaluators:
 HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
-Setup downloads the solve and reference datasets from their current `main` branches,
-verifies both packages, binds them to this checkout's `registry.json`, then
+Setup downloads the solve and reference revisions pinned by this Git checkout in
+`release/datasets.json`, verifies both packages, binds them to this checkout's
+`registry.json`, then
 downloads `images/frontierchallenge-cpu-open-2026.08.docker.tar.zst` from the
-solve dataset. It checks the declared size, SHA-256 and image ID before loading
-the `linux/amd64` image into Docker. No container registry is used. Evaluator-
+solve dataset. It checks the archive's declared size and SHA-256 before loading
+the `linux/amd64` image into Docker, then verifies the loaded image identity.
+No container registry is used. Evaluator-
 local paths are written to `.frontierchallenge/config.env`.
 
+Docker's classic and containerd image stores expose different image IDs. Setup
+accepts the published config digest directly, or verifies that the loaded OCI
+manifest digest links to that exact config inside the SHA-256-verified archive.
+Keep runtime dependencies current with `python -m pip install -e .`; do not
+disable identity checks or change Docker's storage backend to work around this.
+
+For release development only, `--revision main` overrides both pins;
+`--reference-revision` can override the reference revision independently. Normal
+evaluation should keep the checkout's pins so later dataset changes cannot alter
+an otherwise identical run.
+
 ### Full track: build the private ORCA runtime
 
-All 16 ORCA task statements and inputs are released normally. Only ORCA and a
-configured ORCA image are absent. Obtain ORCA 6.0.1 from its official provider,
-install it outside this checkout, and keep the complete directory together.
-Then run:
+All statements and inputs for the 16 tasks that execute ORCA are released
+normally. Only ORCA and a configured ORCA image are absent. Obtain ORCA 6.0.1
+from its official provider, install it outside this checkout, and keep the
+complete directory together. Then run:
 
 ```bash
 ./scripts/build_orca_runtime.sh \
@@ -118,6 +135,11 @@ tasks into evaluator staging, decrypts the matching verifier there, starts the
 agent, and invokes Harbor's verifier after the agent exits. By default Claude
 Code's `WebSearch` and `WebFetch` tools are disabled.
 
+Selection comes from each task's declared `task.json.environment`, validated
+against the registry. Include/exclude filters are applied before image preflight,
+staging, verifier decryption, resume, and Harbor invocation; stale directories
+from an older run cannot add tasks to the effective run.
+
 A healthy run reaches messages like:
 
 ```text
@@ -146,10 +168,14 @@ cat results/harbor/<job>/<trial>/verifier/reward.json
 ```
 
 - `evaluation_complete = 1` means the verifier finished;
-- `passed` is the task's own pass decision and must not be recomputed from a
-  global threshold;
+- official Pass Rate counts completed `task_score > 0.999` evaluations over 97;
+- `passed` uses this same strict threshold in both rewards and summaries;
 - `task_score` is a continuous score in `[0, 1]`.
 
+Score is the mean `task_score` over 97, times 100. Missing and failed evaluations
+contribute zero. Use unrounded scores: exactly `0.999` does not pass. See
+[Scoring](scoring.md) for subsets, repeated attempts, and historical results.
+
 The job aggregate is:
 
 ```bash

benchmarks/frontierchallenge/docs/running.md

@@ -12,7 +12,8 @@ The open track contains the 81 tasks that use the redistributable image:
 ./scripts/run_eval.sh --agent claude-code --model <model>
 ```
 
-The full track adds 16 ORCA tasks. Prepare the licensed local runtime first:
+The full track adds 16 tasks that execute ORCA. Prepare the licensed local
+runtime first:
 
 ```bash
 ./scripts/build_orca_runtime.sh --orca-root /path/to/orca-6.0.1
@@ -22,6 +23,10 @@ The full track adds 16 ORCA tasks. Prepare the licensed local runtime first:
 
 Setup writes verified local paths under `.frontierchallenge/`. The runner
 validates the GitHub/solve/reference registries again before staging anything.
+Track membership follows each task's declared execution environment, not
+software names in its instruction or supplied files. Thus
+`task_098_orca_claisen_thermochemistry`, which only reads precomputed ORCA
+output, remains an open-track task.
 
 ## Runtime
 
@@ -44,7 +49,9 @@ missing. See [Docker](providers/docker.md).
 ```
 
 Use disjoint include lists and distinct job names to shard across machines.
-`summarize_results.py` accepts multiple job directories and merges them.
+`summarize_results.py` accepts one job directory at a time; it does not merge
+shards. A shard's report is partial, not a separate full-benchmark result.
+Do not average shard Pass Rates as though they were full runs.
 
 Concurrency must fit both machine resources and model-provider rate limits.
 Start with one task, then increase gradually.
@@ -62,21 +69,41 @@ lowering longer ones:
 Use a separate stage directory for each concurrent run that changes timeouts.
 Verifier timeouts use `--verifier-timeout-multiplier` (default 40).
 
+Legacy staging caches containing Hugging Face symlinks are automatically
+rebuilt once. Only the effective include/exclude selection is staged, checked
+for ORCA, unsealed, and passed to Harbor; leftover stage directories are ignored.
+
 ## Resume and results
 
-Reusing a job name resumes completed work when the requested task set matches:
+Reusing a job name resumes completed work only when the task selection and
+scoring/log policy match the recorded job:
 
 ```bash
 ./scripts/run_eval.sh --agent claude-code --model <model> --job-name <same-name>
 ```
 
+The runner records its policy in
+`<jobs-dir>/.frontierchallenge-policies/<job-name>.json`. Keep that sidecar
+alongside jobs when moving them. A missing/old policy marker, changed task
+selection, or invalid job metadata is refused before staging. Use a fresh
+`--job-name` or `--jobs-dir`; old rewards and logs are not silently rewritten
+or mixed into a new-policy job. Existing results can still be summarized
+separately from their scores, but their original rewards remain historical.
+
 Results are written under `results/harbor/<job>/`. Read the aggregate with:
 
 ```bash
 python3 scripts/summarize_results.py results/harbor/<job>
 cat results/harbor/<job>/summary.json
 ```
 
+Official Pass Rate requires completed `task_score > 0.999`; `passed` means the
+same thing in rewards and summaries. The default denominator is 97, including missing tasks.
+See [Scoring](scoring.md) for partial scores and explicitly labeled subsets.
+Automatic summaries use the runner's current selection. If a job directory
+contains trials from an older selection, use repeatable `--task-id <id>` with
+the standalone summarizer, or start a fresh job directory.
+
 Before a long run, verify one task reaches `evaluation_complete = 1`, confirm
 the selected backend in the startup banner, and confirm the local ORCA runtime
 before selecting the full track.

benchmarks/frontierchallenge/docs/scoring.md

@@ -2,32 +2,72 @@
 
 FrontierChallenge reports two numbers over a fixed denominator of 97 tasks:
 
-- **Pass Rate:** tasks whose verifier writes `passed = 1`, divided by 97.
+- **Pass Rate:** completed evaluations with `task_score > 0.999`, divided by 97.
 - **Score:** the mean of `task_score` across all 97, usually reported times 100.
 
 Unrun tasks and harness failures count as zero in the fixed denominator. The
-summarizer marks an incomplete run as partial instead of averaging only the
-tasks that happened to finish.
+summarizer marks an incomplete run as partial while retaining the denominator
+97. It never drops missing or failed tasks from the headline metrics.
+
+The comparison is strict: `0.999` does not pass; `0.9991` and `1.0` pass.
+Use unrounded scores without an additional epsilon or per-task threshold.
+Compare the stored `task_score`, not a rounded display: for example,
+`0.9990000000000001` passes even if displayed as `0.999`. Score normalization
+and partial-credit arithmetic are unchanged.
+`evaluation_complete == 1` is required for a valid score. Invalid scores
+(non-numeric, non-finite, or outside `[0, 1]`) contribute zero and are flagged.
 
 ## Authoritative fields
 
 Each trial writes `verifier/reward.json`:
 
 | Field | Meaning |
 |---|---|
-| `passed` | the task's own pass decision; do not derive it from a global threshold |
+| `passed` | 1 only when evaluation completed and valid `task_score > 0.999`; otherwise 0 |
 | `task_score` | score from 0 to 1 |
 | `evaluation_complete` | whether verification completed |
 
-The 97 verifiers do not share one pass threshold. For full-mark counts, use
-`task_score >= 0.999`; judge averaging can produce a value just below 1.
+There is only one pass field, `passed`, with the same meaning in
+`verifier/reward.json`, `summary.csv`, and `summary.json`. A completed score
+of 0.9991 passes; a score of 0.999 does not. No alternate pass field is emitted.
+
+After authenticating and unsealing the reference, the runtime applies the
+strict score threshold to the staged reward adapter before Harbor runs it. The
+encrypted reference archives and partial-credit rubrics remain unchanged.
+The summarizer also derives `passed` from score and completion when processing
+older results, discarding their old pass decision rather than copying it.
+
+Native grader diagnostics are generated in a temporary verifier directory.
+Before publication, pass decisions are removed from diagnostic JSON and text;
+only `reward.json` retains the benchmark's pass field. Partial scores and
+ordinary error diagnostics are preserved. Raw grader logs are not published.
+
+The single-field guarantee applies to new-policy jobs. Legacy jobs are refused
+on resume, rather than mixing their old rewards with new results. Summarizing
+an old job does not rewrite its original rewards or logs; see
+[resume and upgrade rules](running.md#resume-and-results).
 
-Summarize one or more Harbor job directories with:
+This policy is identified by `metric_definition: score-gt-0.999` in the
+summary. Recompute historical results from raw rewards before comparing them;
+results computed with per-task thresholds, `== 1.0`, or `>= 0.999` are not interchangeable.
+
+Summarize a Harbor job directory with:
 
 ```bash
 python3 scripts/summarize_results.py results/harbor/<job>
 ```
 
+Use one predeclared attempt per task. Duplicate task trials are rejected rather
+than silently counted twice or selected by their score. For a deliberately
+separate subset report (for example, the 81-task open track), specify
+`--expected-total 81` and report that denominator explicitly; it is not the
+97-task headline metric. `--expected-total` must not be reduced to the number
+of tasks that happened to succeed. Missing tasks still contribute zero.
+
+For repeated trials, use `run_eval.sh --n-attempts N --no-summary` and summarize
+each predeclared attempt separately. Use this runtime's summary for the
+fixed-denominator headline metric; Harbor may aggregate only attempted trials.
+
 ## Verifiers and judges
 
 Each task has a frozen verifier. Deterministic checks validate submitted files,
@@ -66,7 +106,7 @@ solve-side hash and an encrypted-verifier hash. Setup refuses mixed releases.
 Report:
 
 - denominator 97, with missing tasks counted as zero;
-- Pass Rate from `passed`, not from a new threshold;
+- Pass Rate from completed `task_score > 0.999`;
 - mean `task_score` times 100;
 - agent, model, judge model, and judge repetitions;
 - pinned Docker image identity and ORCA version for full-track runs;

benchmarks/frontierchallenge/docs/submitting.md

@@ -31,8 +31,9 @@ ORCA version; see [Scoring](scoring.md).
 listed if it is labelled as one, with the count of attempted tasks. It cannot
 be listed as a score over a smaller denominator.
 
-**The `passed` field, not a threshold.** See
-[Scoring](scoring.md#the-two-headline-numbers).
+**Pass Rate.** Count only completed evaluations with
+`task_score > 0.999`; do not apply per-task thresholds or round scores.
+See [Scoring](scoring.md).
 
 **Judge configuration stated.** `gpt-5.6-sol`, `reasoning_effort=high`,
 `JUDGE_REPEATS=3` with `--no-judge-override` is the definitional setting. Any

benchmarks/frontierchallenge/docs/task-format.md

@@ -97,7 +97,11 @@ the task's source/provenance definition, and should be described that way:
 1. runs the LLM judge `JUDGE_REPEATS` times, if the task has one (77 do),
 2. combines them with `statistics.fmean`,
 3. calls the task's own grader with that value as the rubric component,
-4. emits `task_score` and `passed`.
+4. emits `task_score` and `passed`, which is 1 only for a completed valid score above 0.999.
+
+The runtime applies this single pass rule to the staged reward adapter after
+unsealing. Official Pass Rate is computed by the summarizer from completed
+`task_score > 0.999` evaluations, independently of per-task thresholds.
 
 The full tree is stored in the gated dataset's `verifier.fcref`, including the
 solved reference run, reference fixtures, grader source, judge prompt, and