[skill] evaluation: mandate 8 SciCode runs and report the mean

plugins/modelopt/skills/compare-results/SKILL.md

@@ -71,9 +71,9 @@ the validated runs are comparable:
    precision the baseline is**, and apply the gate relative to that baseline
    rather than to an assumed BF16.
 
-For SciCode, keep `num_repeats: 1` to limit sandbox workload. If variance is a
-concern, run multiple independent matched baseline/candidate pairs instead of
-increasing repeats within one run.
+For SciCode, keep `num_repeats: 1` and require **8 runs per side**, comparing
+the two means — see the evaluation skill's `recipes/tasks/aa/scicode.md`. Fewer
+than 8 valid runs on a side is `INDETERMINATE`, not a delta.
 
 If any item differs, either rerun with matched settings or label the result as
 not an apples-to-apples quantization comparison.

plugins/modelopt/skills/day0-release/SKILL.md

@@ -181,11 +181,13 @@ For example, on a 1 % gate on DeepSeek-V4-Pro, both tasks that originally failed
 | IFBench | 16 | **0.63 pp** | PASS |
 
 Add precision by submitting the benchmark **more times**, not by raising
-`num_repeats` within a run — see `recipes/tasks/aa/scicode.md` for why.
+`num_repeats` within a run. SciCode's floor is **8 runs per side**, pooled as a
+mean — `recipes/tasks/aa/scicode.md`.
 
 **Re-running does not guarantee fresh samples.** With a warm NEL response cache a "re-run" can
-replay cached responses — two runs came back bit-identical to 16 digits. Confirm the score
-actually moved before counting a run as an independent repeat.
+replay cached responses — two runs came back bit-identical to 16 digits. But an equal score is a
+signal to check provenance (invocation id, output dir, response artifacts), **not** proof of replay:
+discrete-scored tasks tie legitimately. Confirm replay before discarding a run as a duplicate.
 
 After recording the external status, produce per-task deltas and run:
 

plugins/modelopt/skills/evaluation/SKILL.md

@@ -137,7 +137,7 @@ Run `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`.
   | **GDPVal** (Stirrup agent, agentic) | `recipes/tasks/gym/gdpval.md` + `references/gym-gdpval.md`, `recipes/examples/gym/example_gdpval.yaml` | **Yes** | any AA request (see the AA rule below) |
   | **MRCR** (simple agent, long-context) | `recipes/tasks/gym/mrcr.md`, `recipes/examples/gym/example_mrcr.yaml` | **No** | only when the user asks for MRCR by name, or for long-context coverage |
 
-**AA rule:** If the user mentions "AA" / "Artificial Analysis", generate the `recipes/tasks/aa/` tasks (one multi-task config) **plus a companion standalone GDPVal config** (`recipes/tasks/gym/gdpval.md`, via the GDPVal branch) — GDPVal is part of the AA suite but a different harness, so it's its own config, never added to the `aa/` `tasks` list. Do not add MMLU-Pro, AIME 2025, or LiveCodeBench unless explicitly asked. GDPVal is the heaviest AA task (standalone, multi-hour, needs the SIF sandbox + judge) — surface it and let the user opt out per run.
+**AA rule:** If the user mentions "AA" / "Artificial Analysis", generate the `recipes/tasks/aa/` tasks (one multi-task config) **plus a companion standalone GDPVal config** (`recipes/tasks/gym/gdpval.md`, via the GDPVal branch) — GDPVal is part of the AA suite but a different harness, so it's its own config, never added to the `aa/` `tasks` list. Do not add MMLU-Pro, AIME 2025, or LiveCodeBench unless explicitly asked. GDPVal is the heaviest AA task (standalone, multi-hour, needs the SIF sandbox + judge) — surface it and let the user opt out per run. **SciCode needs 8 submissions, not one**, reported as their mean (`recipes/tasks/aa/scicode.md`).
 
 **Shortcut path** (when task list is known up front, e.g. "run AA"):
 
@@ -381,7 +381,7 @@ Re-submit again if it's preempted again — each resume re-deploys, then skips a
 
 Implications for the agent:
 
-- Do **not** lower `num_repeats`, split heavy tasks (AA-LCR, SciCode) into separate configs, or otherwise carve up the eval to fit inside 4h. Let NEL chain.
+- Do **not** lower `num_repeats`, split heavy tasks (AA-LCR, SciCode) into separate configs, or otherwise carve up the eval to fit inside 4h. Let NEL chain. (SciCode's mandatory 8 submissions are independent scored runs, not a walltime workaround.)
 - Do **not** treat a walltime timeout as a failed run. Check `nel status` / `nel info` and the dependent job's logs before declaring failure. `references/run-validation.md` covers what a real failure looks like vs an expected resume event.
 - Bumping `data_parallel_size` / `parallelism` to finish faster is fine when the goal is wall-clock latency, not a walltime workaround — but it's optional, not required, for runs longer than 4h.
 

plugins/modelopt/skills/evaluation/recipes/tasks/aa/scicode.md

@@ -19,12 +19,34 @@ harvesting requirements beyond the task YAML fragment.
   it unless you have a memory reason to.
 - **Parallelism:** set task-level `parallelism: 8` exactly. Use the same value
   for baseline and candidate.
-- **One run is not enough to gate on.** Scored single-shot at `temperature 1.0`,
-  SciCode's run-to-run noise rivals a 1 % gate: a paired comparison moved **3.92 pp
-  and changed sign** once it was repeated. Keep `num_repeats: 1` and instead submit
-  the benchmark **multiple times**, pooling across runs — repeating inside one run
-  multiplies exposure to code-execution sandbox errors. Pool until the standard
-  error is below the threshold, or report the task `INDETERMINATE`.
+- **Repeats: 8 runs, not `num_repeats: 8`.** Keep `num_repeats: 1` — in-run
+  repeats multiply code-execution sandbox exposure — and submit the task 8 times
+  instead. See [8 runs](#8-runs-mandatory).
+
+## 8 runs (mandatory)
+
+One run is never a reportable SciCode score: at `temperature 1.0` its noise
+rivals a 1 % gate — a paired comparison moved **3.92 pp and changed sign** when
+repeated, and a DeepSeek-V4-Pro drop read 2.96 pp (`REGRESSION`) at 1 run but
+**-0.96 pp** (`PASS`) at 8. The count is fixed, not a variance judgment call.
+
+- **Submit until 8 *valid* runs** — normally 8 submissions, more if any fail
+  validation; 8 per side for a comparison (8-vs-1 is not apples-to-apples). In a
+  multi-task AA config, run the suite once, then SciCode alone for the rest:
+  `for _ in $(seq 7); do nel run --config <cfg> -t ns_scicode; done`.
+- **Each must be a fresh `nel run`.** Re-submitting a run's `run.sub` resumes
+  from its response cache and replays generations. Equal scores are a **signal
+  to check, not proof** — SciCode's score is discrete, so independent runs tie
+  often; confirm replay from provenance (invocation id, output dir, response
+  artifacts) before discarding, then resubmit to restore the pool to 8.
+- **Report the mean** of the 8, with `stdev / sqrt(8)` and the run count.
+  Validate each run (`references/run-validation.md`) before averaging it in;
+  resubmit to replace invalid runs rather than pooling a sandbox-crashed one.
+- **Expect an export to time out.** The 8 exports land on the CPU partition
+  together and each reinstalls the launcher against a fixed 30 min sbatch
+  limit. The score survives in the run artifacts — re-submit that run's
+  `export.sbatch`; it is not a lost run.
+- **Fewer than 8 valid runs is `INDETERMINATE`**, never a pass/fail verdict.
 
 ## YAML Fragment
 
@@ -39,9 +61,16 @@ Use this inside the top-level `evaluation.tasks` list:
         parallelism: 8
         extra:
           args: ++prompt_config=eval/scicode/default ++with_background=true
-          num_repeats: 1 # keep at 1; pool across multiple runs to gate (see above)
+          num_repeats: 1 # keep at 1; submit this config 8x and average (see above)
 ```
 
 ## Score Extraction from mlflow
 
-Result (0-100): `scicode_pass_at_1_avg-of-1_subtask_accuracy`
+Per-run score (0-100): `scicode_pass_at_1_subtask_accuracy`
+
+**No `avg-of-N` segment at `num_repeats: 1`** — the harness adds one only when
+N > 1 (cf. `gpqa_pass_at_1_avg-of-16_symbolic_correct`); the `avg-of-1` name
+silently harvests nothing.
+
+The reported SciCode result is the **mean of that field across the 8 runs**, not
+any single run's value.

plugins/modelopt/skills/evaluation/references/quantization-benchmarks.md

@@ -63,8 +63,9 @@ merged into the `aa/` multi-task list.
   do **not** lower them for quant comparisons, or noise will mask real
   regressions. The field name differs by harness: `n_samples` for simple-evals
   (AIME `64`) and tau2-bench (Tau2 `8`); `num_repeats` for nemo-skills
-  (AA-LCR/GPQA `16`, AA-Omniscience `10`, LiveCodeBench/SciCode `8`, IFBench `5`,
-  MMLU-Pro `1`).
+  (AA-LCR/GPQA `16`, AA-Omniscience `10`, LiveCodeBench `8`, IFBench `5`,
+  MMLU-Pro `1`). **SciCode is the exception** — `num_repeats: 1` x 8 separate
+  submissions, averaged (`tasks/aa/scicode.md`).
 - **Judge / user-simulator endpoints** are required by AA-LCR, HLE AA,
   AA-Omniscience, and Tau2-Bench Telecom. Keep the judge and (for Tau2)
   user-simulator models fixed across baseline and quantized runs for

plugins/modelopt/skills/evaluation/tests/evals.json

@@ -71,5 +71,19 @@
       "Routes baseline-vs-quantized comparability verification to the compare-results skill before presenting an accuracy delta",
       "Provides SSH-based log monitoring commands for SLURM execution"
     ]
+  },
+  {
+    "name": "scicode-eight-run-average",
+    "skills": ["evaluation"],
+    "query": "Evaluate my NVFP4 checkpoint on the AA suite including SciCode on SLURM",
+    "files": [],
+    "expected_behavior": [
+      "Keeps num_repeats at 1 and plans 8 independent SciCode submissions instead",
+      "Re-submits SciCode alone with -t ns_scicode rather than re-running the whole suite",
+      "Reports the SciCode score as the mean of the 8 runs with standard error and run count",
+      "Confirms cache replay from run provenance rather than equal scores before excluding a run",
+      "Re-submits replacements until 8 valid independent runs exist before reporting a mean",
+      "Reports SciCode as INDETERMINATE when fewer than 8 valid runs are available"
+    ]
   }
 ]