Skip to content

Report when the gold check had nothing to check against - #2293

Open
ATMAECHO wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
ATMAECHO:gold-check-unimplemented
Open

Report when the gold check had nothing to check against#2293
ATMAECHO wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
ATMAECHO:gold-check-unimplemented

Conversation

@ATMAECHO

@ATMAECHO ATMAECHO commented Aug 7, 2026

Copy link
Copy Markdown

The problem

Task.validate returns True unconditionally, so a taskset that does not override it
passes the gold check for every item without anything having been checked.

Of the five in-tree tasksets defining a Task subclass, only lean overrides validate.
harbor, openenv, textarena and nemo_gym inherit the base implementation. For those,
validate --only-gold reports valid_rate: 1.0 — which reads as a passing check, but
measures nothing. The failure is silent and flattering: absence of a check is reported as
agreement.

The change

34 lines in verifiers/v1/cli/validate.py:

  • gold_implemented(task) — whether the taskset actually wrote a gold check
    (type(task).validate is not Task.validate, so inheriting from an overriding class
    still counts).
  • _run_gold marks each row checked: true|false.
  • summarize reports gold_unchecked in gold and all modes (_gold_row reads the
    nested gold row in all mode).
  • run_validate warns once, naming the classes that don't override validate.

Validation behaviour is unchanged. This only separates "checked and valid" from
"never checked".

A stronger alternative, if you'd prefer it

Add an unchecked value to reason so it fails loudly rather than being reported in the
summary. I kept this additive because that would touch FINAL_REASONS, _is_final and
resume semantics — happy to switch if you'd rather have the louder version.

Verification

Per AGENTS.md ("to check your own work, write a temporary script instead of committing new
tests") I used a temporary script, not a committed test. It checks:

  • the predicate on synthetic cases: inherits Task.validate, overrides it, and inherits a
    class that overrides it;
  • all five in-tree taskset classes, imported live — 4 unchecked, lean checked;
  • summarize in gold, all and setup modes, including a row with no checked field;
  • a negative control — the old always-true predicate — which fires, so the checks aren't
    passing vacuously.

All pass. uv run ruff check and uv run ruff format --check are clean.

One caveat I'd rather state than hide: I could not run uv run pytest tests/ on this
machine. verifiers/v1/runtimes/limiters.py imports fcntl, which is Unix-only, so the
package doesn't import on Windows. I stashed the change, collected, un-stashed, and
collected again — the failure set is byte-identical (38 both ways, same 6 collection
errors), so nothing here is caused by this change. Still worth a CI run on Linux.

Note

Report unchecked gold validations when a task does not override Task.validate

  • Adds a gold_unchecked count to the summary in gold and all modes, tracking results that passed without a real gold check being implemented.
  • Each result row from _run_gold now includes a checked boolean derived from whether the task's class overrides Task.validate.
  • Before validation runs, run_validate logs a warning listing any task classes whose gold checks will always pass due to missing validate overrides.

Macroscope summarized 4a74e93.

`Task.validate` returns `True` unconditionally, so a taskset that does not
override it passes the gold check for every item without anything having been
checked. Of the five in-tree tasksets defining a `Task` subclass, only `lean`
overrides `validate`; `harbor`, `openenv`, `textarena` and `nemo_gym` inherit
the base implementation. For those, `validate --only-gold` reports
`valid_rate: 1.0`, which reads as a passing check but measures nothing.

Mark such rows with `checked: false`, count them as `gold_unchecked` in the
summary, and warn once per taskset naming the classes involved. Validation
behaviour is unchanged; this only distinguishes "checked and valid" from
"never checked".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant