Report when the gold check had nothing to check against - #2293
Open
ATMAECHO wants to merge 1 commit into
Open
Conversation
`Task.validate` returns `True` unconditionally, so a taskset that does not override it passes the gold check for every item without anything having been checked. Of the five in-tree tasksets defining a `Task` subclass, only `lean` overrides `validate`; `harbor`, `openenv`, `textarena` and `nemo_gym` inherit the base implementation. For those, `validate --only-gold` reports `valid_rate: 1.0`, which reads as a passing check but measures nothing. Mark such rows with `checked: false`, count them as `gold_unchecked` in the summary, and warn once per taskset naming the classes involved. Validation behaviour is unchanged; this only distinguishes "checked and valid" from "never checked".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Task.validatereturnsTrueunconditionally, so a taskset that does not override itpasses the gold check for every item without anything having been checked.
Of the five in-tree tasksets defining a
Tasksubclass, onlyleanoverridesvalidate.harbor,openenv,textarenaandnemo_gyminherit the base implementation. For those,validate --only-goldreportsvalid_rate: 1.0— which reads as a passing check, butmeasures nothing. The failure is silent and flattering: absence of a check is reported as
agreement.
The change
34 lines in
verifiers/v1/cli/validate.py:gold_implemented(task)— whether the taskset actually wrote a gold check(
type(task).validate is not Task.validate, so inheriting from an overriding classstill counts).
_run_goldmarks each rowchecked: true|false.summarizereportsgold_uncheckedingoldandallmodes (_gold_rowreads thenested gold row in
allmode).run_validatewarns once, naming the classes that don't overridevalidate.Validation behaviour is unchanged. This only separates "checked and valid" from
"never checked".
A stronger alternative, if you'd prefer it
Add an
uncheckedvalue toreasonso it fails loudly rather than being reported in thesummary. I kept this additive because that would touch
FINAL_REASONS,_is_finalandresume semantics — happy to switch if you'd rather have the louder version.
Verification
Per AGENTS.md ("to check your own work, write a temporary script instead of committing new
tests") I used a temporary script, not a committed test. It checks:
Task.validate, overrides it, and inherits aclass that overrides it;
leanchecked;summarizeingold,allandsetupmodes, including a row with nocheckedfield;passing vacuously.
All pass.
uv run ruff checkanduv run ruff format --checkare clean.One caveat I'd rather state than hide: I could not run
uv run pytest tests/on thismachine.
verifiers/v1/runtimes/limiters.pyimportsfcntl, which is Unix-only, so thepackage doesn't import on Windows. I stashed the change, collected, un-stashed, and
collected again — the failure set is byte-identical (38 both ways, same 6 collection
errors), so nothing here is caused by this change. Still worth a CI run on Linux.
Note
Report unchecked gold validations when a task does not override
Task.validategold_uncheckedcount to the summary ingoldandallmodes, tracking results that passed without a real gold check being implemented._run_goldnow includes acheckedboolean derived from whether the task's class overridesTask.validate.run_validatelogs a warning listing any task classes whose gold checks will always pass due to missingvalidateoverrides.Macroscope summarized 4a74e93.