Say where the corpus gate is and what to do when it reports an improvement (#529, #500) - #948
Merged
Merged
Conversation
…ement (#529, #500) The corpus now runs in the suite, so it runs on every commit, and a contributor who makes the library solve something new will meet it as a failing test rather than as a green one. AGENTS.md says what to do about that -- record the improvement in Corpus.cs in the same change -- because a gate that fails on getting better is confusing exactly once and then only if nobody wrote it down. It also says what the gate is not. It is small and takes about a second because everything in the suite is paid for on every commit; the harnesses in work/ are where a measurement generates its own inputs and takes minutes. A finding from one of those that is worth keeping belongs in the corpus as a new problem. CLAUDE.md's paragraph on the harnesses said they all live outside the repository, which is now one sentence short of true. #529 #500 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The corpus from #945 runs in the suite, so it runs on every commit — which means a contributor who makes the library solve something new meets it as a failing test. That is deliberate: the record must not drift away from the library. But a gate that fails on getting better is confusing exactly once, and only if nobody wrote down what to do.
AGENTS.mdnow says it:ExpectinCorpus.csin the same change — that is how the improvement gets recorded rather than absorbed;BREAKING-CHANGES.mdtoo.It also says what the gate is not. It is small and takes about a second, because everything in the suite is paid for on every commit. The harnesses in
work/are where a measurement generates its own inputs, takes minutes, and gets read by a person. The two are not substitutes — and a finding from a harness that is worth keeping belongs in the corpus as a new problem.CLAUDE.md's paragraph saying the measurement harnesses all live outside the repository was one sentence short of true once #945 merged.Documentation only; no code, no behaviour, no test changes.
🤖 Generated with Claude Code