Add resumable DART100 dataset lifecycle evidence tooling - #3
Open
kimminhyun-ai wants to merge 6 commits into
Open
Add resumable DART100 dataset lifecycle evidence tooling#3kimminhyun-ai wants to merge 6 commits into
kimminhyun-ai wants to merge 6 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Deployment-specific dataset → teach → compare-inference evidence harness. Verifies the immutable public HF dataset manifest against 100 registered IDs and canonical hashes; adopts existing jobs, persists submission intent, and stops on ambiguous outcomes instead of resubmitting.
Separates execution coverage from answer accuracy, includes every canonical row, hashes raw responses for resume, and refuses to unload unowned runtime patches. No automatic publication, consent/PII bypass, incentive payout, or claim that 100 patches are 100 distinct models.
Validation
Nine regression tests pass in Node 24 inside a read-only Docker container (2 CPU, 2 GiB, network disabled). A nested CommonJS package boundary fixes the initial ES-module import failure, whose log is retained.
Live deployment: existing first job reused without retraining; 16 compare calls complete with 5/8 primary and 1/8 alternate answers correct. Second dataset is training. All 100 lessons, HF model/service deployment and public Ainize listings remain incomplete. The live snapshot predates two extra resume/error-log guards documented in README.
Limitations
Fixed existing deployment paths and prerequisites; not a fresh-machine bootstrap. Source publication only, not an npm release or a merged/deployed production change.