Skip to content

Use public native scoring without original-machine archives - #52

Merged
Xubqpanda merged 2 commits into
mainfrom
fix/public-native-scorers
Nov 10, 2026
Merged

Xubqpanda merged 2 commits into
mainfrom
fix/public-native-scorers

Conversation

@Xubqpanda

Copy link
Copy Markdown
Collaborator

Fresh evaluation of ALFWorld, BIRD Mini-Dev, ChessBench puzzles, Jericho and ScienceWorld previously required an original-machine archive of the interpreter, standard library, shared libraries, packages and launcher. Replace that default with public scorer source and locally prepared execution dependencies. A compatible new machine can prepare from the public recipe instead of recovering every historical byte.

All five adapters now select public-native-local-v1. Metrics, native replay, task input contracts and actor/scorer separation remain in place. Scorer code runs from a private read-only snapshot with isolated Python and the selected local dependency environment. Receipts record public source identity and actual local Python, relevant SQLite, dependency versions and runtime identity. Independent local installations are admitted; local changes during a run and changed score reuse identities remain detectable.

BIRD adds an empty standard-library execution recipe, Chess adds its public chess recipe, and ALFWorld/Jericho/ScienceWorld share their existing execution recipes. The old native manifests are unchanged and only selected by explicit historical archive controls. Historical tools/docs now identify that scope rather than presenting archives as new-machine prerequisites.

Preparation: python tools/prepare_dataset.py --dataset --runtime execution --python /absolute/python3.12. No original 3.12 patch release, Python executable digest or host libc bytes are required.

Validation:

  • Relevant public-scoring, native/cache/compatibility and execution-runtime regression suite: 189 passed, 21 skipped (historical archive/extra environment controls).
  • Final public-scoring controls after the last guard/probe refinement: 17 passed.
  • CI guards, documentation, wheel resources and clean-install contracts: 193 passed.
  • Actual BIRD SQLite and Chess engine controls passed with authored positive/negative cases, including actor import shadows and missing historical archive roots. Two distinct prepared local installations ran the same scorer.
  • Published benchmark tasks, real models, and complete ALFWorld/Jericho/ScienceWorld engine trials were not run. Their default declarations and public dependency paths are tested; target-host engine/data validation remains separate.

See docs/public-native-scoring.md for preparation, measurement and archival boundaries.

@Xubqpanda
Xubqpanda merged commit 37c5d35 into main Nov 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant