Repository navigation
Avoid eager dataset setup and repeated startup runtime validation - #53
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Importing a configured 134-task ALFWorld pool eagerly built every SETUP, and each called resolve_execution: full venv hashing plus a Python probe. Make ALFWorld, Jericho, ScienceWorld and DS-1000 declarations lazy so key enumeration does not construct setup/verifier values.
Registry declaration batches verify each execution artifact once and reuse metadata copies within that operation. Provider snapshots retain the artifact admitted at entry instead of re-admitting it per task. Actual provider RPC startup, actor/service/verifier launch, input consumption and score-reuse admission still validate freshly. No process-global permission cache is introduced.
The repository scan also fixes full-verifier traversal for inventory kind reporting, repeated native-research manifest/index construction, repeated SpreadsheetBench native inventories during planning, shared source/JAR/ROM reads, frozen-session catalog subprocess repetition, and a duplicate TravelPlanner world check. World/database byte checks remain fresh per selected task; only the immediate duplicate is removed. Source hashing is scoped rather than relying on process-wide timestamp caches, including filesystems with coarse timestamps.
Controlled evidence: the same authored 134-task pool produces 134 runtime resolutions at import before the change and 0 after it. A 134-lookup metadata batch admits once; a subsequent scope and explicit launch admission still recheck, including a damaged-artifact control. No real ALFWorld GB-read or wall-time improvement is claimed.
Validation: