Skip to content

Avoid eager dataset setup and repeated startup runtime validation - #53

Merged
Xubqpanda merged 1 commit into
mainfrom
perf/lazy-dataset-runtime-declarations
Nov 12, 2026
Merged

Xubqpanda merged 1 commit into
mainfrom
perf/lazy-dataset-runtime-declarations

Conversation

@Xubqpanda

Copy link
Copy Markdown
Collaborator

Importing a configured 134-task ALFWorld pool eagerly built every SETUP, and each called resolve_execution: full venv hashing plus a Python probe. Make ALFWorld, Jericho, ScienceWorld and DS-1000 declarations lazy so key enumeration does not construct setup/verifier values.

Registry declaration batches verify each execution artifact once and reuse metadata copies within that operation. Provider snapshots retain the artifact admitted at entry instead of re-admitting it per task. Actual provider RPC startup, actor/service/verifier launch, input consumption and score-reuse admission still validate freshly. No process-global permission cache is introduced.

The repository scan also fixes full-verifier traversal for inventory kind reporting, repeated native-research manifest/index construction, repeated SpreadsheetBench native inventories during planning, shared source/JAR/ROM reads, frozen-session catalog subprocess repetition, and a duplicate TravelPlanner world check. World/database byte checks remain fresh per selected task; only the immediate duplicate is removed. Source hashing is scoped rather than relying on process-wide timestamp caches, including filesystems with coarse timestamps.

Controlled evidence: the same authored 134-task pool produces 134 runtime resolutions at import before the change and 0 after it. A 134-lookup metadata batch admits once; a subsequent scope and explicit launch admission still recheck, including a damaged-artifact control. No real ALFWorld GB-read or wall-time improvement is claimed.

Validation:

  • Dataset/runtime/provider regression: 238 passed, 43 environment-dependent skips.
  • CI guards, wheel resources and clean-install checks: 193 passed.
  • Final snapshot API/entry-admission regression: 50 passed.
  • Focused performance controls: 13 passed, covering lazy pool imports, selected-only construction, scoped admission, catalog copies/config changes, manifest/native inventory reuse, restored-mtime source edits and fresh world checks.
  • Scan covered 1,522 tracked Python files plus Shell/JS/TS checks. No released benchmark or model run, and no shared service changes.

@Xubqpanda
Xubqpanda merged commit 6612c22 into main Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant