Repository navigation
fix(benchmark): preserve goals and capture rules without prescribing work order - #6983
Open
huangruiteng wants to merge 3 commits into
Open
huangruiteng wants to merge 3 commits into
huangruiteng wants to merge 3 commits into
Conversation
Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
…raints Signed-off-by: huangruiteng <huangrt01@163.com>
This was referenced Oct 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal And Delivered Outcome
Native benchmark prompts mixed objective/completion and capture requirements with implementation-order advice. This change retains continued optimization and usable capture candidates while removing adapter-imposed work ordering and prior-improvement submission requirements.
Outcome basis: maintainer-requested prompt correction; intended base:
main. The existing Python adapter owns these instructions; no new capability, protocol vocabulary or decision owner is introduced.Author Declaration
Written by: model_agent; OpenAI Codex.
Implemented against
No written specification for the exact prompt edit; the request stated in this PR is the basis. Research acceptance follows long-horizon research RFC §11 at base
4e6baa51afdc2ad522012c1f691ae166c9d67e7b.benchmark/edgebench/prompts.pybenchmark/runtime/planning.pybenchmark/runtime/worker.pySelf-check: reviewed the final prompt clauses against the public provider's automatic archive extraction and submission loop; checked the complete diff, retained authority boundaries and public/private scan. No private trajectory or hidden evaluator content is included.
Scope And Continuation
Complete within this adapter scope. New native trials across all five EdgeBench workers receive the revised default policy. Shared
loopx-plannedprefix removal applies to SForge and Harbor. Continuation wording changes only whennative_goal_continueis enabled.Core first-delivery guidance remains unchanged. The independent shared-planning clarification is #6986. Submission mechanics, scoring, cooldown, final selection, budgets, blind/best-only task wrappers, game prompts, optional observations, stride and checkpoint treatments retain their existing owners. Existing trials are not rewritten. New studies must pin runner/product revisions; mixed versions cannot isolate feedback mode alone. RFC E3 remains the owner for outcome qualification.
The bounded refactor reuses the existing native instruction helper and product planning packet. No new abstraction, model-controlled sampling gate or completion oracle was added.
Validation
9f180a28a8; unchanged shared planning/worker paths also qualified atd4036f7d7e.071a884012754127f28f5b84be10f533a42db527, isolated test environment.PYTHONPATH=.:tests uv run --extra test python -m pytest benchmark/tests/test_edgebench_prompts.py benchmark/tests/test_edgebench_feedback.py benchmark/tests/test_sforge_runtime.py -q -ra --tb=short: 425 passed, no skips. Real local CLI Goal bootstrap/configure/planner/replan; five-profile prompt delivery and synthetic registration.test_task_entry.pyandtest_shared_codex_runtime.py: 182 passed in the previous 465-test runtime run atd4036f7d7e; their production paths and test files are unchanged in this follow-up.previouserrors in untouchedconversation_binding.tsand an untouched workspace-display assertion: Node compatibility, browser suite. New-head CI reruns independently; no green-CI claim.Initial obsolete prompt assertions and optional dependency/import setup were corrected before the final local runs. No unresolved local failures are excluded. The prompt does not claim atomic capture or eliminate the provider's possible partial-write race.
Frontend / Visual Evidence
UI impact: none. Research-adapter instructions and developer/RFC documentation only; no frontend, Lark journey or first-screen changes. Preview/visual review: N/A. Source data: none.
Type of Change
LoopX Area
Technical Direction
S11 / long-horizon benchmark evidence, RFC E1 delivery boundary and E3 study separation. Both RFC language checkpoints are updated. Shared-authority / TypeScript migration fixture impact: N/A; no control-plane state or backend refactor.
Boundary Checklist