Skip to content

fix(benchmark): preserve goals and capture rules without prescribing work order - #6983

Open
huangruiteng wants to merge 3 commits into
mainfrom
codex/benchmark-context-only-prompts
Open

huangruiteng wants to merge 3 commits into
mainfrom
codex/benchmark-context-only-prompts

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Native benchmark prompts mixed objective/completion and capture requirements with implementation-order advice. This change retains continued optimization and usable capture candidates while removing adapter-imposed work ordering and prior-improvement submission requirements.

  • Keep pursuing the original task objective: implementation, passing checks and submission alone do not establish completion. Respect task acceptance, budget, explicit stops and genuine blockers.
  • When automatic evaluation is enabled, explain that the host samples on-disk submission paths independently of manual submissions and local validation. Keep a runnable candidate there, save completed changes, retain a recoverable best-supported version and bind results to their source version. With interval zero, report periodic evaluation as disabled.
  • Remove the shared benchmark planning strategy prefix and the native wrapper's incremental-implementation workflow. Explicitly add no local-test pass or prior-improvement prerequisite; task-owned rules remain authoritative.
  • Opted-in native Goal continuation follows the original task and evaluation rules, without requiring improvement before submission.

Outcome basis: maintainer-requested prompt correction; intended base: main. The existing Python adapter owns these instructions; no new capability, protocol vocabulary or decision owner is introduced.

Author Declaration

Written by: model_agent; OpenAI Codex.

Implemented against

No written specification for the exact prompt edit; the request stated in this PR is the basis. Research acceptance follows long-horizon research RFC §11 at base 4e6baa51afdc2ad522012c1f691ae166c9d67e7b.

Criterion Disposition Owning path Evidence
Preserve optimization objective and genuine stopping boundaries implemented benchmark/edgebench/prompts.py Objective/completion clauses and task-owned explicit-stop case
Preserve automatic capture and source/result meaning implemented Same native wrapper Sampling on/off crossed with network modes, actual CLI registration and five worker profiles
Remove imposed work order, retain task-owned validation rules implemented Native wrapper and benchmark/runtime/planning.py Early-diagnostic and local-validation-first tasks preserved verbatim; exact Core packet readback
Continuation preserves task-owned submission eligibility implemented benchmark/runtime/worker.py Optional continuation, default-off, deadline, reserve and cancellation tests
Earlier useful feedback or better scores out_of_scope RFC E3 matched studies No effectiveness claim or new benchmark run

Self-check: reviewed the final prompt clauses against the public provider's automatic archive extraction and submission loop; checked the complete diff, retained authority boundaries and public/private scan. No private trajectory or hidden evaluator content is included.

Scope And Continuation

Complete within this adapter scope. New native trials across all five EdgeBench workers receive the revised default policy. Shared loopx-planned prefix removal applies to SForge and Harbor. Continuation wording changes only when native_goal_continue is enabled.

Core first-delivery guidance remains unchanged. The independent shared-planning clarification is #6986. Submission mechanics, scoring, cooldown, final selection, budgets, blind/best-only task wrappers, game prompts, optional observations, stride and checkpoint treatments retain their existing owners. Existing trials are not rewritten. New studies must pin runner/product revisions; mixed versions cannot isolate feedback mode alone. RFC E3 remains the owner for outcome qualification.

The bounded refactor reuses the existing native instruction helper and product planning packet. No new abstraction, model-controlled sampling gate or completion oracle was added.

Validation

  • Tested revision: 9f180a28a8; unchanged shared planning/worker paths also qualified at d4036f7d7e.
  • Run state: finished. Input classes: synthetic, public_fixture.
  • Environment: Python 3.13.13; Harbor 0.24.0; SForge 1.1.0 from public provider revision 071a884012754127f28f5b84be10f533a42db527, isolated test environment.
Check kind Result Public-safe evidence / limitation
integration / real_entrypoint passed PYTHONPATH=.:tests uv run --extra test python -m pytest benchmark/tests/test_edgebench_prompts.py benchmark/tests/test_edgebench_feedback.py benchmark/tests/test_sforge_runtime.py -q -ra --tb=short: 425 passed, no skips. Real local CLI Goal bootstrap/configure/planner/replan; five-profile prompt delivery and synthetic registration.
unit / integration passed test_task_entry.py and test_shared_codex_runtime.py: 182 passed in the previous 465-test runtime run at d4036f7d7e; their production paths and test files are unchanged in this follow-up.
regression_parity passed Five cases reject baseline adapter implementations. Mutations deleting continued-optimization or runnable-sampling guidance are rejected. Blind/best-only wrappers and optional checkpoint storage guidance match baseline byte-for-byte on tested inputs.
static passed Ruff; diff whitespace check; bilingual RFC check; public-boundary scan of all twelve changed files and PR body
CI failed Earlier head exposed TypeScript nullable previous errors in untouched conversation_binding.ts and an untouched workspace-display assertion: Node compatibility, browser suite. New-head CI reruns independently; no green-CI claim.
real_backend / effectiveness not_run No live judge, paid model or benchmark campaign. Provider code and synthetic integration qualify prompt delivery, not actual score improvement or sampling reliability under concurrent writes.

Initial obsolete prompt assertions and optional dependency/import setup were corrected before the final local runs. No unresolved local failures are excluded. The prompt does not claim atomic capture or eliminate the provider's possible partial-write race.

Frontend / Visual Evidence

UI impact: none. Research-adapter instructions and developer/RFC documentation only; no frontend, Lark journey or first-screen changes. Preview/visual review: N/A. Source data: none.

Type of Change

  • Bug fix (default prompt-policy behavior change)
  • Documentation update
  • Test update

LoopX Area

  • Benchmark boundary (adapters, runners, verifiers, scoring, evidence)

Technical Direction

S11 / long-horizon benchmark evidence, RFC E1 delivery boundary and E3 study separation. Both RFC language checkpoints are updated. Shared-authority / TypeScript migration fixture impact: N/A; no control-plane state or backend refactor.

Boundary Checklist

  • Diff and PR exclude private state, credentials, raw traces, verifier output, internal links and local machine paths.
  • Related work checked; no duplicate submission-recovery implementation or benchmark launch.
  • Change remains scoped to the adapter correction.
  • Both RFC language files changed; bilingual check passes.
  • UI impact marked none.
  • Every commit includes a DCO Signed-off-by trailer.

Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
…raints

Signed-off-by: huangruiteng <huangrt01@163.com>
@huangruiteng huangruiteng changed the title fix(benchmark): keep adapter prompts focused on task context fix(benchmark): preserve goals and capture rules without prescribing work order Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant