Problem
packages/opencode/src/altimate/prompts/builder.txt carries a ## Pre-Execution Protocol section that makes sql_analyze + altimate_core_validate mandatory before every sql_execute. builder is a PRIMARY agent, so that section reaches every builder surface at once: dbt authoring, interactive chat, and headless question-answering runs.
An internal pre-registered paired prompt ablation (540 trials on a public data-question benchmark, one binary across both arms, per-trial system prompts verified by sha256) measured what the section costs on the question-answering surface:
|
macro Pass@1 |
| control |
0.6667 |
| treatment (protocol removed, among other changes) |
0.6807 |
| delta |
+0.0140 |
- query-blocked sign-flip permutation, 20,000 resamples: p = 0.7358
- cluster-bootstrap 95% CI: [-0.0400, +0.0674]
That is a null on score. The point estimate is positive and the interval comfortably contains zero — it is not an improvement and must not be reported as one.
What did move:
|
control |
treatment |
delta |
| wall clock per trial (restricted mean) |
440.9s |
319.4s |
-27.6% |
| model turns per trial |
11.9 |
8.6 |
-27.7% |
| generation seconds per trial |
187.6 |
127.2 |
-32.2% |
altimate_core_validate calls (arm total) |
1,476 |
0 |
-1,476 |
sql_analyze calls (arm total) |
1,329 |
0 |
-1,329 |
sql_execute calls (arm total) |
1,716 |
2,554 |
+49% |
| trials hitting the 900s timeout |
27 |
16 |
-11 |
The 2,805 ritual tool calls going to zero is the number directly attributable to this text. The ritual is prompt-ordered: deleting the order deletes it completely, not partially. The freed budget went into actual querying (sql_execute +49%).
Why not just delete it
Two reasons, both from the experiment's own caveats:
- The latency win is not attributable to this section alone. The treatment arm bundled five coupled changes; the experiment explicitly declined to attribute and a factorial was not run.
- The measurement covers data questions only. dbt authoring and interactive chat are unmeasured builder surfaces where a pre-execution discipline may genuinely earn its place. Deleting on this evidence would over-generalise from one workload.
Proposal
Scope the section to task shape rather than delete it, following the precedent already in the tree: SessionTermination.completionInstruction moved a run-mode-only instruction out of builder.txt and injects it only when headless AND the agent is builder.
Drop the protocol only in the cell that was measured (headless + builder + no dbt project in the workspace) and keep it everywhere else, including whenever the workspace cannot be classified. The cost of keeping it is 27% latency on one workload; the cost of wrongly dropping it is unmeasured.
Out of scope
main also carries a ## Finish Protocol section (shipped in #1171) — a second mandatory ritual in the same family, added after the binary the ablation measured was built. No measurement covers it and this issue does not propose touching it.
A follow-up measurement on dbt tasks is needed before the gate could be widened, or the section deleted outright.
Problem
packages/opencode/src/altimate/prompts/builder.txtcarries a## Pre-Execution Protocolsection that makessql_analyze+altimate_core_validatemandatory before everysql_execute.builderis a PRIMARY agent, so that section reaches every builder surface at once: dbt authoring, interactive chat, and headless question-answering runs.An internal pre-registered paired prompt ablation (540 trials on a public data-question benchmark, one binary across both arms, per-trial system prompts verified by sha256) measured what the section costs on the question-answering surface:
That is a null on score. The point estimate is positive and the interval comfortably contains zero — it is not an improvement and must not be reported as one.
What did move:
altimate_core_validatecalls (arm total)sql_analyzecalls (arm total)sql_executecalls (arm total)The 2,805 ritual tool calls going to zero is the number directly attributable to this text. The ritual is prompt-ordered: deleting the order deletes it completely, not partially. The freed budget went into actual querying (
sql_execute+49%).Why not just delete it
Two reasons, both from the experiment's own caveats:
Proposal
Scope the section to task shape rather than delete it, following the precedent already in the tree:
SessionTermination.completionInstructionmoved a run-mode-only instruction out ofbuilder.txtand injects it only when headless AND the agent isbuilder.Drop the protocol only in the cell that was measured (headless + builder + no dbt project in the workspace) and keep it everywhere else, including whenever the workspace cannot be classified. The cost of keeping it is 27% latency on one workload; the cost of wrongly dropping it is unmeasured.
Out of scope
mainalso carries a## Finish Protocolsection (shipped in #1171) — a second mandatory ritual in the same family, added after the binary the ablation measured was built. No measurement covers it and this issue does not propose touching it.A follow-up measurement on dbt tasks is needed before the gate could be widened, or the section deleted outright.