feat(validators): deterministic completion gates — zero-write, build-green, literal deliverables, config/dialect lints

[anandgupta42]

Issue for this PR

Closes #1174

Type of change

• New feature
• Bug fix

What does this PR do?

Adds five deterministic completion-gate validators to the existing ALTIMATE_VALIDATORS_ENABLED lane, and closes a structural blind spot in that lane. Every check is answer-free — it asserts structure, invariants, or the task's own literal contract, never a known-correct output — so the gates work on unseen tasks.

Closes the zero-write blind spot. Both pre-existing validators key on "did the session modify models", so a session that authored nothing passed every gate by default. dbt-nothing-built is an inverse gate: in a dbt project, with no session-authored files and no fresh successful run artifact, the session is not done. It is deliberately conservative — appliesTo returns false unless a task document literally names required deliverables, or ALTIMATE_VALIDATORS_REQUIRE_ARTIFACTS=1 is set — so genuinely read-only/analysis sessions are unaffected.

New validators

Validator	Catches
dbt-nothing-built	Declared done without producing any deliverable
dbt-build-green	Edited-but-never-built; artifact predating the session; fresh artifact where the model errored, is missing, or was edited after the build
dbt-deliverable-names	Required model/relation names not produced; self-chosen substitutes. Diffs literal names against filesystem inventory u manifest names/aliases
dbt-incremental-config	merge/delete+insert without unique_key; missing is_incremental() guard where the task literally requires idempotency; non-deterministic calls inside the guard predicate
dbt-dialect-guard	Unguarded warehouse-specific functions, only in projects that already use target.type guards (or opt-in via env)

Conservative by construction. No fuzzy matching anywhere — required names come from three literal tiers only. No discoverable source of required names means a silent skip, never a false failure. dbt_project.yml-inherited config is intentionally not resolved rather than guessed. Non-determinism outside an is_incremental() predicate is advisory detail, never blocking. Out-of-scope build failures are telemetry, not a block. Per the lane's existing contract, a validator that throws soft-passes, so a buggy check cannot brick a session.

Also adds docs/internal/deterministic-checks-engine-split.md, assessing two further candidate checks that need real SQL parsing rather than filesystem/regex analysis, and where each belongs relative to the engine's existing capabilities.

Reviewer-critical: these gates almost never get to run on a current build

None of the dispatch code is in this PR. git diff main...HEAD touches no file under packages/opencode/src/session/. The hook already exists on main; this PR only adds validators to the registry it calls. The consequence is that the value of this PR is contingent on how often that hook fires, which is a property of the harness, not of these five validators.

The gate condition, at packages/opencode/src/session/prompt.ts:1405-1412:

if (
  validatorsActive &&
  result !== "stop" &&
  result !== "compact" &&
  processor.message.finish === "stop" &&
  !processor.message.error &&
  validatorCount > 0
)

Two conditions skip dispatch entirely, and neither logs anything beyond the validator_hook_reached line:

1. processor.message.finish === "stop" — the model must declare a clean stop. Any other finish reason (tool-calls, length, unknown, or an error) skips.
2. result !== "compact" — processor.process() returns "compact" when the step ended in compaction (packages/opencode/src/session/processor.ts:661), and those steps skip too. result === "stop" (blocked, or an assistant-message error) skips as well.

Measured on a pre-#1171 build, 9 trials / 72 validator_hook_reached events:

finish	result	count
tool-calls	continue	54
tool-calls	compact	17
stop	continue	1

The gate fired on 1.4% of step-ends. All seven validators registered every time, so registration is fine, and the one dispatch went hook → dispatch_enter → dispatch_result with no throw or hang: the mechanism is correct whenever it is reached. The accurate word is structurally starved, not broken — sessions on that build end by exhausting turns or by compacting, not by declaring completion.

#1171 is the dependency. It replaces trusting a bare provider finish-stop with explicit DONE-token termination (session/termination.ts). Until it lands, enabling this lane on a current build is close to a no-op. A reviewer evaluating this PR on merit should know that the pre-#1171 measurements say nothing about whether these gates work.

What the single dispatch showed, reported because it is the only enforcement event in the run and a reviewer should hear it here rather than find it later:

checks=4  dbt-build-green ok=True  dbt-incremental-config ok=True
          dbt-schema-verify ok=True  dbt-tests-pass ok=FALSE

Three of this PR's five validators did not declare themselves applicable at all. The two that ran both passed. The only validator that failed was dbt-tests-pass, which is pre-existing, not one of the five — and it converted nothing: the off, shadow and enforce arms all passed that task, so the retry it triggered was spent on work that would have succeeded anyway. This is N = 1 on a build where the gate is starved; it is not a verdict on the five, in either direction.

Open design question for the reviewer, deliberately not acted on here. The two endings where dispatch is silent — budget exhaustion and compaction — are arguably the two where a completion gate is most valuable, because they are the endings most likely to hand a user unfinished work. There is a defensible reading (the gates check a claim of completion; no claim, nothing to validate; a stalled run has already failed by other means) and a worrying one (this is a gate that is quiet precisely when it should be loud, which is the same shape as the dbt-build-green-passes-with-zero-models bug this PR exists to fix). Changing it means running validators on abnormal finish reasons and converting a silent stall into an explicit failure — a change to shared harness code outside this PR's scope. Flagging it for a decision rather than making it.

Recommendation: shadow mode first, not enable-by-default

docs/internal/validator-e2e-evidence.md reports an end-to-end run against real dbt projects rather than fixtures. The recommendation from it is ALTIMATE_VALIDATORS_SHADOW=1 only, and it rests on two independent findings.

1. Five reproducible false positives, since fixed — and the class of bug matters more than the count. Across 38 known-good states the gates fired five times, every one on ordinary dbt practice rather than on a defect: editing an ephemeral model, disabling a model on purpose, touching a file seconds after a green build, a nested {% if %} inside a target.type guard, and a dialect function name inside a string literal. In enforce mode each costs a session a synthetic retry turn, and in two of the five the fix hint asked for something impossible. All five now have regression tests asserting no firing on the known-good state, and the evidence run also found that dbt-build-green returned green having checked nothing when the session's last dbt command was a dbt test (which overwrites run_results.json with test nodes only). That is fixed too — coverage now also reads the model DDL under <target>/run/, which a test invocation does not touch.

Fixing the observed false positives does not establish that no others exist. Shadow mode is how you find out, on real traffic, without spending anyone's retry budget.

2. There is no conversion evidence in either direction. The planned A/B was 10 tasks x 2 arms x 2 rollouts. Three sessions completed: N = 1 paired task plus one unpaired run, terminated for machine capacity. The single pair needed no retry, so there was nothing to convert. Nothing here shows these gates turn a failure into a pass, and nothing here shows they do not — the honest word is untested, not disproven. What a properly powered run would need (three arms including shadow, a task set where each gate is reachable, a pre-grade workspace snapshot) is specified in the evidence document.

Cost. ~2-10 ms per dispatch on a small project, but ~1-3.5 s on a 2 000-model project even when the session touched no models, because each validator walks the tree independently. That cost is paid on every dispatch, and consolidating the walks is recorded as a follow-up rather than done here.

ALTIMATE_VALIDATORS_ENABLED=1 is not the same decision as "enable these five." That flag activates all seven registered validators, including the two pre-existing ones. Probed against a green, complete project, dbt-schema-verify and dbt-tests-pass both returned ok:false with zero actual mismatches and zero actual test failures — every one was a subprocess that did not return a parseable result, and both treat "could not verify" as "blocks". They also cost 11-14 s each. None of that is this PR's doing, but it is what switching the lane on today would actually do.

Suggested sequence: shadow these five and measure the real fire rate; then run the three-arm A/B on isolated infrastructure; then decide on enforcement from those numbers.

How did you verify your code works?

• Validator suites: bun test test/altimate/validators/ -> 574 pass, 132 skip, 0 fail. That includes regression tests for each observed false positive, for the test-overwrite blind spot and its discriminating converse, and for the shared task-parsing and SQL-scrubbing utilities.
• Full suite: bun test test/session/ test/altimate/ -> 5039 pass, 649 skip, 2 fail. Both failures are pre-existing and were confirmed by re-running them on a detached origin/main checkout: a 5 s timeout in test/session/prompt.test.ts and a PostgreSQL driver E2E that requires a local database.
• bun run typecheck clean; marker guard (--markers --base main --strict) clean.
• Negative control: a live session in a non-dbt TypeScript repo with the lane and the artifact opt-in both forced on. Zero validators executed.

Not verified, stated plainly:

• No conversion measurement. See above — N = 1, inconclusive.
• Fixing the five observed false positives does not bound the false-positive rate. The 38 known-good states are a sample, not a proof.
• The lane's dispatch hook is still skipped when a step ends in compaction, so in compaction-heavy sessions these gates fire rarely. Pre-existing, out of scope here, noted in the engine-split document.
• Per-dispatch cost on large projects has not been optimised; each validator still walks the project tree independently.

Deferred and declined review findings are recorded with rationale in .github/meta/harness-review-followups.md.

Repo note: script/upstream/analyze.ts fails out of the box in a fresh worktree with Cannot find package 'minimatch' (no longer a transitive dep since glob@13). Worked around transiently to run the marker check; worth fixing separately.

Screenshots / recordings

N/A - no user-visible surface; these run inside the completion-gate lane.

Checklist

• Tests added for new behavior
• Typecheck passes
• Marker guard passes
• No changes to default behavior (lane remains opt-in via env)

---

Note

Medium Risk
These gates can block session completion and inject synthetic retries from filesystem/task parsing heuristics; mitigations are opt-in/shadow mode and conservative skips, but documented gaps (prose mining, bare node names, mutable task files) remain.

Overview
Adds five opt-in completion validators to the altimate validator lane and registers them before the existing schema-verify / tests-pass checks so “nothing built” and “build not green” run first.

dbt-nothing-built blocks finish when a task (or ALTIMATE_VALIDATORS_REQUIRE_ARTIFACTS) names deliverables but the session authored nothing matching those names and has no fresh successful build evidence—including treating existing on-disk deliverables as satisfied so stale task docs don’t trap unrelated work.

dbt-build-green ties session-edited models to fresh build evidence: rejects stale or non-executing run_results.json (e.g. dbt compile), falls back to <target>/run/ DDL when tests overwrite artifacts, exempts ephemeral/disabled models, and emits finer verdicts (fresh-build, build-unproven, nothing-verified, etc.) so shadow telemetry doesn’t count unverified passes as green.

dbt-deliverable-names compares literal task contract names to produced model/seed/snapshot inventory (workspace + dbt roots for required files). dbt-incremental-config and dbt-dialect-guard lint touched models for incremental contradictions and unguarded warehouse-specific calls (dialect guard only when the project already uses target.type guards or env opt-in).

Shared behaviour implied by the new modules includes resolveDbtSourcePaths, split run-results predicates for model coverage vs deliverable production, sanitizeForPrompt on retry text, and a 60s post-build mtime grace window.

Also adds internal docs: deferred review inventory (.github/meta/harness-review-followups.md), engine-vs-lane split assessment, and a superseded E2E evidence report with rollout guidance (shadow first).

^{Reviewed by Cursor Bugbot for commit dd858ef. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Summary by cubic

Closes #1174 by adding five deterministic completion gates to the altimate validator lane and closing its zero-write blind spot — sessions that declared done with nothing built, stale builds, missing deliverables, contradictory incremental configs, or unguarded warehouse-specific SQL. The gates are answer-free, off by default, and conservative, skipping rather than failing on ambiguous evidence. The second review sweep fixed silent under-firing gates, blocking false positives, and artifact miscounting, all locked down with fixture-based regression tests.

The gates

• dbt-nothing-built blocks when the session produced nothing and the task literally names deliverables (or ALTIMATE_VALIDATORS_REQUIRE_ARTIFACTS=1 is set); a named deliverable already on disk satisfies it. Only models, seeds, and snapshots contribute names, successful seed/snapshot artifacts count as produced, and data/ is no longer treated as a default seed path.
• dbt-build-green blocks edited models not covered by a fresh successful build. Known-non-executing commands (dbt compile) are rejected, unrecognized commands trusted, and seed/snapshot/clone artifacts fall through to build-unproven DDL-only coverage. Ephemeral/disabled exemptions stay per-axis and can't be contradicted by source config; dead {% if false %} / {% raw %} arms no longer strip live config; staleness dates from build completion rather than model DDL.
• dbt-deliverable-names diffs literal task deliverable names against project model/seed/snapshot names; dbt-incremental-config flags merge/delete+insert without unique_key, missing idempotency guards, and non-determinism in the incremental arm; dbt-dialect-guard flags unguarded warehouse functions in projects already using target.type guards.
• Review sweeps closed parser edge cases across all gates: task-file dedup by file identity, requirement parsing starting after the verb, unique_key = None as absent, and/or predicates only when no WHERE-like clause exists, idempotency demands read across every task file, column-0 config keys so vars: can't shadow them, Jinja modulo-tolerant guards, identifier-bounded is_incremental() matches, and repository text sanitized before entering synthetic instruction turns.

Evidence and rollout

• The E2E document measured a pre-review commit and carries a superseding banner; A/B conversion impact remains unmeasured at N=1.
• All gates are off by default; run ALTIMATE_VALIDATORS_SHADOW=1 first — shadow suppresses only the retry, not the validator or warehouse cost.
• Dispatch is still skipped when a step ends in compaction; pre-existing.
• docs/internal/deterministic-checks-engine-split.md scopes engine work: dbt-incremental-config duplicates the engine's dbt_config_lint and should be rewired; only dbt-dialect-guard's function list warrants reconciling with L033. The dialect-guard branch-semantics gap and script/upstream/analyze.ts's missing minimatch dep remain recorded follow-ups.

^{Written for commit dd858ef. Summary will update on new commits.}

Summary by CodeRabbit

• New Features

  • Added dbt completion checks for fresh, successful builds of edited models.
  • Added checks confirming required models and files were delivered.
  • Added validation for warehouse-specific SQL usage and incremental model configuration.
  • Added safeguards when no dbt artifacts are produced after changes.
  • Validators now run in a defined order and provide actionable failure guidance.

• Bug Fixes

  • Improved SQL and Jinja parsing to reduce false positives.
  • Improved handling of versioned, exempt, and vendored dbt models.

• Documentation

  • Added guidance and evidence for evaluating deterministic validator behavior.