stash eql verify: assert the installed EQL surface is complete

[coderdan]

Fixes #890.

The gap

A partial EQL install — domains present, some of their comparison functions or operators absent — reported success at install time and failed at query time on a specific predicate (weight >= x). Nothing detected it: isInstalled() is a presence test (do the two schemas exist), and eql validate checks the columns an application declared, not whether the installed operator surface is internally coherent.

stash eql verify

Asserts the installed EQL surface is complete and coherent, independent of any application schema:

• The manifest is the bundle itself. parseExpectedSurface() parses the pinned @cipherstash/eql install SQL into everything it creates unconditionally — 95 domains, 2 composite types, 3,266 function/aggregate overloads, 3,025 operators (identity = name + operand types), 1 cast. A bundle bump updates the expectation automatically; there is no hand-maintained list to drift. Dollar-quoted bodies are stripped first, which is also what keeps the bundle's DO-block conditionals out of the unconditional set.
• Read-only catalog diff. A handful of pg_catalog queries; format_type normalisation makes the catalog's spellings (_text, int8) meet the bundle's (text[], public.eql_v3_bigint). Runs in under a second.
• Expected absence ≠ damage. The bundle's two conditional halves are modelled explicitly: the ORE operator class absent with all 20 _ore domains carrying the eql_ore_unavailable poison CHECK is the supported managed-Postgres configuration and reads as info. The opclass absent with an incomplete fallback — or present with leftover poison — is damage.
• Per-domain report. Damage is grouped by the domain it concerns (eql_v3_double_ord: Operator \>= (…)` is missing), exits 1. --jsonemits the structured report with astatus discriminator (complete/incomplete/not-installed/version-mismatch`) for agents.
• Version mismatch is handled, not noise. When the installed EQL differs from the pinned bundle, the object-level diff is skipped (the pinned bundle is the wrong manifest to compare against) and the command suggests eql upgrade.

stash eql install now runs the same check before declaring success, so "install succeeded" means the full query-time surface is present, not just that the SQL committed.

Example output (against a deliberately broken install)

◇  EQL surface ──────────────────────────────────╮
│  installed version   missing                   │
│  pinned bundle       3.0.4                     │
│  domains             95/95                     │
│  functions           3265/3266  <- incomplete  │
│  operators           3024/3025  <- incomplete  │
│  ORE operator class  present                   │
├────────────────────────────────────────────────╯
■  install-wide:
│    - `eql_v3.version()` is missing or failed — the bundle always installs it.
■  eql_v3_double_ord:
│    - Operator `>= (public.eql_v3_double_ord, public.eql_v3_double_ord)` is missing.
└  The EQL install is incomplete — see the damage above.   (exit 1)

Coverage

• Unit (src/installer/__tests__/verify.test.ts, 18 tests): parser assertions against the real pinned bundle — including the issue's exact predicate operator, the quoted-name (eql_v3_internal."-") handling, and that the conditional ore_domain_unavailable is excluded — plus every differ classification (fallback vs incoherent ORE states, missing overloads, version mismatch, not-installed).
• Live Postgres (verify.live.test.ts, gated on STASH_TEST_DATABASE_URL): installs the real bundle, asserts the surface reads complete with exact full counts (proving the bundle/catalog spelling normalisation for every one of the 3,025 operators), then drops an operator and version() and asserts the damage is named and attributed. This is the check no fake can provide.
• E2E: eql verify added to the smoke command list and --help coverage.

Also: registry/manifest entry, help banner, skills/stash-cli/SKILL.md (new eql verify section + install self-verify note), and a stash minor changeset.

Notes for review

• The issue's caveat stands: the reported eql_v3_double_ord case was not reproduced (likely the rolled-back install it was fighting), but the class of failure is now detected whatever the cause.
• The verify tail in eql install treats a verification error (connection dropped, etc.) as a warning, not an install failure — the install itself committed.

https://claude.ai/code/session_01AwM5Cm5ddasXozb6stxPR1

[changeset-bot]

🦋 Changeset detected

Latest commit: 4bd720f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages

Name	Type
stash	Minor
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/stack	Minor
@cipherstash/stack-drizzle	Minor
@cipherstash/stack-supabase	Minor
@cipherstash/stack-prisma	Minor
@cipherstash/wizard	Minor
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch
@cipherstash/prisma-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderdan]

Review performed by GPT-5.6-sol. Five inline findings were posted on the install-time verification and catalog surface checks.

[coderdan]

Code review — 10 findings

Review performed by Claude Fable 5 (multi-agent pass: 8 finder angles, cross-confirmation, adversarial verification). One candidate was refuted during verification: pg's Client.end() promise has no reject path, so the unguarded finally await around it is harmless. Everything below survived verification.

The three most severe findings form a pattern: the new guarantee doesn't hold in exactly the places it's most needed.

---

1. Version-mismatch report returns ok: true / exit 0 — the headline scenario reads as success

packages/cli/src/installer/verify.ts:467

A damaged install made by an older CLI pinning an older EQL — the command's headline scenario — produces status: 'version-mismatch' with ok: true and exit 0, with every object-level check skipped. A CI gate stash eql verify || fail passes on the damaged database, and an agent consuming --json rules out install damage. Compounding: install.ts:196 branches on report.ok, so the same path announces "EQL surface verified — install is complete." for a verification that never ran.

ok conflates "checked, fine" with "could not check"; it's hand-set at three construction sites and interpreted differently by its two consumers (install.ts checks ok, verifyCommand switches on status).

2. The already-installed early return skips verification — a plain re-run after a failed verify exits 0

packages/cli/src/commands/db/install.ts:167

Install commits, verify reports incomplete, command exits 1. The user re-runs stash eql install without --force — the natural retry. isInstalled() is a schemas-exist presence test, so the CLI prints "EQL is already installed... Nothing to do." and exits 0 on the very database it just called damaged (and the skipped cs_migrations tracking schema is never installed on this path).

This contradicts both the changeset ("runs the same check automatically before declaring success") and skills/stash-cli/SKILL.md:469's new claim that "install succeeded now means the full query-time surface is present" — which is also false for the catch branch at install.ts:212 that warns-and-continues on a verification error. Either verify on the already-installed path too, or soften the skill claim.

3. The parser↔catalog spelling seam now hard-gates every install, and its only real test runs in no CI workflow

packages/cli/src/installer/verify.ts:194

verify.live.test.ts is the only guard on the regex-parsed manifest, and nothing in .github/ sets STASH_TEST_DATABASE_URL. The unit suite is tautological on spelling (completeInstall builds the installed surface from the parser's own output).

A routine @cipherstash/eql bump that writes RIGHTARG = int (catalog: integer), a double precision operand (the LEFTARG regex captures one token), a DEFAULT arg (argType folds it into the type), or reformats ); makes expected keys stop meeting the OPERATORS_SQL/format_type spellings — and every stash eql install on that CLI release exits 1 with phantom damage, with green CI. The opposite drift fails silently: the column-0 ^CREATE anchors drop any construct that moves inside a DO block (the bundle already did exactly this to CREATE DOMAIN, forcing the raw-text special case at line 161), and the loose pins (>2000 operators) won't notice a family vanishing — verify then reports "complete" while missing the #890 class it exists to catch.

Run the live suite in CI (the integration jobs already have Docker), or better, ship the object manifest from the @cipherstash/eql build instead of regex-parsing its SQL.

---

4. Bare catch {} on the version() probe conflates "missing" with "call failed", then bypasses the mismatch guard

packages/cli/src/installer/verify.ts:395

EXECUTE denied or statement_timeout on SELECT eql_v3.version() looks identical to the function being absent, and a null installedVersion also bypasses the version-mismatch guard (!== null at line 467) — both fall through to a full phantom object diff and exit 1. A restricted application role lacking EXECUTE on eql_v3.version() — the exact installed-but-ungranted state install.ts:168-171 documents — gets "damage" on a healthy install. An older v3 install predating version() (acknowledged at installer/index.ts:313) gets hundreds of phantom findings and the wrong remediation (--force reinstall vs eql upgrade). Distinguish undefined_function (42883) from other SQLSTATEs in the catch.

5. ORE poison-constraint count is database-wide and exact-match — one stray constraint flips a healthy install to damage

packages/cli/src/installer/verify.ts:325

ORE_STATE_SQL counts every domain constraint named eql_ore_unavailable database-wide (contypid <> 0, no schema filter, no join to the expected ORE domains), and the classifier requires exactly 0 or exactly 20. A superuser install plus one same-named CHECK on any user domain → incoherent-poisoned → install exits 1; on a fallback install an extra same-named constraint makes poisonedDomains ≠ 20 → incoherent-unpoisoned damage. The bundle poisons exactly the 20 *_ore domains by fixed name — join the count against expected.oreDomains (typnamespace + typname) instead of name-only.

6. --database-url with a missing value is silently booleanised — verify judges a different database than the user targeted

packages/cli/src/bin/main.ts:276

parseArgs sets flags['database-url'] = true when the next token starts with - (main.ts:176-181); the verify dispatch passes only values['database-url'] and never checks the flags entry, so verify falls back to env/config resolution with no error — reporting "complete" for the env-resolved dev database while the intended prod URL was never contacted. stash env guards exactly this (main.ts:600, nameMissingValue: flags.name === true); verify (and the pre-existing eql preflight) lack the guard — worth fixing for the flag whose whole job is targeting.

7. eql verify inverts eql install's --database-url precedence — config overrides the flag

packages/cli/src/commands/eql/verify.ts:89

A stash.config.ts found anywhere up the directory tree (findConfigFile walks to root) overrides the flag, and in --json mode the redirect is only a stderr warning with no report field naming the database actually checked. The skill teaches stash eql install --database-url 'postgres://prod' as a one-shot that deliberately skips config loading; the paired verify run from a subdirectory of a repo whose config sets a dev literal verifies dev, prints "complete", exits 0 while prod stays damaged — and an agent reading --json stdout cannot detect the redirect. Either match install's flag-wins semantics or add the resolved target to the JSON report.

8. Function check compares row counts per name, never signatures

packages/cli/src/installer/verify.ts:541

present < expectedOverloads means a surplus same-name function masks a genuinely missing current overload: expected 5, present = 4 current + 1 stale/hand-created = 5 → no damage reported, yet the missing signature still fails at query time — the false-negative class #890 targets. Not produced by supported install paths (the bundle opens with DROP SCHEMA CASCADE), but constructible; the parser already computes per-signature sets and pg_get_function_identity_arguments could diff them exactly.

9. resolveVerifyDatabaseUrl is a near-verbatim copy of preflight's resolver; two differently-spelled exit predicates

packages/cli/src/commands/eql/verify.ts:85

Identical to resolvePreflightDatabaseUrl (packages/cli/src/commands/db/preflight.ts:20-48) except one word in the warning string, and the file duplicates the whole verify flow across json/interactive branches with two exit predicates (!report.ok vs a status switch). The next fix to config-precedence/json-quieting lands in one copy and the two commands diverge; a new status value handled in one predicate but not the other gives --json a different exit code than interactive mode for the same database state. Extract the resolver (parameterised on the verb) and use one shared exit predicate.

10. skills/stash-indexing not updated — AGENTS.md's own drift table maps this change to it

skills/stash-indexing/SKILL.md:133

The skill still walks users through hand-verifying the exact incoherent ORE state eql verify now detects and names, and its related-commands list omits eql verify. A customer following the shipped skill after a broken ORE install hand-runs catalog SQL, never told the command their CLI now runs at install time already diagnoses it — and nothing in CI catches this (the mechanical manifest check covers only stash-cli).

[coderdan]

All 15 findings from the two review passes are addressed in 286cca7. Summary, keyed to the Fable review's numbering (the five GPT-5.6-sol inline findings overlap with findings 2, 5, and 8, plus two of their own — noted inline):

Fixed as reported

• Finding 1 — version-mismatch read as success — ok now means exactly "checked and complete". A version mismatch reports ok: false and exits 1; the install tail branches on the same field. Unit test pins it.
• Finding 2 / inline (install.ts, already-installed path skips verification) — the early exit now runs the same surface check, so a plain re-run over a damaged database exits 1 with the damage report instead of "Nothing to do." Verified end-to-end against a deliberately broken database. The skill claim was also reworded to match the catch branch (a verification error warns and points at eql verify rather than failing a committed install).
• Finding 3 — live suite runs in no CI workflow — tests.yml's run-tests job already carries a Postgres service; STASH_TEST_DATABASE_URL now points the CLI's .live.test.ts suites at it, so the parser↔catalogue spelling seam is guarded on every PR, both Node matrix legs, fork PRs included (no credentials needed). A TYPE_ALIASES map additionally absorbs non-canonical spellings (int8, timestamptz, …) a future bundle might introduce.
• Finding 4 — bare catch on the version() probe — SQLSTATE 42883 (genuinely missing → damage) is now distinguished from everything else (EXECUTE denied, timeout → verification failure, reported as such); no more phantom full diff on a healthy-but-restricted install.
• Finding 5 / inline (poison count database-wide) — the count now joins contypid against the expected ORE domains by qualified name.
• Finding 8 / inline (count-level function check) — replaced with type-only signature comparison: the parser emits per-name signature sets, the catalogue side renders proargtypes through format_type() under a pinned empty search_path (deterministic qualification; composite arrays come out ore_block_256_term[], not _ore_block_256_term). A stale same-name function can no longer mask a missing overload — a unit test constructs exactly the 4-current-plus-1-impostor case.
• Inline (pgcrypto presence-only) — now checks extnamespace against the same supported-schema list the preflight uses (shared export).
• Finding 6 — booleanised --database-url — eql verify and eql preflight reject a value-less flag up front (JSON envelope in --json mode), same pattern as stash env's nameMissingValue.
• Finding 7 — config overrides the flag — eql verify --database-url is now a one-shot exactly like eql install's: it bypasses config loading, so the database named is the database judged. Registry and skill updated.
• Finding 9 — duplicated resolver / two exit predicates — one shared resolveDiagnosticDatabaseUrl (parameterised on flagWins + verb) serves both commands, and verifyCommand derives its exit from report.ok in both output modes.
• Finding 10 — stash-indexing drift — the hand-run ORE-state SQL walkthrough now leads with stash eql verify (keeping the index-opclass query for the one thing verify can't see: which opclass an existing index bound), and the reference list names the command.

One suggested fix amended, with evidence

• Inline (operator identity omits oprnamespace) — implemented as suggested first, and the live suite immediately failed: the bundle creates most operators unqualified, so they follow the install-time search_path — on any database with a "$user" schema named after the installing role (the live container is exactly this), they legitimately land there, not in public. Only the six ore_block_256 comparison operators are explicitly public.-qualified. A public-only scope therefore phantom-fails healthy installs. Operator identity stays schema-agnostic, with the reasoning documented at the parser and the query; the masking scenario it leaves open (a hand-built identical operator on EQL domains in another schema standing in for a dropped one) is not distinguishable from a functional operator by catalogue inspection anyway.

Full validation: 20 unit + 3 live tests green (the live DB is the $user-schema layout that falsified the namespace scope), 1311-test CLI suite green, 110 e2e green, biome clean, and the broken-database → plain-rerun-fails → --force-heals → verify-0 loop confirmed manually.

[tobyhede]

Review — 3 issues

Verified against 286cca78; each finding was adversarially re-checked before filing.

1. eql install hard-fails on a version mismatch, with contradictory advice

packages/cli/src/commands/db/install.ts:251

Two fixes from the earlier passes combine badly. diffSurface now returns ok: false for status: 'version-mismatch' (verify.ts:583), and verifySurfaceOrExit was added to the already-installed path — but it branches on report.ok alone, so skew and real damage take the same branch.

Against a DB carrying 3.0.2 with a CLI pinning 3.0.4, stash eql install prints three inconsistent things and exits 1:

• headline — "The installed EQL surface is incomplete." (nothing was checked; counts is null)
• finding — "…Run stash eql upgrade, then verify again." (severity: warning)
• error — "Re-run with stash eql install --force"

Pre-PR this path was p.outro('Nothing to do.'), exit 0. Two consequences: idempotent provisioning scripts break, and stash init dies mid-run — its direct-install route calls installCommand (init/steps/install-eql.ts), and process.exit(1) escapes the surrounding try/catch.

Suggested: branch on report.status === 'version-mismatch' before the ok check and return, leaving eql verify's strict gate as-is.

Worth deciding which remedy is correct first: stash eql upgrade is installer.install(), the same DDL as --force, and it calls loadStashConfig, which exits 1 with no stash.config.ts. On a one-shot --database-url run, --force is the only remedy that actually works.

Nothing covers this path — there are no unit tests for installCommand.

2. Live suites race on one database in CI

.github/workflows/tests.yml:286

STASH_TEST_DATABASE_URL on the whole pnpm run test step enables four live suites at once: verify.live (new), plus guarded-grants.live, preflight.live, applied.live. packages/cli/vitest.config.ts sets no pool, fileParallelism, or maxWorkers, so vitest 3.2.7 runs them in parallel forks against one database and one eql_v3 / eql_v3_internal pair.

verify.live.test.ts's beforeAll installs the full bundle, which opens with DROP SCHEMA IF EXISTS eql_v3 CASCADE. guarded-grants.live.test.ts grants USAGE, CREATE on those schemas and asserts on pg_default_acl rows scoped to them. Recreated schemas get new OIDs and zero ACLs, so its ALTER DEFAULT PRIVILEGES … IN SCHEMA eql_v3 as MEMBER fails on the USAGE it no longer holds — and the pg_has_role guard in grants.ts gates on membership only, so the error propagates rather than being caught.

These files already carry race-safety comments about each other ("Race-safe against the preflight live suite creating it concurrently"), but only for idempotent creates — not a destructive drop.

packages/migrate/vitest.config.ts already sets fileParallelism: false with this exact justification. Same fix here, or give the live suites their own step.

3. Valueless --database-url still unguarded on the destructive subcommands

packages/cli/src/bin/main.ts:281

rejectMissingDatabaseUrlValue is wired into preflight and verify only. parseArgs booleanises --database-url when the next token starts with -, leaving values['database-url'] undefined — indistinguishable from "not passed" — so resolveDatabaseUrl silently falls through to DATABASE_URL and prints nothing.

Against the built CLI:

$ DATABASE_URL='postgres://…:59999/envdb' stash eql install --database-url --force
■  Fatal error: Failed to connect to database: connect ECONNREFUSED 127.0.0.1:59999

The valid path prints "Using DATABASE_URL from --database-url flag"; that line is the only tell, and it is absent here. --force then skips the isInstalled() early exit and runs DROP SCHEMA … CASCADE with no confirmation, never naming the target database.

Pre-existing rather than introduced here — this PR narrows it for two commands. But one await rejectMissingDatabaseUrlValue(flags) at the top of dispatch() covers every subcommand and replaces both per-case calls. A valueless --database-url is always a typo.

Minor

verify.live.test.ts has no afterAll, so the dropped >= (public.eql_v3_double_ord, …) operator and eql_v3.version() outlive the file. Low impact — the sibling live suites all tolerate a missing install, CI Postgres is a per-job container, and the next run's beforeAll reinstall repairs it. Still a one-line fix worth taking.

[coderdan]

All four findings addressed in 4bd720f.

Finding 1 — version mismatch hard-failing the install. verifySurfaceOrExit now branches on status === 'version-mismatch' before the ok check: it warns with the skew and returns, so the already-installed path ends at "Nothing to do.", exit 0 — restoring the pre-verification idempotence that provisioning scripts and stash init's direct-install route rely on. Damage still exits 1, and eql verify's strict gate is untouched. On the remedy question: the mismatch finding now names both — stash eql upgrade, or stash eql install --force --database-url ... for a database without a stash.config.ts — since as you note, upgrade dies without a config on exactly the one-shot runs. The path is no longer uncovered: a new unit suite (install-verify-gate.test.ts) pins all four outcomes of verifySurfaceOrExit (complete, damage → exit 1, mismatch → no exit, verification error → warn and continue).

Finding 2 — live-suite races. Took the scoped version of the migrate fix: the CLI vitest config now defines two projects, unit (everything but *.live.test.ts, default parallelism) and live (only the live suites, fileParallelism: false). The four suites sharing the database run serially; the ~1300 unit tests don't pay for it. Confirmed against the local container with all four suites enabled — they run one file at a time and pass.

Finding 3 — valueless --database-url on destructive commands. Moved rejectMissingDatabaseUrlValue to the top of dispatch(), replacing both per-case calls, so every subcommand rejects it before any I/O. Reproduced your --database-url --force case against the built CLI: it now exits 1 with the "needs a value" error before the permission check ever starts, and the JSON envelope still comes out in --json mode. E2E-pinned in the smoke suite.

Minor — no afterAll. Added: the suite reinstalls the bundle after its tests, so the dropped operator and version() don't outlive the file into whatever the serialised live project runs next. Verified the database reads healthy after the suite completes.

Validation: 1315 unit tests (including the 4 new gate tests), the four serialised live suites green against the local container, 111 e2e, biome clean.