From 7eb689ea5983ff8d07a5a65c4c07eb398dcbff30 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 18 Aug 2026 22:26:05 +1000 Subject: [PATCH] Detect the ORE-unavailable case at install time (#891) The EQL bundle skips the ORE btree operator class when the installing role cannot create one, and poisons every `_ord_ore` domain with a loud-failure CHECK in its place. That is a supported configuration -- but nothing said so where the choice between `types.*Ord` and `types.*OrdOre` is actually made, so operators discovered the trade at query time. - `eql preflight` probes whether the role can create an operator class and reports it as a non-blocking row. Probed, not inferred from `superuser`: RDS and Aurora let their admin role create one while cloud-hosted Supabase does not, so `rolsuper` is not evidence either way. The probe attempts the DDL in a transaction it always rolls back, so preflight stays read-only, and reports `unknown` when it could not ask rather than guessing. - `eql install` names the consequence and the remedy on its own line. - `eql status` reports the ORE state, so the answer survives the install output. - The remedy now names a type that exists. The previous wording pointed at the `_ord_ope` domains; the bundle creates those, but `@cipherstash/stack` ships no `types.*OrdOpe` factory, so it named a column type no schema author could declare. - The state machine, the catalogue probe and the copy move into one module shared by preflight, install, status, verify and validate, so the five cannot drift into disagreeing about the same catalogue fact. Claude-Session: https://claude.ai/code/session_01AwM5Cm5ddasXozb6stxPR1 --- .changeset/ore-unavailable-at-install.md | 14 ++ .../scaffold/drizzle.generated.ts | 7 + .../scaffold/generic.generated.ts | 7 + .../commands/db/__tests__/preflight.test.ts | 35 ++++ packages/cli/src/commands/db/install.ts | 13 +- packages/cli/src/commands/db/preflight.ts | 19 +++ packages/cli/src/commands/db/status.ts | 37 ++++- packages/cli/src/commands/eql/validate.ts | 19 +-- packages/cli/src/commands/eql/verify.ts | 10 +- packages/cli/src/commands/init/utils.ts | 14 ++ .../cli/src/installer/__tests__/ore.test.ts | 150 ++++++++++++++++++ .../__tests__/preflight.live.test.ts | 36 +++++ .../installer/__tests__/verify.live.test.ts | 25 ++- packages/cli/src/installer/index.ts | 69 ++++++++ packages/cli/src/installer/ore.ts | 144 +++++++++++++++++ packages/cli/src/installer/verify.ts | 116 +++++++------- skills/stash-cli/SKILL.md | 6 +- 17 files changed, 632 insertions(+), 89 deletions(-) create mode 100644 .changeset/ore-unavailable-at-install.md create mode 100644 packages/cli/src/installer/__tests__/ore.test.ts create mode 100644 packages/cli/src/installer/ore.ts diff --git a/.changeset/ore-unavailable-at-install.md b/.changeset/ore-unavailable-at-install.md new file mode 100644 index 000000000..cd228f51d --- /dev/null +++ b/.changeset/ore-unavailable-at-install.md @@ -0,0 +1,14 @@ +--- +'stash': minor +--- + +Report the ORE-unavailable case once, at install time, instead of leaving it to surface as a failing predicate the first time a column is cast. + +The EQL bundle skips the ORE btree operator class when the installing role cannot create one and poisons every `_ord_ore` domain with a loud-failure CHECK in its place. That is a supported configuration — but nothing said so where the choice between `types.*Ord` and `types.*OrdOre` is actually made, so the trade was discovered at query time. + +- **`stash eql preflight` now probes whether the role can create an operator class** and reports it as a non-blocking `ORE operator class` row (`creatable` / `not creatable` / `unknown`; `canCreateOperatorClass` in `--json`). It is *probed*, not inferred from `superuser`: `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, but AWS RDS and Aurora let their admin role create one while cloud-hosted Supabase does not, so `rolsuper` is not evidence either way. The probe attempts the DDL inside a transaction it always rolls back, leaving preflight read-only; a probe that could not ask reports `unknown` rather than guessing. +- **`stash eql install` names the consequence and the remedy** on its own line when the fallback was installed, rather than as a parenthetical on the "verified" line. +- **`stash eql status` reports the ORE state** on a v3 install, so the answer survives past the install output. +- **The remedy now names a type that exists.** The previous wording pointed at the `_ord_ope` domains; the bundle creates those, but `@cipherstash/stack` ships no `types.*OrdOpe` factory, so it named a column type no schema author could declare. Every command now says `types.*Ord` (`public.eql_v3_*_ord`), which is the same CLLW-OPE ordering and has a factory behind it. +- The ORE state machine, the catalogue probe, and this copy now live in one module shared by `eql preflight`, `eql install`, `eql status`, `eql verify`, and `eql validate`, so the five commands cannot drift into disagreeing about the same catalogue fact. +- The scaffolded encryption client's type cheat-sheet now says why ordered columns should be `*Ord` rather than `*OrdOre`. diff --git a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts index c423ccb83..8bec63e43 100644 --- a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts +++ b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts @@ -23,6 +23,13 @@ * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the `*Ord` factories above, not `*OrdOre`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * `_ord_ore` domain, and each write to one fails a CHECK. `*Ord` orders and + * indexes on any role. `stash eql status` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): diff --git a/packages/cli/__fixtures__/scaffold/generic.generated.ts b/packages/cli/__fixtures__/scaffold/generic.generated.ts index 80981180c..3692db5bb 100644 --- a/packages/cli/__fixtures__/scaffold/generic.generated.ts +++ b/packages/cli/__fixtures__/scaffold/generic.generated.ts @@ -23,6 +23,13 @@ * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the `*Ord` factories above, not `*OrdOre`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * `_ord_ore` domain, and each write to one fails a CHECK. `*Ord` orders and + * indexes on any role. `stash eql status` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): diff --git a/packages/cli/src/commands/db/__tests__/preflight.test.ts b/packages/cli/src/commands/db/__tests__/preflight.test.ts index 5e1eb23a1..dbdf78855 100644 --- a/packages/cli/src/commands/db/__tests__/preflight.test.ts +++ b/packages/cli/src/commands/db/__tests__/preflight.test.ts @@ -14,6 +14,7 @@ const CAPABLE: PreflightResult = { eqlV3InternalSchemaPresent: false, canDropEqlV3Schema: null, canDropEqlV3InternalSchema: null, + canCreateOperatorClass: true, missing: [], ok: true, } @@ -78,6 +79,40 @@ describe('renderPreflightReport', () => { expect(report).toContain('<- blocks: not on the EQL search_path') }) + // #891: the ORE trade is reported so it is known before a schema is + // written. It must never read as a blocker — the bundle's fallback makes an + // install without the operator class a complete install. + it('names the ORE consequence for a role that cannot create an operator class', () => { + const report = renderPreflightReport({ + ...CAPABLE, + currentUser: 'sandbox_exec', + isSuperuser: false, + canCreateOperatorClass: false, + }) + expect(report).toMatch(/ORE operator class\s+not creatable/) + expect(report).toContain('<- skips: ORE opclass') + expect(report).toContain('`types.*Ord`, not `types.*OrdOre`') + expect(report).not.toContain('<- blocks') + }) + + it('leaves the ORE row unannotated for a role that can create one', () => { + const report = renderPreflightReport(CAPABLE) + expect(report).toMatch(/ORE operator class\s+creatable/) + expect(report).not.toContain('<- skips: ORE opclass') + }) + + // A probe that could not ask the question must not be rendered as either + // answer — "unknown" is the honest third state. + it('renders an unanswerable ORE probe as unknown, not as no', () => { + const report = renderPreflightReport({ + ...CAPABLE, + canCreateOperatorClass: null, + }) + expect(report).toMatch(/ORE operator class\s+unknown/) + expect(report).toContain('`stash eql verify` reports it after install') + expect(report).not.toContain('not creatable') + }) + it('adds the drop-ownership row only when an EQL schema exists', () => { expect(renderPreflightReport(CAPABLE)).not.toContain('can drop EQL schemas') const blocked = renderPreflightReport({ diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index 5b401cb6a..cd8569ccc 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -4,6 +4,7 @@ import { resolveDatabaseUrl } from '@/config/database-url.js' import { findConfigFile, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' import { EQLInstaller } from '@/installer/index.js' +import { describeOreState } from '@/installer/ore.js' import { type VerifyReport, verifyEqlSurface } from '@/installer/verify.js' import { messages } from '@/messages.js' import { detectPackageManager, runnerCommand } from '../init/utils.js' @@ -265,11 +266,13 @@ export async function verifySurfaceOrExit( return } if (report.ok) { - s.stop( - report.ore?.state === 'fallback' - ? 'EQL surface verified (ORE operator class skipped — expected for this role; use the `_ord_ope` ordering domains).' - : 'EQL surface verified — install is complete.', - ) + s.stop('EQL surface verified — install is complete.') + // Reported once, here, rather than left to surface as a failing predicate + // the first time someone casts a column (#891). `eql status` and + // `eql verify` say the same thing later, so the answer is recoverable. + if (report.ore?.state === 'fallback') { + p.log.info(describeOreState('fallback').message) + } return } s.stop('The installed EQL surface is incomplete.') diff --git a/packages/cli/src/commands/db/preflight.ts b/packages/cli/src/commands/db/preflight.ts index d5b8543bd..b60c53397 100644 --- a/packages/cli/src/commands/db/preflight.ts +++ b/packages/cli/src/commands/db/preflight.ts @@ -2,6 +2,7 @@ import * as p from '@clack/prompts' import { emitJsonError, emitJsonEvent } from '@/commands/auth/events.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { EQLInstaller, type PreflightResult } from '@/installer/index.js' +import { describeOreCreatable, ORE_FALLBACK_REMEDY } from '@/installer/ore.js' import { resolveDiagnosticDatabaseUrl } from './resolve-diagnostic-url.js' /** See {@link resolveDiagnosticDatabaseUrl} — preflight keeps config-wins. */ @@ -90,6 +91,14 @@ export async function preflightCommand( 'Not a member of `postgres`: in Supabase mode the optional `ALTER DEFAULT PRIVILEGES FOR ROLE postgres` statements are skipped. The install is complete without them — they only cover EQL objects created outside stash tooling, and stash re-grants every object on each install/upgrade.', ) } + // Said here so the ordering trade is known before a schema is written. The + // same sentence is printed by `eql install` and `eql status` against the + // installed state, so the answer survives past this command's output (#891). + if (result.canCreateOperatorClass === false) { + p.log.info( + `This role cannot create an operator class, so \`eql install\` will skip the ORE one and install its loud-failure fallback instead. That is a complete install. ${ORE_FALLBACK_REMEDY}`, + ) + } p.outro('This role can install EQL.') } @@ -140,6 +149,16 @@ export function renderPreflightReport(result: PreflightResult): string { ? '<- blocks: CREATE EXTENSION pgcrypto' : undefined, ], + // Never annotated as a blocker: the bundle skips the class and installs + // its loud-failure fallback, which is a complete install (#891). + // Same label `eql verify` uses for the installed state, so the row an + // operator reads before the install and the row they read after it are + // recognisably about the same thing. The value carries the tense. + [ + 'ORE operator class', + describeOreCreatable(result.canCreateOperatorClass).value, + describeOreCreatable(result.canCreateOperatorClass).annotation, + ], ['eql_v3 schema', result.eqlV3SchemaPresent ? 'present' : 'absent'], [ 'eql_v3_internal', diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index 34ce0aece..b10c26a64 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -3,6 +3,8 @@ import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' import { EQLInstaller } from '@/installer/index.js' +import { describeOreState } from '@/installer/ore.js' +import { readOreState } from '@/installer/verify.js' export async function statusCommand(options: { databaseUrl?: string } = {}) { const pm = detectPackageManager() @@ -93,7 +95,40 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { ) } - // 3. Encrypt configuration. + // 3. The ORE half of the install. + // + // Reported here so the trade an operator was told about at install time is + // recoverable afterwards, without re-reading scrollback (#891). Only when + // v3 is installed: the state is a property of the v3 bundle's conditional + // half, and reads as 'fallback' on a database that has no EQL at all. + if (installedV3) { + s.start('Checking ORE operator class...') + const oreClient = createPgClient(config.databaseUrl) + try { + await oreClient.connect() + const ore = await readOreState(oreClient) + s.stop('ORE state checked.') + const described = describeOreState(ore.state) + if (described.severity === 'damage') { + p.log.error(described.message) + } else { + p.log.info(described.message) + } + } catch (error) { + // Advisory, not a gate: a status run that could not read one row should + // still report everything else it read. + s.stop('ORE state check failed.') + p.log.warn( + `Could not determine the ORE operator class state: ${ + error instanceof Error ? error.message : String(error) + }`, + ) + } finally { + await oreClient.end().catch(() => {}) + } + } + + // 4. Encrypt configuration. // // `public.eql_v2_configuration` is a v2 + CipherStash Proxy artifact: the v2 // install creates it and Proxy reads it. EQL v3 has no configuration table — diff --git a/packages/cli/src/commands/eql/validate.ts b/packages/cli/src/commands/eql/validate.ts index f55691fdd..c9ecdbbfc 100644 --- a/packages/cli/src/commands/eql/validate.ts +++ b/packages/cli/src/commands/eql/validate.ts @@ -5,6 +5,7 @@ import { fetchPhysicalColumns } from '@/commands/encrypt/lib/db-readers.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadEncryptSchemas, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' +import { ORE_OPCLASS_PRESENT_EXPR } from '@/installer/ore.js' // --------------------------------------------------------------------------- // The vocabulary @@ -604,21 +605,11 @@ function identifiersIn(args: string): string[] { } /** - * Whether the ORE btree operator class exists. Mirrors the EQL bundle's own - * fallback test (`ore_fallback.sql`), with one deliberate difference: - * `to_regtype` returns NULL where the bundle's `::regtype` cast raises, so this - * degrades to `false` on a database with no EQL installed instead of throwing. - * That is why the not-installed case is detected and reported separately. + * Whether the ORE btree operator class exists. The expression is shared with + * `eql verify` and `eql status` (`installer/ore.ts`) so the three commands + * cannot drift into disagreeing about the same catalogue fact. */ -const ORE_AVAILABLE_SQL = ` - SELECT EXISTS ( - SELECT 1 - FROM pg_catalog.pg_opclass c - JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod - WHERE am.amname = 'btree' - AND c.opcdefault - AND c.opcintype = to_regtype('eql_v3_internal.ore_block_256') - ) AS ore_available` +const ORE_AVAILABLE_SQL = `SELECT ${ORE_OPCLASS_PRESENT_EXPR} AS ore_available` const EQL_INSTALLED_SQL = ` SELECT EXISTS ( diff --git a/packages/cli/src/commands/eql/verify.ts b/packages/cli/src/commands/eql/verify.ts index d323bd75f..f17cfe0da 100644 --- a/packages/cli/src/commands/eql/verify.ts +++ b/packages/cli/src/commands/eql/verify.ts @@ -2,6 +2,7 @@ import * as p from '@clack/prompts' import { emitJsonError, emitJsonEvent } from '@/commands/auth/events.js' import { resolveDiagnosticDatabaseUrl } from '@/commands/db/resolve-diagnostic-url.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { describeOreState } from '@/installer/ore.js' import type { SurfaceFinding, VerifyReport } from '@/installer/verify.js' import { verifyEqlSurface } from '@/installer/verify.js' @@ -144,14 +145,7 @@ export function renderSurfaceCounts(report: VerifyReport): string { ] }, ), - [ - 'ORE operator class', - ore.state === 'indexable' - ? 'present' - : ore.state === 'fallback' - ? 'skipped (expected on managed Postgres)' - : 'INCOHERENT', - ], + ['ORE operator class', describeOreState(ore.state).value], ] const labelWidth = Math.max(...rows.map(([label]) => label.length)) return rows diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index 95458c123..58a63a48b 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -400,6 +400,13 @@ const DRIZZLE_PLACEHOLDER = `/** * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the \`*Ord\` factories above, not \`*OrdOre\`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * \`_ord_ore\` domain, and each write to one fails a CHECK. \`*Ord\` orders and + * indexes on any role. \`stash eql status\` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): @@ -468,6 +475,13 @@ const GENERIC_PLACEHOLDER = `/** * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the \`*Ord\` factories above, not \`*OrdOre\`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * \`_ord_ore\` domain, and each write to one fails a CHECK. \`*Ord\` orders and + * indexes on any role. \`stash eql status\` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): diff --git a/packages/cli/src/installer/__tests__/ore.test.ts b/packages/cli/src/installer/__tests__/ore.test.ts new file mode 100644 index 000000000..74112bc7f --- /dev/null +++ b/packages/cli/src/installer/__tests__/ore.test.ts @@ -0,0 +1,150 @@ +import { describe, expect, it } from 'vitest' +import { bundledExpectedSurface } from '@/installer/verify.js' +import { + classifyOreState, + describeOreCreatable, + describeOreState, + ORE_FALLBACK_REMEDY, + ORE_OPCLASS_PRESENT_EXPR, + type OreSurfaceState, +} from '../ore.js' + +/** + * The shared ORE model (#891). These assertions are about the *copy* as much + * as the logic: the whole point of the module is that `eql preflight`, + * `eql install`, `eql status` and `eql verify` say one thing about ORE, and + * that the thing they say names a remedy a schema author can actually type. + */ +describe('classifyOreState', () => { + const expectedPoisoned = 20 + + it('reads a privileged install as indexable', () => { + expect( + classifyOreState({ + opclassPresent: true, + poisonedDomains: 0, + expectedPoisoned, + }), + ).toBe('indexable') + }) + + it('reads the managed-Postgres skip with a full fallback as fallback', () => { + expect( + classifyOreState({ + opclassPresent: false, + poisonedDomains: expectedPoisoned, + expectedPoisoned, + }), + ).toBe('fallback') + }) + + it('reads a partial fallback as incoherent, not as the supported skip', () => { + expect( + classifyOreState({ + opclassPresent: false, + poisonedDomains: expectedPoisoned - 1, + expectedPoisoned, + }), + ).toBe('incoherent-unpoisoned') + }) + + it('reads leftover poison alongside a present opclass as incoherent', () => { + expect( + classifyOreState({ + opclassPresent: true, + poisonedDomains: 1, + expectedPoisoned, + }), + ).toBe('incoherent-poisoned') + }) +}) + +describe('describeOreState', () => { + it('treats both healthy states as info and both incoherent ones as damage', () => { + expect(describeOreState('indexable').severity).toBe('info') + expect(describeOreState('fallback').severity).toBe('info') + expect(describeOreState('incoherent-poisoned').severity).toBe('damage') + expect(describeOreState('incoherent-unpoisoned').severity).toBe('damage') + }) + + it('names the consequence and the remedy on the fallback path', () => { + const { message } = describeOreState('fallback') + // The consequence: not a failed install. + expect(message).toContain('not a failed install') + // The remedy, in the words a schema author types. + expect(message).toContain('types.*Ord') + expect(message).toContain(ORE_FALLBACK_REMEDY) + }) + + /** + * The bundle creates `public.eql_v3__ord_ope` domains, but + * `@cipherstash/stack` ships no `types.*OrdOpe` factory — so naming the + * `_ord_ope` domains as the remedy sends a schema author to a column type + * they cannot declare. This is the regression that guard exists for. + */ + it('does not point a schema author at the factory-less `_ord_ope` domains', () => { + const states: OreSurfaceState[] = [ + 'indexable', + 'fallback', + 'incoherent-poisoned', + 'incoherent-unpoisoned', + ] + for (const state of states) { + expect(describeOreState(state).message).not.toContain('_ord_ope') + expect(describeOreState(state).message).not.toContain('OrdOpe') + } + }) + + it('gives every state a short value for a report row', () => { + expect(describeOreState('indexable').value).toBe('present') + expect(describeOreState('fallback').value).toContain('skipped') + expect(describeOreState('incoherent-poisoned').value).toBe('INCOHERENT') + }) +}) + +describe('describeOreCreatable', () => { + it('annotates only the "no" answer, and never as a blocker', () => { + expect(describeOreCreatable(true)).toEqual({ value: 'creatable' }) + const no = describeOreCreatable(false) + expect(no.value).toBe('not creatable') + expect(no.annotation).toContain('<- skips:') + expect(no.annotation).not.toContain('<- blocks') + }) + + it('renders an unanswerable probe as its own third state', () => { + const unknown = describeOreCreatable(null) + expect(unknown.value).toBe('unknown') + expect(unknown.annotation).toContain('stash eql verify') + }) +}) + +describe('ORE_OPCLASS_PRESENT_EXPR', () => { + /** + * `to_regtype` (not a `::regtype` cast) is what lets the expression return + * `false` on a database with no EQL installed instead of raising — the + * not-installed case is detected and reported separately by every caller. + */ + it('degrades rather than raises on a database with no EQL', () => { + expect(ORE_OPCLASS_PRESENT_EXPR).toContain('to_regtype') + expect(ORE_OPCLASS_PRESENT_EXPR).not.toContain('::regtype') + }) + + it('probes the default btree opclass over the ORE block type', () => { + expect(ORE_OPCLASS_PRESENT_EXPR).toContain('eql_v3_internal.ore_block_256') + expect(ORE_OPCLASS_PRESENT_EXPR).toContain('c.opcdefault') + expect(ORE_OPCLASS_PRESENT_EXPR).toContain(`am.amname = 'btree'`) + }) +}) + +describe('the expected-poison count', () => { + /** + * `describeOreState('fallback')` claims *every* ORE domain carries the + * fallback. That claim is only meaningful if the set it is counted against + * comes from the bundle, so pin the derivation rather than the prose. + */ + it('is derived from the pinned bundle, not hand-maintained', () => { + const { oreDomains } = bundledExpectedSurface() + expect(oreDomains.length).toBeGreaterThan(0) + for (const domain of oreDomains) expect(domain).toMatch(/_ore$/) + }) +}) diff --git a/packages/cli/src/installer/__tests__/preflight.live.test.ts b/packages/cli/src/installer/__tests__/preflight.live.test.ts index 6252cc9c9..ebd7daecc 100644 --- a/packages/cli/src/installer/__tests__/preflight.live.test.ts +++ b/packages/cli/src/installer/__tests__/preflight.live.test.ts @@ -129,4 +129,40 @@ describeLive('EQLInstaller.preflight — live Postgres', () => { }).preflight() expect(asMember.memberOfPostgres).toBe(true) }) + + /** + * The ORE probe (#891) attempts `CREATE OPERATOR FAMILY` and rolls it back. + * Only a live server can prove the two things that matter: that the + * privilege gate answers `false` for an unprivileged role rather than + * throwing, and that the rollback leaves nothing behind — the claim that + * preflight is read-only. + */ + it('probes operator-class creation truthfully, and leaves nothing behind', async () => { + const { EQLInstaller } = await import('../index.js') + await adminQuery( + `DO $$ BEGIN CREATE ROLE ${MEMBER_ROLE} LOGIN PASSWORD '${MEMBER_PASSWORD}'; + EXCEPTION WHEN duplicate_object THEN NULL; END $$`, + ) + + const asSuperuser = await new EQLInstaller({ + databaseUrl: DATABASE_URL as string, + }).preflight() + expect(asSuperuser.canCreateOperatorClass).toBe(true) + + // A plain login role cannot: `CREATE OPERATOR FAMILY` is superuser-gated, + // and the 42501 it raises is the answer, not a probe failure. + const asOutsider = await new EQLInstaller({ + databaseUrl: urlAs(MEMBER_ROLE, MEMBER_PASSWORD), + }).preflight() + expect(asOutsider.isSuperuser).toBe(false) + expect(asOutsider.canCreateOperatorClass).toBe(false) + + // The successful arm rolled back: no operator family survives. This is + // what keeps `eql preflight` honest about being read-only. + const leftovers = await adminQuery<{ n: number }>( + `SELECT count(*)::int AS n FROM pg_catalog.pg_opfamily + WHERE opfname = 'stash_preflight_opclass_probe'`, + ) + expect(leftovers[0]?.n).toBe(0) + }) }) diff --git a/packages/cli/src/installer/__tests__/verify.live.test.ts b/packages/cli/src/installer/__tests__/verify.live.test.ts index 8fed58e84..781555360 100644 --- a/packages/cli/src/installer/__tests__/verify.live.test.ts +++ b/packages/cli/src/installer/__tests__/verify.live.test.ts @@ -19,7 +19,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { EQLInstaller } from '../index.js' -import { verifyEqlSurface } from '../verify.js' +import { readOreState, verifyEqlSurface } from '../verify.js' const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL const describeLive = DATABASE_URL ? describe : describe.skip @@ -72,6 +72,29 @@ describeLive('verifyEqlSurface — live Postgres', () => { expect(report.counts?.casts.present).toBe(report.counts?.casts.expected) }, 60_000) + /** + * `eql status` reads the ORE half through {@link readOreState} rather than + * the full surface diff (#891). Both must answer the same question the same + * way against the same database — a cheap read that disagreed with `verify` + * would be worse than no read at all. + */ + it('reads the same ORE state through the standalone probe as through verify', async () => { + const url = DATABASE_URL ?? '' + const report = await verifyEqlSurface(url) + const { default: pg } = await import('pg') + const client = new pg.Client({ connectionString: url }) + await client.connect() + try { + const ore = await readOreState(client) + expect(ore.state).toBe(report.ore?.state) + expect(ore.opclassPresent).toBe(report.ore?.opclassPresent) + expect(ore.poisonedDomains).toBe(report.ore?.poisonedDomains) + expect(ore.expectedPoisoned).toBe(report.ore?.expectedPoisoned) + } finally { + await client.end().catch(() => undefined) + } + }, 60_000) + it('names a dropped comparison operator, attributed to its domain', async () => { // The failure class from #890: the domain exists, its comparison surface // does not, and `weight >= x` errors at query time. diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 7f09c0f76..634df5765 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -82,6 +82,17 @@ export interface PreflightResult { */ canDropEqlV3Schema: boolean | null canDropEqlV3InternalSchema: boolean | null + /** + * Whether this role can create the ORE btree operator class the `_ord_ore` + * domains need (#891). `null` when the probe could not answer. + * + * Never blocks: the bundle skips the class and installs its loud-failure + * fallback instead, which is a supported configuration. It is reported so + * the trade is known before a schema is written, not after a query fails. + * See {@link probeOperatorClassCreate} for why this is probed rather than + * inferred from `isSuperuser`. + */ + canCreateOperatorClass: boolean | null missing: string[] ok: boolean } @@ -147,6 +158,57 @@ const PREFLIGHT_SQL = ` /** The schemas the pinned bundle accepts `pgcrypto` in (its search_path). */ export const SUPPORTED_PGCRYPTO_SCHEMAS = ['extensions', 'public'] +/** + * Can this role create the ORE btree operator class? (#891) + * + * Asked of the server rather than inferred, because `rolsuper` is the wrong + * question. `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, + * but managed platforms differ on whether their admin role clears that gate: + * AWS RDS and Aurora do (with `rolsuper = f`), cloud-hosted Supabase does not. + * Predicting from `rolsuper` would tell an RDS operator their ORE domains are + * unavailable when they work — exactly the blanket claim about "managed + * Postgres" this whole change exists to stop making. + * + * `CREATE OPERATOR FAMILY` shares the privilege gate with `CREATE OPERATOR + * CLASS` and needs no member operators, so it is the cheapest statement that + * tests it. The whole probe runs in a transaction that is always rolled back, + * so preflight stays observably read-only. + * + * Returns `null` when the attempt could not answer the question — a read-only + * replica (`25006`), a statement timeout, no `public` schema to create into. + * Callers must render that as unknown, never as either answer. + */ +async function probeOperatorClassCreate( + client: pg.ClientBase, +): Promise { + // A name no bundle uses, so a probe that somehow escaped its rollback is + // recognisable rather than mistaken for an EQL object. + const probeName = 'public.stash_preflight_opclass_probe' + try { + await client.query('BEGIN') + } catch { + return null + } + try { + await client.query(`CREATE OPERATOR FAMILY ${probeName} USING btree`) + return true + } catch (error) { + // 42501 insufficient_privilege is the gate itself — a real "no". Anything + // else (no CREATE on public, read-only transaction, timeout) is a probe + // that failed to ask the question. + const code = + typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code?: unknown }).code) + : undefined + return code === '42501' ? false : null + } finally { + // Always: on the success path this is what keeps preflight read-only, and + // on the failure path it clears the aborted transaction. A rollback that + // itself fails leaves nothing behind — the connection is closed next. + await client.query('ROLLBACK').catch(() => {}) + } +} + export class EQLInstaller { private readonly databaseUrl: string @@ -222,6 +284,9 @@ export class EQLInstaller { 'ownership of the existing EQL schemas (a reinstall begins with DROP SCHEMA eql_v3 / eql_v3_internal CASCADE, which needs the owner, a member of the owning role, or a superuser)', ) } + // After the capability read, so a probe that somehow poisons the session + // cannot affect any of the answers above. + const canCreateOperatorClass = await probeOperatorClassCreate(client) return { currentUser: String(row.role_name ?? 'unknown'), isSuperuser, @@ -234,7 +299,11 @@ export class EQLInstaller { eqlV3InternalSchemaPresent: row.eql_v3_internal_present === true, canDropEqlV3Schema, canDropEqlV3InternalSchema, + canCreateOperatorClass, missing, + // Deliberately not folded into `missing`: the bundle's ORE fallback + // means an install without the operator class is complete, not + // blocked. ok: missing.length === 0, } } catch (error) { diff --git a/packages/cli/src/installer/ore.ts b/packages/cli/src/installer/ore.ts new file mode 100644 index 000000000..8728245c6 --- /dev/null +++ b/packages/cli/src/installer/ore.ts @@ -0,0 +1,144 @@ +import { EQL_V3_INTERNAL_SCHEMA_NAME } from './grants.js' + +/** + * The ORE half of an EQL install, in one place (#891). + * + * The EQL bundle wraps `CREATE OPERATOR CLASS` for the ORE btree opclass in a + * guarded `DO` block that swallows `insufficient_privilege` (42501). Where the + * installing role cannot clear that gate, the class is skipped and the bundle + * poisons every `_ord_ore` / `_search_ore` domain with an always-raising + * `eql_ore_unavailable` CHECK instead — so the gap fails loudly at write time + * rather than silently producing an index that never engages. + * + * That is a supported configuration, not a failed install. What it costs the + * operator is the ORE ordering flavour; the OPE one (`types.*Ord`) orders and + * indexes on any role. This module holds the catalogue probe, the state + * machine over it, and the single copy every command uses to say so — before + * the install (`eql preflight`), at the install (`eql install`), and after it + * (`eql status`, `eql verify`). The failure this addresses was an operator + * discovering the trade at query time. + */ + +/** + * Whether the default btree opclass over `eql_v3_internal.ore_block_256` + * exists. Mirrors the EQL bundle's own fallback test (`ore_fallback.sql`) with + * one deliberate difference: `to_regtype` returns NULL where the bundle's + * `::regtype` cast raises, so this degrades to `false` on a database with no + * EQL installed instead of throwing. Callers that care about the difference + * detect "not installed" separately. + * + * A bare SQL *expression*, so callers can select it alone or fold it into a + * wider row. + */ +export const ORE_OPCLASS_PRESENT_EXPR = `EXISTS ( + SELECT 1 + FROM pg_catalog.pg_opclass c + JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod + WHERE am.amname = 'btree' + AND c.opcdefault + AND c.opcintype = to_regtype('${EQL_V3_INTERNAL_SCHEMA_NAME}.ore_block_256') + )` + +/** + * How the ORE half of an install reads. Only the first two are healthy: the + * bundle either created the operator class (privileged install) or skipped it + * and poisoned every ORE domain so the gap fails loudly (the managed-Postgres + * install). The two `incoherent-*` states are half-applied combinations the + * bundle never produces on its own. + */ +export type OreSurfaceState = + | 'indexable' + | 'fallback' + | 'incoherent-unpoisoned' + | 'incoherent-poisoned' + +/** Classify an observed ORE state. Pure. */ +export function classifyOreState(observed: { + opclassPresent: boolean + poisonedDomains: number + expectedPoisoned: number +}): OreSurfaceState { + if (observed.opclassPresent) { + return observed.poisonedDomains === 0 ? 'indexable' : 'incoherent-poisoned' + } + return observed.poisonedDomains === observed.expectedPoisoned + ? 'fallback' + : 'incoherent-unpoisoned' +} + +/** + * The remedy, named once. + * + * It deliberately names `types.*Ord` and not the `_ord_ope` domains: the + * bundle creates `public.eql_v3__ord_ope`, but `@cipherstash/stack` ships + * no `types.*OrdOpe` factory, so pointing a schema author there names a column + * type they cannot declare. `types.*Ord` (`public.eql_v3__ord`) is the same + * CLLW-OPE ordering and is the one with an SDK factory behind it. + */ +export const ORE_FALLBACK_REMEDY = + 'Ordered columns must use OPE ordering: declare them `types.*Ord` (`public.eql_v3_*_ord`), which orders and indexes on any role. `types.*OrdOre` is unusable here — every write to one fails its `eql_ore_unavailable` CHECK.' + +/** + * One line naming what an ORE state means for the operator. `severity` is what + * a caller should render it as; `'info'` states are not damage. + */ +export function describeOreState(state: OreSurfaceState): { + severity: 'info' | 'damage' + /** Short value for a report row. */ + value: string + /** Full sentence, consequence and remedy included. */ + message: string +} { + switch (state) { + case 'indexable': + return { + severity: 'info', + value: 'present', + message: + 'ORE operator class present — the `types.*OrdOre` domains are usable and ORE ordered indexes engage.', + } + case 'fallback': + return { + severity: 'info', + value: 'skipped (expected on managed Postgres)', + message: `ORE operator class not created — this role cannot create one, and the EQL bundle installed its loud-failure fallback instead. This is the supported managed-Postgres configuration, not a failed install. ${ORE_FALLBACK_REMEDY}`, + } + case 'incoherent-poisoned': + return { + severity: 'damage', + value: 'INCOHERENT', + message: + 'The ORE operator class exists, but ORE domains still carry the `eql_ore_unavailable` poison CHECK — writes to those domains fail although ORE works. Reinstall with `stash eql install --force`.', + } + case 'incoherent-unpoisoned': + return { + severity: 'damage', + value: 'INCOHERENT', + message: + 'The ORE operator class is absent, but only some ORE domains carry the loud-failure fallback — the rest would fail at index/ORDER BY time with opaque errors instead. Reinstall with `stash eql install --force`.', + } + } +} + +/** + * What `eql preflight` can say about ORE *before* anything is installed. + * + * `null` is "could not determine" and must never be rendered as either answer. + */ +export function describeOreCreatable(creatable: boolean | null): { + value: string + annotation?: string +} { + if (creatable === null) { + return { + value: 'unknown', + annotation: + '<- could not probe; `stash eql verify` reports it after install', + } + } + if (creatable) return { value: 'creatable' } + return { + value: 'not creatable', + annotation: '<- skips: ORE opclass; use `types.*Ord`, not `types.*OrdOre`', + } +} diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index c1d2f67cb..ef25f6cd1 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -3,6 +3,12 @@ import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' import { SUPPORTED_PGCRYPTO_SCHEMAS } from './index.js' +import { + classifyOreState, + describeOreState, + ORE_OPCLASS_PRESENT_EXPR, + type OreSurfaceState, +} from './ore.js' /** * `stash eql verify` — assert that the installed EQL surface is complete and @@ -56,16 +62,11 @@ export interface ExpectedSurface { } /** - * How the ORE half of the install reads. Only the first two are healthy: - * the bundle either created the operator class (superuser install) or - * skipped it and poisoned every ORE domain so the gap fails loudly - * (managed-Postgres install — a supported configuration, not damage). + * How the ORE half of the install reads. Defined in `./ore.js`, which owns the + * whole ORE model — the catalogue probe, the state machine, and the copy every + * command renders — and re-exported here because `VerifyReport` carries it. */ -export type OreSurfaceState = - | 'indexable' - | 'fallback' - | 'incoherent-unpoisoned' - | 'incoherent-poisoned' +export type { OreSurfaceState } export interface SurfaceFinding { severity: 'damage' | 'warning' | 'expected' @@ -385,14 +386,7 @@ const CASTS_SQL = ` */ const ORE_STATE_SQL = ` SELECT - EXISTS ( - SELECT 1 - FROM pg_catalog.pg_opclass c - JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod - WHERE am.amname = 'btree' - AND c.opcdefault - AND c.opcintype = to_regtype('${EQL_V3_INTERNAL_SCHEMA_NAME}.ore_block_256') - ) AS ore_opclass_present, + ${ORE_OPCLASS_PRESENT_EXPR} AS ore_opclass_present, ( SELECT count(*)::int FROM pg_catalog.pg_constraint c @@ -722,48 +716,22 @@ export function diffSurface( // The ORE conditional: exactly one of the two halves must be present, in // full. Anything else is a half-working state the bundle never produces. const expectedPoisoned = expected.oreDomains.length - let oreState: OreSurfaceState - if (installed.oreOpclassPresent) { - oreState = - installed.poisonedDomains === 0 ? 'indexable' : 'incoherent-poisoned' - } else { - oreState = - installed.poisonedDomains === expectedPoisoned - ? 'fallback' - : 'incoherent-unpoisoned' - } - switch (oreState) { - case 'indexable': - findings.push({ - severity: 'expected', - kind: 'opclass', - message: - 'ORE operator class present — ORE ordered indexes are available (superuser install).', - }) - break - case 'fallback': - findings.push({ - severity: 'expected', - kind: 'opclass', - message: - 'ORE operator class absent, and every ORE domain carries the loud-failure fallback. This is the supported managed-Postgres configuration (creating the class requires superuser), not a failed install — use the `_ord_ope` ordering domains.', - }) - break - case 'incoherent-poisoned': - findings.push({ - severity: 'damage', - kind: 'opclass', - message: `The ORE operator class exists, but ${installed.poisonedDomains} domain${installed.poisonedDomains === 1 ? ' still carries' : 's still carry'} the \`eql_ore_unavailable\` poison CHECK — writes to those domains fail although ORE works. Reinstall with \`stash eql install --force\`.`, - }) - break - case 'incoherent-unpoisoned': - findings.push({ - severity: 'damage', - kind: 'opclass', - message: `The ORE operator class is absent, but only ${installed.poisonedDomains} of ${expectedPoisoned} ORE domains carry the loud-failure fallback — the rest would fail at index/ORDER BY time with opaque errors instead. Reinstall with \`stash eql install --force\`.`, - }) - break - } + const oreState = classifyOreState({ + opclassPresent: installed.oreOpclassPresent, + poisonedDomains: installed.poisonedDomains, + expectedPoisoned, + }) + const oreDescription = describeOreState(oreState) + findings.push({ + severity: oreDescription.severity === 'damage' ? 'damage' : 'expected', + kind: 'opclass', + // The counts only mean anything in the two incoherent states, where they + // say how far the half-application got. + message: + oreDescription.severity === 'damage' + ? `${oreDescription.message} (${installed.poisonedDomains} of ${expectedPoisoned} ORE domains carry the poison CHECK.)` + : oreDescription.message, + }) const damaged = findings.some((finding) => finding.severity === 'damage') return { @@ -839,3 +807,33 @@ export async function verifyEqlSurface( await client.end() } } + +/** + * Read just the ORE half of an install — the two catalogue values and the + * state they classify to (#891). + * + * `eql status` wants the ORE answer and nothing else. Routing it through + * {@link verifyEqlSurface} would work but would read the whole 3,000-operator + * surface to render one row, and would report a version mismatch as a reason + * to say nothing — whereas the ORE state is legible whatever bundle is + * installed, because both halves of the conditional are catalogue facts rather + * than a diff against the pinned manifest. + */ +export async function readOreState(client: pg.ClientBase): Promise<{ + opclassPresent: boolean + poisonedDomains: number + expectedPoisoned: number + state: OreSurfaceState +}> { + const expected = bundledExpectedSurface() + const result = await client.query<{ + ore_opclass_present: boolean + poisoned_domains: number + }>(ORE_STATE_SQL, [expected.oreDomains]) + const observed = { + opclassPresent: result.rows[0]?.ore_opclass_present === true, + poisonedDomains: result.rows[0]?.poisoned_domains ?? 0, + expectedPoisoned: expected.oreDomains.length, + } + return { ...observed, state: classifyOreState(observed) } +} diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 9973c7e62..516ec77e3 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -363,10 +363,12 @@ stash eql status #### `eql preflight` -Read-only report of whether the connected role can install EQL, run before anything is attempted. It probes: `current_user`, superuser, **membership of `postgres`**, `CREATE` on the database and on `public`, `pgcrypto`, and whether the `eql_v3` / `eql_v3_internal` schemas already exist. Each blocked row names the statement it blocks. Exits 1 when a gap would abort `eql install`; `--json` emits the structured result for agents (stdout is pure JSON). +Read-only report of whether the connected role can install EQL, run before anything is attempted. It probes: `current_user`, superuser, **membership of `postgres`**, `CREATE` on the database and on `public`, `pgcrypto`, whether the role **can create an operator class**, and whether the `eql_v3` / `eql_v3_internal` schemas already exist. Each blocked row names the statement it blocks. Exits 1 when a gap would abort `eql install`; `--json` emits the structured result for agents (stdout is pure JSON). Membership of `postgres` is reported but never blocks: `eql install` handles a non-member role by skipping the owner-scoped `ALTER DEFAULT PRIVILEGES FOR ROLE postgres` statements, which are **optional** — the install is complete without them (see `eql install` below). This matters on managed platforms whose database role is not `postgres` and not a member of it (e.g. Lovable's `sandbox_exec`). +**The `ORE operator class` row never blocks either** (`canCreateOperatorClass` in `--json`; `creatable` / `not creatable` / `unknown`). It answers the one schema-design question you want settled *before* writing types: `not creatable` means `eql install` will skip the ORE operator class and install the bundle's loud-failure fallback in its place, so **declare ordered columns `types.*Ord`, not `types.*OrdOre`** — every write to an `_ord_ore` domain on such a database fails its `eql_ore_unavailable` CHECK. This is probed, not inferred from `superuser`: `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, but AWS RDS and Aurora let their admin role create one while cloud-hosted Supabase does not — so a role with `rolsuper = f` is not evidence either way. The probe attempts the DDL inside a transaction it always rolls back, so preflight stays read-only; `unknown` means it could not ask (a read-only replica, say) and must not be read as either answer. + | Flag | Description | |---|---| | `--json` | Machine-readable result instead of the table. Stdout is pure JSON even on failure: success is `{ status: 'ok', ... }`, blockers are `{ status: 'blocked', ... }` (exit 1), and failures — including a missing/malformed DATABASE_URL — are the shared `{ status: 'error', code, message }` envelope | @@ -492,6 +494,8 @@ The install SQL is safe to re-run — columns and data survive — but it cascad Whether EQL is installed and at which version, plus database permission status. It retains read-only EQL v2/config-table diagnostics for existing deployments. +On a v3 install it also reports the **ORE operator class** state, so the ordering trade `eql install` named once is recoverable later without re-reading scrollback: either the class is present (`types.*OrdOre` usable), or it was skipped and every `_ord_ore` domain carries the loud-failure fallback — the supported managed-Postgres configuration, where ordered columns must be `types.*Ord`. Anything else is damage and points at `eql install --force`. Run `eql verify` for the full surface check. + #### `eql validate` — validate the encryption schema ```bash