diff --git a/.changeset/ore-unavailable-at-install.md b/.changeset/ore-unavailable-at-install.md new file mode 100644 index 000000000..cd228f51d --- /dev/null +++ b/.changeset/ore-unavailable-at-install.md @@ -0,0 +1,14 @@ +--- +'stash': minor +--- + +Report the ORE-unavailable case once, at install time, instead of leaving it to surface as a failing predicate the first time a column is cast. + +The EQL bundle skips the ORE btree operator class when the installing role cannot create one and poisons every `_ord_ore` domain with a loud-failure CHECK in its place. That is a supported configuration — but nothing said so where the choice between `types.*Ord` and `types.*OrdOre` is actually made, so the trade was discovered at query time. + +- **`stash eql preflight` now probes whether the role can create an operator class** and reports it as a non-blocking `ORE operator class` row (`creatable` / `not creatable` / `unknown`; `canCreateOperatorClass` in `--json`). It is *probed*, not inferred from `superuser`: `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, but AWS RDS and Aurora let their admin role create one while cloud-hosted Supabase does not, so `rolsuper` is not evidence either way. The probe attempts the DDL inside a transaction it always rolls back, leaving preflight read-only; a probe that could not ask reports `unknown` rather than guessing. +- **`stash eql install` names the consequence and the remedy** on its own line when the fallback was installed, rather than as a parenthetical on the "verified" line. +- **`stash eql status` reports the ORE state** on a v3 install, so the answer survives past the install output. +- **The remedy now names a type that exists.** The previous wording pointed at the `_ord_ope` domains; the bundle creates those, but `@cipherstash/stack` ships no `types.*OrdOpe` factory, so it named a column type no schema author could declare. Every command now says `types.*Ord` (`public.eql_v3_*_ord`), which is the same CLLW-OPE ordering and has a factory behind it. +- The ORE state machine, the catalogue probe, and this copy now live in one module shared by `eql preflight`, `eql install`, `eql status`, `eql verify`, and `eql validate`, so the five commands cannot drift into disagreeing about the same catalogue fact. +- The scaffolded encryption client's type cheat-sheet now says why ordered columns should be `*Ord` rather than `*OrdOre`. diff --git a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts index c423ccb83..8bec63e43 100644 --- a/packages/cli/__fixtures__/scaffold/drizzle.generated.ts +++ b/packages/cli/__fixtures__/scaffold/drizzle.generated.ts @@ -23,6 +23,13 @@ * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the `*Ord` factories above, not `*OrdOre`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * `_ord_ore` domain, and each write to one fails a CHECK. `*Ord` orders and + * indexes on any role. `stash eql status` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): diff --git a/packages/cli/__fixtures__/scaffold/generic.generated.ts b/packages/cli/__fixtures__/scaffold/generic.generated.ts index 80981180c..3692db5bb 100644 --- a/packages/cli/__fixtures__/scaffold/generic.generated.ts +++ b/packages/cli/__fixtures__/scaffold/generic.generated.ts @@ -23,6 +23,13 @@ * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the `*Ord` factories above, not `*OrdOre`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * `_ord_ore` domain, and each write to one fails a CHECK. `*Ord` orders and + * indexes on any role. `stash eql status` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): diff --git a/packages/cli/src/commands/db/__tests__/preflight.test.ts b/packages/cli/src/commands/db/__tests__/preflight.test.ts index 5e1eb23a1..dbdf78855 100644 --- a/packages/cli/src/commands/db/__tests__/preflight.test.ts +++ b/packages/cli/src/commands/db/__tests__/preflight.test.ts @@ -14,6 +14,7 @@ const CAPABLE: PreflightResult = { eqlV3InternalSchemaPresent: false, canDropEqlV3Schema: null, canDropEqlV3InternalSchema: null, + canCreateOperatorClass: true, missing: [], ok: true, } @@ -78,6 +79,40 @@ describe('renderPreflightReport', () => { expect(report).toContain('<- blocks: not on the EQL search_path') }) + // #891: the ORE trade is reported so it is known before a schema is + // written. It must never read as a blocker — the bundle's fallback makes an + // install without the operator class a complete install. + it('names the ORE consequence for a role that cannot create an operator class', () => { + const report = renderPreflightReport({ + ...CAPABLE, + currentUser: 'sandbox_exec', + isSuperuser: false, + canCreateOperatorClass: false, + }) + expect(report).toMatch(/ORE operator class\s+not creatable/) + expect(report).toContain('<- skips: ORE opclass') + expect(report).toContain('`types.*Ord`, not `types.*OrdOre`') + expect(report).not.toContain('<- blocks') + }) + + it('leaves the ORE row unannotated for a role that can create one', () => { + const report = renderPreflightReport(CAPABLE) + expect(report).toMatch(/ORE operator class\s+creatable/) + expect(report).not.toContain('<- skips: ORE opclass') + }) + + // A probe that could not ask the question must not be rendered as either + // answer — "unknown" is the honest third state. + it('renders an unanswerable ORE probe as unknown, not as no', () => { + const report = renderPreflightReport({ + ...CAPABLE, + canCreateOperatorClass: null, + }) + expect(report).toMatch(/ORE operator class\s+unknown/) + expect(report).toContain('`stash eql verify` reports it after install') + expect(report).not.toContain('not creatable') + }) + it('adds the drop-ownership row only when an EQL schema exists', () => { expect(renderPreflightReport(CAPABLE)).not.toContain('can drop EQL schemas') const blocked = renderPreflightReport({ diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index 5b401cb6a..cd8569ccc 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -4,6 +4,7 @@ import { resolveDatabaseUrl } from '@/config/database-url.js' import { findConfigFile, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' import { EQLInstaller } from '@/installer/index.js' +import { describeOreState } from '@/installer/ore.js' import { type VerifyReport, verifyEqlSurface } from '@/installer/verify.js' import { messages } from '@/messages.js' import { detectPackageManager, runnerCommand } from '../init/utils.js' @@ -265,11 +266,13 @@ export async function verifySurfaceOrExit( return } if (report.ok) { - s.stop( - report.ore?.state === 'fallback' - ? 'EQL surface verified (ORE operator class skipped — expected for this role; use the `_ord_ope` ordering domains).' - : 'EQL surface verified — install is complete.', - ) + s.stop('EQL surface verified — install is complete.') + // Reported once, here, rather than left to surface as a failing predicate + // the first time someone casts a column (#891). `eql status` and + // `eql verify` say the same thing later, so the answer is recoverable. + if (report.ore?.state === 'fallback') { + p.log.info(describeOreState('fallback').message) + } return } s.stop('The installed EQL surface is incomplete.') diff --git a/packages/cli/src/commands/db/preflight.ts b/packages/cli/src/commands/db/preflight.ts index d5b8543bd..b60c53397 100644 --- a/packages/cli/src/commands/db/preflight.ts +++ b/packages/cli/src/commands/db/preflight.ts @@ -2,6 +2,7 @@ import * as p from '@clack/prompts' import { emitJsonError, emitJsonEvent } from '@/commands/auth/events.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { EQLInstaller, type PreflightResult } from '@/installer/index.js' +import { describeOreCreatable, ORE_FALLBACK_REMEDY } from '@/installer/ore.js' import { resolveDiagnosticDatabaseUrl } from './resolve-diagnostic-url.js' /** See {@link resolveDiagnosticDatabaseUrl} — preflight keeps config-wins. */ @@ -90,6 +91,14 @@ export async function preflightCommand( 'Not a member of `postgres`: in Supabase mode the optional `ALTER DEFAULT PRIVILEGES FOR ROLE postgres` statements are skipped. The install is complete without them — they only cover EQL objects created outside stash tooling, and stash re-grants every object on each install/upgrade.', ) } + // Said here so the ordering trade is known before a schema is written. The + // same sentence is printed by `eql install` and `eql status` against the + // installed state, so the answer survives past this command's output (#891). + if (result.canCreateOperatorClass === false) { + p.log.info( + `This role cannot create an operator class, so \`eql install\` will skip the ORE one and install its loud-failure fallback instead. That is a complete install. ${ORE_FALLBACK_REMEDY}`, + ) + } p.outro('This role can install EQL.') } @@ -140,6 +149,16 @@ export function renderPreflightReport(result: PreflightResult): string { ? '<- blocks: CREATE EXTENSION pgcrypto' : undefined, ], + // Never annotated as a blocker: the bundle skips the class and installs + // its loud-failure fallback, which is a complete install (#891). + // Same label `eql verify` uses for the installed state, so the row an + // operator reads before the install and the row they read after it are + // recognisably about the same thing. The value carries the tense. + [ + 'ORE operator class', + describeOreCreatable(result.canCreateOperatorClass).value, + describeOreCreatable(result.canCreateOperatorClass).annotation, + ], ['eql_v3 schema', result.eqlV3SchemaPresent ? 'present' : 'absent'], [ 'eql_v3_internal', diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index 34ce0aece..b10c26a64 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -3,6 +3,8 @@ import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' import { EQLInstaller } from '@/installer/index.js' +import { describeOreState } from '@/installer/ore.js' +import { readOreState } from '@/installer/verify.js' export async function statusCommand(options: { databaseUrl?: string } = {}) { const pm = detectPackageManager() @@ -93,7 +95,40 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { ) } - // 3. Encrypt configuration. + // 3. The ORE half of the install. + // + // Reported here so the trade an operator was told about at install time is + // recoverable afterwards, without re-reading scrollback (#891). Only when + // v3 is installed: the state is a property of the v3 bundle's conditional + // half, and reads as 'fallback' on a database that has no EQL at all. + if (installedV3) { + s.start('Checking ORE operator class...') + const oreClient = createPgClient(config.databaseUrl) + try { + await oreClient.connect() + const ore = await readOreState(oreClient) + s.stop('ORE state checked.') + const described = describeOreState(ore.state) + if (described.severity === 'damage') { + p.log.error(described.message) + } else { + p.log.info(described.message) + } + } catch (error) { + // Advisory, not a gate: a status run that could not read one row should + // still report everything else it read. + s.stop('ORE state check failed.') + p.log.warn( + `Could not determine the ORE operator class state: ${ + error instanceof Error ? error.message : String(error) + }`, + ) + } finally { + await oreClient.end().catch(() => {}) + } + } + + // 4. Encrypt configuration. // // `public.eql_v2_configuration` is a v2 + CipherStash Proxy artifact: the v2 // install creates it and Proxy reads it. EQL v3 has no configuration table — diff --git a/packages/cli/src/commands/eql/validate.ts b/packages/cli/src/commands/eql/validate.ts index f55691fdd..c9ecdbbfc 100644 --- a/packages/cli/src/commands/eql/validate.ts +++ b/packages/cli/src/commands/eql/validate.ts @@ -5,6 +5,7 @@ import { fetchPhysicalColumns } from '@/commands/encrypt/lib/db-readers.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadEncryptSchemas, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' +import { ORE_OPCLASS_PRESENT_EXPR } from '@/installer/ore.js' // --------------------------------------------------------------------------- // The vocabulary @@ -604,21 +605,11 @@ function identifiersIn(args: string): string[] { } /** - * Whether the ORE btree operator class exists. Mirrors the EQL bundle's own - * fallback test (`ore_fallback.sql`), with one deliberate difference: - * `to_regtype` returns NULL where the bundle's `::regtype` cast raises, so this - * degrades to `false` on a database with no EQL installed instead of throwing. - * That is why the not-installed case is detected and reported separately. + * Whether the ORE btree operator class exists. The expression is shared with + * `eql verify` and `eql status` (`installer/ore.ts`) so the three commands + * cannot drift into disagreeing about the same catalogue fact. */ -const ORE_AVAILABLE_SQL = ` - SELECT EXISTS ( - SELECT 1 - FROM pg_catalog.pg_opclass c - JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod - WHERE am.amname = 'btree' - AND c.opcdefault - AND c.opcintype = to_regtype('eql_v3_internal.ore_block_256') - ) AS ore_available` +const ORE_AVAILABLE_SQL = `SELECT ${ORE_OPCLASS_PRESENT_EXPR} AS ore_available` const EQL_INSTALLED_SQL = ` SELECT EXISTS ( diff --git a/packages/cli/src/commands/eql/verify.ts b/packages/cli/src/commands/eql/verify.ts index d323bd75f..f17cfe0da 100644 --- a/packages/cli/src/commands/eql/verify.ts +++ b/packages/cli/src/commands/eql/verify.ts @@ -2,6 +2,7 @@ import * as p from '@clack/prompts' import { emitJsonError, emitJsonEvent } from '@/commands/auth/events.js' import { resolveDiagnosticDatabaseUrl } from '@/commands/db/resolve-diagnostic-url.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { describeOreState } from '@/installer/ore.js' import type { SurfaceFinding, VerifyReport } from '@/installer/verify.js' import { verifyEqlSurface } from '@/installer/verify.js' @@ -144,14 +145,7 @@ export function renderSurfaceCounts(report: VerifyReport): string { ] }, ), - [ - 'ORE operator class', - ore.state === 'indexable' - ? 'present' - : ore.state === 'fallback' - ? 'skipped (expected on managed Postgres)' - : 'INCOHERENT', - ], + ['ORE operator class', describeOreState(ore.state).value], ] const labelWidth = Math.max(...rows.map(([label]) => label.length)) return rows diff --git a/packages/cli/src/commands/init/utils.ts b/packages/cli/src/commands/init/utils.ts index 95458c123..58a63a48b 100644 --- a/packages/cli/src/commands/init/utils.ts +++ b/packages/cli/src/commands/init/utils.ts @@ -400,6 +400,13 @@ const DRIZZLE_PLACEHOLDER = `/** * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the \`*Ord\` factories above, not \`*OrdOre\`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * \`_ord_ore\` domain, and each write to one fails a CHECK. \`*Ord\` orders and + * indexes on any role. \`stash eql status\` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): @@ -468,6 +475,13 @@ const GENERIC_PLACEHOLDER = `/** * types.TextSearch equality + order/range + free-text * types.Json encrypted-JSONB containment + selectors * + * Order columns with the \`*Ord\` factories above, not \`*OrdOre\`. The ORE + * flavour needs a Postgres operator class that only a privileged role can + * create — where the install could not create it, the EQL bundle poisons every + * \`_ord_ore\` domain, and each write to one fails a CHECK. \`*Ord\` orders and + * indexes on any role. \`stash eql status\` reports which case this database + * is in. + * * --- Pattern reference (copy into your real schema, do NOT use as-is) --- * * Encrypted twin column for an existing populated column (path 3 — lifecycle): diff --git a/packages/cli/src/installer/__tests__/ore.test.ts b/packages/cli/src/installer/__tests__/ore.test.ts new file mode 100644 index 000000000..74112bc7f --- /dev/null +++ b/packages/cli/src/installer/__tests__/ore.test.ts @@ -0,0 +1,150 @@ +import { describe, expect, it } from 'vitest' +import { bundledExpectedSurface } from '@/installer/verify.js' +import { + classifyOreState, + describeOreCreatable, + describeOreState, + ORE_FALLBACK_REMEDY, + ORE_OPCLASS_PRESENT_EXPR, + type OreSurfaceState, +} from '../ore.js' + +/** + * The shared ORE model (#891). These assertions are about the *copy* as much + * as the logic: the whole point of the module is that `eql preflight`, + * `eql install`, `eql status` and `eql verify` say one thing about ORE, and + * that the thing they say names a remedy a schema author can actually type. + */ +describe('classifyOreState', () => { + const expectedPoisoned = 20 + + it('reads a privileged install as indexable', () => { + expect( + classifyOreState({ + opclassPresent: true, + poisonedDomains: 0, + expectedPoisoned, + }), + ).toBe('indexable') + }) + + it('reads the managed-Postgres skip with a full fallback as fallback', () => { + expect( + classifyOreState({ + opclassPresent: false, + poisonedDomains: expectedPoisoned, + expectedPoisoned, + }), + ).toBe('fallback') + }) + + it('reads a partial fallback as incoherent, not as the supported skip', () => { + expect( + classifyOreState({ + opclassPresent: false, + poisonedDomains: expectedPoisoned - 1, + expectedPoisoned, + }), + ).toBe('incoherent-unpoisoned') + }) + + it('reads leftover poison alongside a present opclass as incoherent', () => { + expect( + classifyOreState({ + opclassPresent: true, + poisonedDomains: 1, + expectedPoisoned, + }), + ).toBe('incoherent-poisoned') + }) +}) + +describe('describeOreState', () => { + it('treats both healthy states as info and both incoherent ones as damage', () => { + expect(describeOreState('indexable').severity).toBe('info') + expect(describeOreState('fallback').severity).toBe('info') + expect(describeOreState('incoherent-poisoned').severity).toBe('damage') + expect(describeOreState('incoherent-unpoisoned').severity).toBe('damage') + }) + + it('names the consequence and the remedy on the fallback path', () => { + const { message } = describeOreState('fallback') + // The consequence: not a failed install. + expect(message).toContain('not a failed install') + // The remedy, in the words a schema author types. + expect(message).toContain('types.*Ord') + expect(message).toContain(ORE_FALLBACK_REMEDY) + }) + + /** + * The bundle creates `public.eql_v3__ord_ope` domains, but + * `@cipherstash/stack` ships no `types.*OrdOpe` factory — so naming the + * `_ord_ope` domains as the remedy sends a schema author to a column type + * they cannot declare. This is the regression that guard exists for. + */ + it('does not point a schema author at the factory-less `_ord_ope` domains', () => { + const states: OreSurfaceState[] = [ + 'indexable', + 'fallback', + 'incoherent-poisoned', + 'incoherent-unpoisoned', + ] + for (const state of states) { + expect(describeOreState(state).message).not.toContain('_ord_ope') + expect(describeOreState(state).message).not.toContain('OrdOpe') + } + }) + + it('gives every state a short value for a report row', () => { + expect(describeOreState('indexable').value).toBe('present') + expect(describeOreState('fallback').value).toContain('skipped') + expect(describeOreState('incoherent-poisoned').value).toBe('INCOHERENT') + }) +}) + +describe('describeOreCreatable', () => { + it('annotates only the "no" answer, and never as a blocker', () => { + expect(describeOreCreatable(true)).toEqual({ value: 'creatable' }) + const no = describeOreCreatable(false) + expect(no.value).toBe('not creatable') + expect(no.annotation).toContain('<- skips:') + expect(no.annotation).not.toContain('<- blocks') + }) + + it('renders an unanswerable probe as its own third state', () => { + const unknown = describeOreCreatable(null) + expect(unknown.value).toBe('unknown') + expect(unknown.annotation).toContain('stash eql verify') + }) +}) + +describe('ORE_OPCLASS_PRESENT_EXPR', () => { + /** + * `to_regtype` (not a `::regtype` cast) is what lets the expression return + * `false` on a database with no EQL installed instead of raising — the + * not-installed case is detected and reported separately by every caller. + */ + it('degrades rather than raises on a database with no EQL', () => { + expect(ORE_OPCLASS_PRESENT_EXPR).toContain('to_regtype') + expect(ORE_OPCLASS_PRESENT_EXPR).not.toContain('::regtype') + }) + + it('probes the default btree opclass over the ORE block type', () => { + expect(ORE_OPCLASS_PRESENT_EXPR).toContain('eql_v3_internal.ore_block_256') + expect(ORE_OPCLASS_PRESENT_EXPR).toContain('c.opcdefault') + expect(ORE_OPCLASS_PRESENT_EXPR).toContain(`am.amname = 'btree'`) + }) +}) + +describe('the expected-poison count', () => { + /** + * `describeOreState('fallback')` claims *every* ORE domain carries the + * fallback. That claim is only meaningful if the set it is counted against + * comes from the bundle, so pin the derivation rather than the prose. + */ + it('is derived from the pinned bundle, not hand-maintained', () => { + const { oreDomains } = bundledExpectedSurface() + expect(oreDomains.length).toBeGreaterThan(0) + for (const domain of oreDomains) expect(domain).toMatch(/_ore$/) + }) +}) diff --git a/packages/cli/src/installer/__tests__/preflight.live.test.ts b/packages/cli/src/installer/__tests__/preflight.live.test.ts index 6252cc9c9..ebd7daecc 100644 --- a/packages/cli/src/installer/__tests__/preflight.live.test.ts +++ b/packages/cli/src/installer/__tests__/preflight.live.test.ts @@ -129,4 +129,40 @@ describeLive('EQLInstaller.preflight — live Postgres', () => { }).preflight() expect(asMember.memberOfPostgres).toBe(true) }) + + /** + * The ORE probe (#891) attempts `CREATE OPERATOR FAMILY` and rolls it back. + * Only a live server can prove the two things that matter: that the + * privilege gate answers `false` for an unprivileged role rather than + * throwing, and that the rollback leaves nothing behind — the claim that + * preflight is read-only. + */ + it('probes operator-class creation truthfully, and leaves nothing behind', async () => { + const { EQLInstaller } = await import('../index.js') + await adminQuery( + `DO $$ BEGIN CREATE ROLE ${MEMBER_ROLE} LOGIN PASSWORD '${MEMBER_PASSWORD}'; + EXCEPTION WHEN duplicate_object THEN NULL; END $$`, + ) + + const asSuperuser = await new EQLInstaller({ + databaseUrl: DATABASE_URL as string, + }).preflight() + expect(asSuperuser.canCreateOperatorClass).toBe(true) + + // A plain login role cannot: `CREATE OPERATOR FAMILY` is superuser-gated, + // and the 42501 it raises is the answer, not a probe failure. + const asOutsider = await new EQLInstaller({ + databaseUrl: urlAs(MEMBER_ROLE, MEMBER_PASSWORD), + }).preflight() + expect(asOutsider.isSuperuser).toBe(false) + expect(asOutsider.canCreateOperatorClass).toBe(false) + + // The successful arm rolled back: no operator family survives. This is + // what keeps `eql preflight` honest about being read-only. + const leftovers = await adminQuery<{ n: number }>( + `SELECT count(*)::int AS n FROM pg_catalog.pg_opfamily + WHERE opfname = 'stash_preflight_opclass_probe'`, + ) + expect(leftovers[0]?.n).toBe(0) + }) }) diff --git a/packages/cli/src/installer/__tests__/verify.live.test.ts b/packages/cli/src/installer/__tests__/verify.live.test.ts index 8fed58e84..781555360 100644 --- a/packages/cli/src/installer/__tests__/verify.live.test.ts +++ b/packages/cli/src/installer/__tests__/verify.live.test.ts @@ -19,7 +19,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { EQLInstaller } from '../index.js' -import { verifyEqlSurface } from '../verify.js' +import { readOreState, verifyEqlSurface } from '../verify.js' const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL const describeLive = DATABASE_URL ? describe : describe.skip @@ -72,6 +72,29 @@ describeLive('verifyEqlSurface — live Postgres', () => { expect(report.counts?.casts.present).toBe(report.counts?.casts.expected) }, 60_000) + /** + * `eql status` reads the ORE half through {@link readOreState} rather than + * the full surface diff (#891). Both must answer the same question the same + * way against the same database — a cheap read that disagreed with `verify` + * would be worse than no read at all. + */ + it('reads the same ORE state through the standalone probe as through verify', async () => { + const url = DATABASE_URL ?? '' + const report = await verifyEqlSurface(url) + const { default: pg } = await import('pg') + const client = new pg.Client({ connectionString: url }) + await client.connect() + try { + const ore = await readOreState(client) + expect(ore.state).toBe(report.ore?.state) + expect(ore.opclassPresent).toBe(report.ore?.opclassPresent) + expect(ore.poisonedDomains).toBe(report.ore?.poisonedDomains) + expect(ore.expectedPoisoned).toBe(report.ore?.expectedPoisoned) + } finally { + await client.end().catch(() => undefined) + } + }, 60_000) + it('names a dropped comparison operator, attributed to its domain', async () => { // The failure class from #890: the domain exists, its comparison surface // does not, and `weight >= x` errors at query time. diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 7f09c0f76..634df5765 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -82,6 +82,17 @@ export interface PreflightResult { */ canDropEqlV3Schema: boolean | null canDropEqlV3InternalSchema: boolean | null + /** + * Whether this role can create the ORE btree operator class the `_ord_ore` + * domains need (#891). `null` when the probe could not answer. + * + * Never blocks: the bundle skips the class and installs its loud-failure + * fallback instead, which is a supported configuration. It is reported so + * the trade is known before a schema is written, not after a query fails. + * See {@link probeOperatorClassCreate} for why this is probed rather than + * inferred from `isSuperuser`. + */ + canCreateOperatorClass: boolean | null missing: string[] ok: boolean } @@ -147,6 +158,57 @@ const PREFLIGHT_SQL = ` /** The schemas the pinned bundle accepts `pgcrypto` in (its search_path). */ export const SUPPORTED_PGCRYPTO_SCHEMAS = ['extensions', 'public'] +/** + * Can this role create the ORE btree operator class? (#891) + * + * Asked of the server rather than inferred, because `rolsuper` is the wrong + * question. `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, + * but managed platforms differ on whether their admin role clears that gate: + * AWS RDS and Aurora do (with `rolsuper = f`), cloud-hosted Supabase does not. + * Predicting from `rolsuper` would tell an RDS operator their ORE domains are + * unavailable when they work — exactly the blanket claim about "managed + * Postgres" this whole change exists to stop making. + * + * `CREATE OPERATOR FAMILY` shares the privilege gate with `CREATE OPERATOR + * CLASS` and needs no member operators, so it is the cheapest statement that + * tests it. The whole probe runs in a transaction that is always rolled back, + * so preflight stays observably read-only. + * + * Returns `null` when the attempt could not answer the question — a read-only + * replica (`25006`), a statement timeout, no `public` schema to create into. + * Callers must render that as unknown, never as either answer. + */ +async function probeOperatorClassCreate( + client: pg.ClientBase, +): Promise { + // A name no bundle uses, so a probe that somehow escaped its rollback is + // recognisable rather than mistaken for an EQL object. + const probeName = 'public.stash_preflight_opclass_probe' + try { + await client.query('BEGIN') + } catch { + return null + } + try { + await client.query(`CREATE OPERATOR FAMILY ${probeName} USING btree`) + return true + } catch (error) { + // 42501 insufficient_privilege is the gate itself — a real "no". Anything + // else (no CREATE on public, read-only transaction, timeout) is a probe + // that failed to ask the question. + const code = + typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code?: unknown }).code) + : undefined + return code === '42501' ? false : null + } finally { + // Always: on the success path this is what keeps preflight read-only, and + // on the failure path it clears the aborted transaction. A rollback that + // itself fails leaves nothing behind — the connection is closed next. + await client.query('ROLLBACK').catch(() => {}) + } +} + export class EQLInstaller { private readonly databaseUrl: string @@ -222,6 +284,9 @@ export class EQLInstaller { 'ownership of the existing EQL schemas (a reinstall begins with DROP SCHEMA eql_v3 / eql_v3_internal CASCADE, which needs the owner, a member of the owning role, or a superuser)', ) } + // After the capability read, so a probe that somehow poisons the session + // cannot affect any of the answers above. + const canCreateOperatorClass = await probeOperatorClassCreate(client) return { currentUser: String(row.role_name ?? 'unknown'), isSuperuser, @@ -234,7 +299,11 @@ export class EQLInstaller { eqlV3InternalSchemaPresent: row.eql_v3_internal_present === true, canDropEqlV3Schema, canDropEqlV3InternalSchema, + canCreateOperatorClass, missing, + // Deliberately not folded into `missing`: the bundle's ORE fallback + // means an install without the operator class is complete, not + // blocked. ok: missing.length === 0, } } catch (error) { diff --git a/packages/cli/src/installer/ore.ts b/packages/cli/src/installer/ore.ts new file mode 100644 index 000000000..8728245c6 --- /dev/null +++ b/packages/cli/src/installer/ore.ts @@ -0,0 +1,144 @@ +import { EQL_V3_INTERNAL_SCHEMA_NAME } from './grants.js' + +/** + * The ORE half of an EQL install, in one place (#891). + * + * The EQL bundle wraps `CREATE OPERATOR CLASS` for the ORE btree opclass in a + * guarded `DO` block that swallows `insufficient_privilege` (42501). Where the + * installing role cannot clear that gate, the class is skipped and the bundle + * poisons every `_ord_ore` / `_search_ore` domain with an always-raising + * `eql_ore_unavailable` CHECK instead — so the gap fails loudly at write time + * rather than silently producing an index that never engages. + * + * That is a supported configuration, not a failed install. What it costs the + * operator is the ORE ordering flavour; the OPE one (`types.*Ord`) orders and + * indexes on any role. This module holds the catalogue probe, the state + * machine over it, and the single copy every command uses to say so — before + * the install (`eql preflight`), at the install (`eql install`), and after it + * (`eql status`, `eql verify`). The failure this addresses was an operator + * discovering the trade at query time. + */ + +/** + * Whether the default btree opclass over `eql_v3_internal.ore_block_256` + * exists. Mirrors the EQL bundle's own fallback test (`ore_fallback.sql`) with + * one deliberate difference: `to_regtype` returns NULL where the bundle's + * `::regtype` cast raises, so this degrades to `false` on a database with no + * EQL installed instead of throwing. Callers that care about the difference + * detect "not installed" separately. + * + * A bare SQL *expression*, so callers can select it alone or fold it into a + * wider row. + */ +export const ORE_OPCLASS_PRESENT_EXPR = `EXISTS ( + SELECT 1 + FROM pg_catalog.pg_opclass c + JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod + WHERE am.amname = 'btree' + AND c.opcdefault + AND c.opcintype = to_regtype('${EQL_V3_INTERNAL_SCHEMA_NAME}.ore_block_256') + )` + +/** + * How the ORE half of an install reads. Only the first two are healthy: the + * bundle either created the operator class (privileged install) or skipped it + * and poisoned every ORE domain so the gap fails loudly (the managed-Postgres + * install). The two `incoherent-*` states are half-applied combinations the + * bundle never produces on its own. + */ +export type OreSurfaceState = + | 'indexable' + | 'fallback' + | 'incoherent-unpoisoned' + | 'incoherent-poisoned' + +/** Classify an observed ORE state. Pure. */ +export function classifyOreState(observed: { + opclassPresent: boolean + poisonedDomains: number + expectedPoisoned: number +}): OreSurfaceState { + if (observed.opclassPresent) { + return observed.poisonedDomains === 0 ? 'indexable' : 'incoherent-poisoned' + } + return observed.poisonedDomains === observed.expectedPoisoned + ? 'fallback' + : 'incoherent-unpoisoned' +} + +/** + * The remedy, named once. + * + * It deliberately names `types.*Ord` and not the `_ord_ope` domains: the + * bundle creates `public.eql_v3__ord_ope`, but `@cipherstash/stack` ships + * no `types.*OrdOpe` factory, so pointing a schema author there names a column + * type they cannot declare. `types.*Ord` (`public.eql_v3__ord`) is the same + * CLLW-OPE ordering and is the one with an SDK factory behind it. + */ +export const ORE_FALLBACK_REMEDY = + 'Ordered columns must use OPE ordering: declare them `types.*Ord` (`public.eql_v3_*_ord`), which orders and indexes on any role. `types.*OrdOre` is unusable here — every write to one fails its `eql_ore_unavailable` CHECK.' + +/** + * One line naming what an ORE state means for the operator. `severity` is what + * a caller should render it as; `'info'` states are not damage. + */ +export function describeOreState(state: OreSurfaceState): { + severity: 'info' | 'damage' + /** Short value for a report row. */ + value: string + /** Full sentence, consequence and remedy included. */ + message: string +} { + switch (state) { + case 'indexable': + return { + severity: 'info', + value: 'present', + message: + 'ORE operator class present — the `types.*OrdOre` domains are usable and ORE ordered indexes engage.', + } + case 'fallback': + return { + severity: 'info', + value: 'skipped (expected on managed Postgres)', + message: `ORE operator class not created — this role cannot create one, and the EQL bundle installed its loud-failure fallback instead. This is the supported managed-Postgres configuration, not a failed install. ${ORE_FALLBACK_REMEDY}`, + } + case 'incoherent-poisoned': + return { + severity: 'damage', + value: 'INCOHERENT', + message: + 'The ORE operator class exists, but ORE domains still carry the `eql_ore_unavailable` poison CHECK — writes to those domains fail although ORE works. Reinstall with `stash eql install --force`.', + } + case 'incoherent-unpoisoned': + return { + severity: 'damage', + value: 'INCOHERENT', + message: + 'The ORE operator class is absent, but only some ORE domains carry the loud-failure fallback — the rest would fail at index/ORDER BY time with opaque errors instead. Reinstall with `stash eql install --force`.', + } + } +} + +/** + * What `eql preflight` can say about ORE *before* anything is installed. + * + * `null` is "could not determine" and must never be rendered as either answer. + */ +export function describeOreCreatable(creatable: boolean | null): { + value: string + annotation?: string +} { + if (creatable === null) { + return { + value: 'unknown', + annotation: + '<- could not probe; `stash eql verify` reports it after install', + } + } + if (creatable) return { value: 'creatable' } + return { + value: 'not creatable', + annotation: '<- skips: ORE opclass; use `types.*Ord`, not `types.*OrdOre`', + } +} diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index c1d2f67cb..ef25f6cd1 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -3,6 +3,12 @@ import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' import { SUPPORTED_PGCRYPTO_SCHEMAS } from './index.js' +import { + classifyOreState, + describeOreState, + ORE_OPCLASS_PRESENT_EXPR, + type OreSurfaceState, +} from './ore.js' /** * `stash eql verify` — assert that the installed EQL surface is complete and @@ -56,16 +62,11 @@ export interface ExpectedSurface { } /** - * How the ORE half of the install reads. Only the first two are healthy: - * the bundle either created the operator class (superuser install) or - * skipped it and poisoned every ORE domain so the gap fails loudly - * (managed-Postgres install — a supported configuration, not damage). + * How the ORE half of the install reads. Defined in `./ore.js`, which owns the + * whole ORE model — the catalogue probe, the state machine, and the copy every + * command renders — and re-exported here because `VerifyReport` carries it. */ -export type OreSurfaceState = - | 'indexable' - | 'fallback' - | 'incoherent-unpoisoned' - | 'incoherent-poisoned' +export type { OreSurfaceState } export interface SurfaceFinding { severity: 'damage' | 'warning' | 'expected' @@ -385,14 +386,7 @@ const CASTS_SQL = ` */ const ORE_STATE_SQL = ` SELECT - EXISTS ( - SELECT 1 - FROM pg_catalog.pg_opclass c - JOIN pg_catalog.pg_am am ON am.oid = c.opcmethod - WHERE am.amname = 'btree' - AND c.opcdefault - AND c.opcintype = to_regtype('${EQL_V3_INTERNAL_SCHEMA_NAME}.ore_block_256') - ) AS ore_opclass_present, + ${ORE_OPCLASS_PRESENT_EXPR} AS ore_opclass_present, ( SELECT count(*)::int FROM pg_catalog.pg_constraint c @@ -722,48 +716,22 @@ export function diffSurface( // The ORE conditional: exactly one of the two halves must be present, in // full. Anything else is a half-working state the bundle never produces. const expectedPoisoned = expected.oreDomains.length - let oreState: OreSurfaceState - if (installed.oreOpclassPresent) { - oreState = - installed.poisonedDomains === 0 ? 'indexable' : 'incoherent-poisoned' - } else { - oreState = - installed.poisonedDomains === expectedPoisoned - ? 'fallback' - : 'incoherent-unpoisoned' - } - switch (oreState) { - case 'indexable': - findings.push({ - severity: 'expected', - kind: 'opclass', - message: - 'ORE operator class present — ORE ordered indexes are available (superuser install).', - }) - break - case 'fallback': - findings.push({ - severity: 'expected', - kind: 'opclass', - message: - 'ORE operator class absent, and every ORE domain carries the loud-failure fallback. This is the supported managed-Postgres configuration (creating the class requires superuser), not a failed install — use the `_ord_ope` ordering domains.', - }) - break - case 'incoherent-poisoned': - findings.push({ - severity: 'damage', - kind: 'opclass', - message: `The ORE operator class exists, but ${installed.poisonedDomains} domain${installed.poisonedDomains === 1 ? ' still carries' : 's still carry'} the \`eql_ore_unavailable\` poison CHECK — writes to those domains fail although ORE works. Reinstall with \`stash eql install --force\`.`, - }) - break - case 'incoherent-unpoisoned': - findings.push({ - severity: 'damage', - kind: 'opclass', - message: `The ORE operator class is absent, but only ${installed.poisonedDomains} of ${expectedPoisoned} ORE domains carry the loud-failure fallback — the rest would fail at index/ORDER BY time with opaque errors instead. Reinstall with \`stash eql install --force\`.`, - }) - break - } + const oreState = classifyOreState({ + opclassPresent: installed.oreOpclassPresent, + poisonedDomains: installed.poisonedDomains, + expectedPoisoned, + }) + const oreDescription = describeOreState(oreState) + findings.push({ + severity: oreDescription.severity === 'damage' ? 'damage' : 'expected', + kind: 'opclass', + // The counts only mean anything in the two incoherent states, where they + // say how far the half-application got. + message: + oreDescription.severity === 'damage' + ? `${oreDescription.message} (${installed.poisonedDomains} of ${expectedPoisoned} ORE domains carry the poison CHECK.)` + : oreDescription.message, + }) const damaged = findings.some((finding) => finding.severity === 'damage') return { @@ -839,3 +807,33 @@ export async function verifyEqlSurface( await client.end() } } + +/** + * Read just the ORE half of an install — the two catalogue values and the + * state they classify to (#891). + * + * `eql status` wants the ORE answer and nothing else. Routing it through + * {@link verifyEqlSurface} would work but would read the whole 3,000-operator + * surface to render one row, and would report a version mismatch as a reason + * to say nothing — whereas the ORE state is legible whatever bundle is + * installed, because both halves of the conditional are catalogue facts rather + * than a diff against the pinned manifest. + */ +export async function readOreState(client: pg.ClientBase): Promise<{ + opclassPresent: boolean + poisonedDomains: number + expectedPoisoned: number + state: OreSurfaceState +}> { + const expected = bundledExpectedSurface() + const result = await client.query<{ + ore_opclass_present: boolean + poisoned_domains: number + }>(ORE_STATE_SQL, [expected.oreDomains]) + const observed = { + opclassPresent: result.rows[0]?.ore_opclass_present === true, + poisonedDomains: result.rows[0]?.poisoned_domains ?? 0, + expectedPoisoned: expected.oreDomains.length, + } + return { ...observed, state: classifyOreState(observed) } +} diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 9973c7e62..516ec77e3 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -363,10 +363,12 @@ stash eql status #### `eql preflight` -Read-only report of whether the connected role can install EQL, run before anything is attempted. It probes: `current_user`, superuser, **membership of `postgres`**, `CREATE` on the database and on `public`, `pgcrypto`, and whether the `eql_v3` / `eql_v3_internal` schemas already exist. Each blocked row names the statement it blocks. Exits 1 when a gap would abort `eql install`; `--json` emits the structured result for agents (stdout is pure JSON). +Read-only report of whether the connected role can install EQL, run before anything is attempted. It probes: `current_user`, superuser, **membership of `postgres`**, `CREATE` on the database and on `public`, `pgcrypto`, whether the role **can create an operator class**, and whether the `eql_v3` / `eql_v3_internal` schemas already exist. Each blocked row names the statement it blocks. Exits 1 when a gap would abort `eql install`; `--json` emits the structured result for agents (stdout is pure JSON). Membership of `postgres` is reported but never blocks: `eql install` handles a non-member role by skipping the owner-scoped `ALTER DEFAULT PRIVILEGES FOR ROLE postgres` statements, which are **optional** — the install is complete without them (see `eql install` below). This matters on managed platforms whose database role is not `postgres` and not a member of it (e.g. Lovable's `sandbox_exec`). +**The `ORE operator class` row never blocks either** (`canCreateOperatorClass` in `--json`; `creatable` / `not creatable` / `unknown`). It answers the one schema-design question you want settled *before* writing types: `not creatable` means `eql install` will skip the ORE operator class and install the bundle's loud-failure fallback in its place, so **declare ordered columns `types.*Ord`, not `types.*OrdOre`** — every write to an `_ord_ore` domain on such a database fails its `eql_ore_unavailable` CHECK. This is probed, not inferred from `superuser`: `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, but AWS RDS and Aurora let their admin role create one while cloud-hosted Supabase does not — so a role with `rolsuper = f` is not evidence either way. The probe attempts the DDL inside a transaction it always rolls back, so preflight stays read-only; `unknown` means it could not ask (a read-only replica, say) and must not be read as either answer. + | Flag | Description | |---|---| | `--json` | Machine-readable result instead of the table. Stdout is pure JSON even on failure: success is `{ status: 'ok', ... }`, blockers are `{ status: 'blocked', ... }` (exit 1), and failures — including a missing/malformed DATABASE_URL — are the shared `{ status: 'error', code, message }` envelope | @@ -492,6 +494,8 @@ The install SQL is safe to re-run — columns and data survive — but it cascad Whether EQL is installed and at which version, plus database permission status. It retains read-only EQL v2/config-table diagnostics for existing deployments. +On a v3 install it also reports the **ORE operator class** state, so the ordering trade `eql install` named once is recoverable later without re-reading scrollback: either the class is present (`types.*OrdOre` usable), or it was skipped and every `_ord_ore` domain carries the loud-failure fallback — the supported managed-Postgres configuration, where ordered columns must be `types.*Ord`. Anything else is damage and points at `eql install --force`. Run `eql verify` for the full surface check. + #### `eql validate` — validate the encryption schema ```bash