skills: add the EQL v3 type → predicate → domain → index capability matrix

.changeset/skills-type-capability-matrix.md

@@ -0,0 +1,16 @@
+---
+'stash': patch
+---
+
+Add a type → predicate → domain → index capability matrix to the `stash-encryption` skill, cross-linked from `stash-indexing` and `stash-postgres`.
+
+Picking the wrong `types.*` factory is silent at authoring time — there is no type error and no runtime warning, just a predicate that never runs. The skills documented the capability *suffixes* and the families they apply to, but never the 40 concrete factories in one lookup, so answering "can `types.Double` do a range query" meant composing two tables and knowing the exceptions. It cannot: `types.Double` is storage-only.
+
+The new matrix has one row per factory with its Postgres column domain, the predicates it supports, the extractor to index it through, and whether it works on managed Postgres. Alongside it: a note on which schema holds what (`public` for column domains, `eql_v3` for query domains and operator functions, `eql_v3_internal` for index-term types) and why the Supabase grants have to cover the last two.
+
+Two corrections came out of writing it:
+
+- The `Ord` vs `OrdOre` callout said the install "disables the `_ord_ore` domains" on managed Postgres. Precisely: the bundle adds an always-raising `eql_ore_unavailable` CHECK to them, so a *write* fails — the domain is unusable, not merely unindexed. The callout now says that, notes that RDS and Aurora do support ORE while cloud-hosted Supabase does not, and points at `stash eql preflight` / `eql status` rather than asking the reader to guess.
+- The `stash-postgres` naming table omitted `types.TextOrdOre` entirely (its `<N>` shorthand covers only the numeric and temporal families). Added.
+
+A new test derives the matrix from the `types` namespace and fails if the skill disagrees — every factory present exactly once, mapped to the domain it actually builds, naming the extractors it actually emits and none it does not, with every ORE row marked unusable where the operator class is absent.

packages/stack/__tests__/eql-v3-capability-matrix-skill.test.ts

@@ -0,0 +1,186 @@
+import { readFileSync } from 'node:fs'
+import { dirname, resolve } from 'node:path'
+import { fileURLToPath } from 'node:url'
+import { describe, expect, it } from 'vitest'
+import { types } from '@/eql/v3'
+
+/**
+ * The capability matrix in `skills/stash-encryption/SKILL.md` must stay true
+ * to the `types` namespace (#892).
+ *
+ * The matrix exists because picking the wrong factory is silent at authoring
+ * time — no type error, no runtime warning, just a predicate that never runs.
+ * A table documenting that, which has itself drifted, is worse than no table:
+ * it converts a discoverable gap into confident wrong guidance, and it ships
+ * inside the `stash` tarball into customers' repos.
+ *
+ * Deriving the matrix at build time was the alternative. This repo has no
+ * markdown codegen and the skills are hand-authored prose around their tables,
+ * so a generator would own a fragment of a file humans edit. Pinning is the
+ * pattern already used for skill content (`skill-supabase-apply.test.ts`), and
+ * it fails in the same place a generator would: the moment source and prose
+ * disagree.
+ *
+ * What is checked is the mechanical half — every factory present, mapped to
+ * the domain the factory actually builds, with the index kinds it actually
+ * emits. The prose around it is a human's job.
+ */
+
+const SKILL = resolve(
+  dirname(fileURLToPath(import.meta.url)),
+  '../../../skills/stash-encryption/SKILL.md',
+)
+
+/** Only the matrix, so an example elsewhere in the file cannot satisfy a row. */
+function matrixRows(): string[] {
+  const body = readFileSync(SKILL, 'utf-8')
+  const start = body.indexOf('#### The capability matrix')
+  expect(start, 'the capability matrix heading is missing').toBeGreaterThan(-1)
+  const end = body.indexOf('That is the whole surface', start)
+  expect(end, 'the matrix closing paragraph is missing').toBeGreaterThan(start)
+  return body
+    .slice(start, end)
+    .split('\n')
+    .filter((line) => line.trimStart().startsWith('| `types.'))
+}
+
+/** The index kinds a factory emits, e.g. `['ope', 'unique']`. */
+function indexKindsFor(name: keyof typeof types): string[] {
+  const factory = types[name] as (column: string) => {
+    build: () => { indexes?: Record<string, unknown> }
+  }
+  return Object.keys(factory('__probe__').build().indexes ?? {}).sort()
+}
+
+function eqlTypeFor(name: keyof typeof types): string {
+  const factory = types[name] as (column: string) => {
+    getEqlType: () => string
+  }
+  return factory('__probe__').getEqlType()
+}
+
+const FACTORY_NAMES = Object.keys(types) as Array<keyof typeof types>
+
+/**
+ * The extractor each index kind is indexed through — the "What to index"
+ * column. `ste_vec` is the one that is not a scalar extractor.
+ */
+const EXTRACTOR_FOR_INDEX: Record<string, string> = {
+  unique: 'eq_term',
+  ope: 'ord_term',
+  ore: 'ord_term_ore',
+  match: 'match_term',
+  ste_vec: 'to_ste_vec_query',
+}
+
+/**
+ * Matched in its backticked form, because `ord_term` is a substring of
+ * `ord_term_ore` — a bare substring test reads an ORE row as also naming the
+ * OPE extractor and the negative assertions below become unsatisfiable.
+ */
+const backticked = (extractor: string) => `\`${extractor}\``
+
+describe('the stash-encryption capability matrix', () => {
+  it('has exactly one row per factory, and no rows for anything else', () => {
+    const rows = matrixRows()
+    const listed = rows.map((row) => {
+      const match = /^\|\s*`types\.(\w+)\(/.exec(row.trimStart())
+      expect(match, `could not read a factory name from row: ${row}`).not.toBe(
+        null,
+      )
+      return (match as RegExpExecArray)[1]
+    })
+    expect([...listed].sort()).toEqual([...FACTORY_NAMES].sort())
+    // No duplicate rows: a factory documented twice can be documented two
+    // different ways.
+    expect(new Set(listed).size).toBe(listed.length)
+  })
+
+  it('names the domain each factory actually builds', () => {
+    const rows = matrixRows()
+    for (const name of FACTORY_NAMES) {
+      const row = rows.find((candidate) =>
+        candidate.trimStart().startsWith(`| \`types.${name}(`),
+      )
+      expect(row, `no matrix row for types.${name}`).toBeDefined()
+      expect(
+        row,
+        `types.${name} builds ${eqlTypeFor(name)}, which its matrix row does not name`,
+      ).toContain(`\`${eqlTypeFor(name)}\``)
+    }
+  })
+
+  /**
+   * The trap the issue was filed about: `types.Double` mints no ORE blocks and
+   * answers no predicate, but its name suggests otherwise. Every storage-only
+   * factory must say so in its own row.
+   */
+  it('marks every storage-only factory as answering nothing', () => {
+    const rows = matrixRows()
+    for (const name of FACTORY_NAMES) {
+      if (indexKindsFor(name).length > 0) continue
+      const row = rows.find((candidate) =>
+        candidate.trimStart().startsWith(`| \`types.${name}(`),
+      )
+      expect(
+        row,
+        `types.${name} emits no index terms, so its row must say storage only`,
+      ).toContain('storage only')
+    }
+  })
+
+  /**
+   * The "What to index" column must name the extractor the column's terms are
+   * actually reachable through — indexing an extractor a domain has no
+   * overload for builds an index that never engages.
+   */
+  it('names the right extractor for every queryable factory', () => {
+    const rows = matrixRows()
+    for (const name of FACTORY_NAMES) {
+      const kinds = indexKindsFor(name)
+      if (kinds.length === 0) continue
+      const row = rows.find((candidate) =>
+        candidate.trimStart().startsWith(`| \`types.${name}(`),
+      )
+      // Rows that defer to a sibling ("as `IntegerOrd`") repeat the ORE
+      // extractor but not the shared predicate list, so only assert the
+      // extractors, which every row carries in full.
+      for (const kind of kinds) {
+        expect(
+          row,
+          `types.${name} emits a '${kind}' index, so its row must name ${EXTRACTOR_FOR_INDEX[kind]}`,
+        ).toContain(backticked(EXTRACTOR_FOR_INDEX[kind]))
+      }
+      // ...and must NOT name an extractor the domain has no overload for. The
+      // numeric `_ord` domains are the live case: they answer `=` through the
+      // injective ordering term and define no `eq_term`.
+      for (const [kind, extractor] of Object.entries(EXTRACTOR_FOR_INDEX)) {
+        if (kinds.includes(kind)) continue
+        expect(
+          row,
+          `types.${name} emits no '${kind}' index, so its row must not name ${extractor}`,
+        ).not.toContain(backticked(extractor))
+      }
+    }
+  })
+
+  /**
+   * Every ORE factory is unusable where the operator class could not be
+   * created — the bundle poisons those domains with an always-raising CHECK,
+   * so a write fails rather than an index quietly not engaging. The matrix
+   * must not soften that into "usable but unindexable".
+   */
+  it('marks every ORE factory as unusable on a database without the opclass', () => {
+    const rows = matrixRows()
+    for (const name of FACTORY_NAMES) {
+      if (!indexKindsFor(name).includes('ore')) continue
+      const row = rows.find((candidate) =>
+        candidate.trimStart().startsWith(`| \`types.${name}(`),
+      )
+      expect(row, `types.${name} is ORE-backed; its row must say so`).toContain(
+        'privileged install only',
+      )
+      expect(row).toContain('unusable')
+    }
+  })
+})

skills/stash-encryption/SKILL.md

@@ -223,7 +223,74 @@ The returned table is also a column accessor (`users.email`). The JS property na
 
 ### The `types` Namespace
 
-Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_<name>`. The naming rule: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`, and `types.Timestamp` builds `public.eql_v3_timestamp`.
+Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_<name>`. The naming rule: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`, and `types.Timestamp` builds `public.eql_v3_timestamp`. **One exception:** `types.Json` builds `public.eql_v3_json_search`, not `eql_v3_json` (and its query-operand domain is `eql_v3.query_json`, not `query_json_search`).
+
+#### The capability matrix
+
+Picking the wrong factory is **silent at authoring time**. There is no type error and no runtime warning — the predicate you wanted simply will not run, and you find out when a query errors or returns nothing. Look your type up here before writing the column.
+
+| SDK factory | Postgres column domain | Predicates it supports | What to index | Managed Postgres |
+|---|---|---|---|---|
+| `types.Text(...)` | `public.eql_v3_text` | none — storage only | — | ✅ |
+| `types.TextEq(...)` | `public.eql_v3_text_eq` | `=` `<>` `IN` | `eq_term` (HMAC btree) | ✅ |
+| `types.TextMatch(...)` | `public.eql_v3_text_match` | `@@` free-text `matches` only | `match_term` (bloom GIN) | ✅ |
+| `types.TextOrd(...)` | `public.eql_v3_text_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `eq_term` **+** `ord_term` (two indexes) | ✅ |
+| `types.TextOrdOre(...)` | `public.eql_v3_text_ord_ore` | as `TextOrd` | `eq_term` **+** `ord_term_ore` — ORE opclass, **privileged install only** | ⛔ unusable — see below |
+| `types.TextSearch(...)` | `public.eql_v3_text_search` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` **and** `@@` | `eq_term` + `ord_term` + `match_term` (three) | ✅ |
+| `types.Integer(...)` | `public.eql_v3_integer` | none — storage only | — | ✅ |
+| `types.IntegerEq(...)` | `public.eql_v3_integer_eq` | `=` `<>` `IN` | `eq_term` (HMAC btree) | ✅ |
+| `types.IntegerOrd(...)` | `public.eql_v3_integer_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` (OPE btree) — one index serves all | ✅ |
+| `types.IntegerOrdOre(...)` | `public.eql_v3_integer_ord_ore` | as `IntegerOrd` | `ord_term_ore` — ORE opclass, **privileged install only** | ⛔ unusable — see below |
+| `types.Smallint(...)` | `public.eql_v3_smallint` | none — storage only | — | ✅ |
+| `types.SmallintEq(...)` | `public.eql_v3_smallint_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.SmallintOrd(...)` | `public.eql_v3_smallint_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.SmallintOrdOre(...)` | `public.eql_v3_smallint_ord_ore` | as `SmallintOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Bigint(...)` | `public.eql_v3_bigint` | none — storage only | — | ✅ |
+| `types.BigintEq(...)` | `public.eql_v3_bigint_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.BigintOrd(...)` | `public.eql_v3_bigint_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.BigintOrdOre(...)` | `public.eql_v3_bigint_ord_ore` | as `BigintOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Numeric(...)` | `public.eql_v3_numeric` | none — storage only | — | ✅ |
+| `types.NumericEq(...)` | `public.eql_v3_numeric_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.NumericOrd(...)` | `public.eql_v3_numeric_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.NumericOrdOre(...)` | `public.eql_v3_numeric_ord_ore` | as `NumericOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Real(...)` | `public.eql_v3_real` | none — storage only | — | ✅ |
+| `types.RealEq(...)` | `public.eql_v3_real_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.RealOrd(...)` | `public.eql_v3_real_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.RealOrdOre(...)` | `public.eql_v3_real_ord_ore` | as `RealOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Double(...)` | `public.eql_v3_double` | **none — storage only** | — | ✅ |
+| `types.DoubleEq(...)` | `public.eql_v3_double_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.DoubleOrd(...)` | `public.eql_v3_double_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.DoubleOrdOre(...)` | `public.eql_v3_double_ord_ore` | as `DoubleOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Date(...)` | `public.eql_v3_date` | none — storage only | — | ✅ |
+| `types.DateEq(...)` | `public.eql_v3_date_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.DateOrd(...)` | `public.eql_v3_date_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.DateOrdOre(...)` | `public.eql_v3_date_ord_ore` | as `DateOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Timestamp(...)` | `public.eql_v3_timestamp` | none — storage only | — | ✅ |
+| `types.TimestampEq(...)` | `public.eql_v3_timestamp_eq` | `=` `<>` `IN` | `eq_term` | ✅ |
+| `types.TimestampOrd(...)` | `public.eql_v3_timestamp_ord` | `=` `<>` `<` `<=` `>` `>=` `ORDER BY` | `ord_term` | ✅ |
+| `types.TimestampOrdOre(...)` | `public.eql_v3_timestamp_ord_ore` | as `TimestampOrd` | `ord_term_ore` — **privileged install only** | ⛔ unusable |
+| `types.Boolean(...)` | `public.eql_v3_boolean` | none — storage only | — | ✅ |
+| `types.Json(...)` | `public.eql_v3_json_search` | `@>` containment + JSONPath selectors | `to_ste_vec_query` (GIN) | ✅ |
+
+That is the whole surface — 40 factories, no others. Three things the table is trying to make unmissable:
+
+1. **A bare factory name is storage-only.** `types.Double` encrypts and decrypts and answers *nothing*. If you want to compare it, you wanted `types.DoubleOrd`; if you want ORE blocks specifically, `types.DoubleOrdOre`. The same holds for every family.
+2. **On the numeric and temporal families, one ordering index serves everything.** Their ordering term is injective (distinct plaintexts give distinct terms), so `=` rides it — there is deliberately **no `eq_term` overload** for those domains, and adding an equality index to a `_ord` numeric column indexes a function that does not exist. Text ordering terms are *not* injective, which is why `text_ord` / `text_ord_ore` / `text_search` carry `eq_term` **as well**.
+3. **`Ord` and `OrdOre` are not interchangeable.** They mint different, non-cross-comparable terms (`eql_v3.ord_term` vs `eql_v3.ord_term_ore`), so you cannot switch one for the other without re-encrypting the column.
+
+`ORDER BY` needs the extractor form (`ORDER BY eql_v3.ord_term(col)`); the ORM integrations emit it for you. The `CREATE INDEX` statements behind the "What to index" column are in the `stash-indexing` skill, and the raw-SQL predicate forms with their `eql_v3.query_*` operand casts are in `stash-postgres`.
+
+#### Where these objects live
+
+Three schemas, and knowing which is which resolves most "function does not exist" errors:
+
+| Schema | Holds | Example |
+|---|---|---|
+| `public` | the **column storage domains** — what you declare a column as | `public.eql_v3_double_ord` |
+| `eql_v3` | the **query-operand domains** and the operator/extractor functions | `eql_v3.query_double_ord`, `eql_v3.ord_term` |
+| `eql_v3_internal` | the **index-term types** and their operators | `eql_v3_internal.ore_block_256` |
+
+So a single encrypted comparison touches all three: a `public` column, cast against an `eql_v3` query domain, comparing `eql_v3_internal` term types. That is also why the Supabase role grants cover `eql_v3` **and** `eql_v3_internal` — granting only the first leaves `anon` / `authenticated` / `service_role` unable to execute the internal term operators the public ones inline to. The `eql_v3` schemas are dropped and recreated by every install; `public` is not, which is why your column domains survive a reinstall.
 
 **Capability suffixes:**
 
@@ -237,7 +304,13 @@ Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_<name
 | `Search` (text only) | Equality + ordering/range + free-text | all three |
 | `Json` (no suffix) | Encrypted-JSONB containment + JSONPath selector queries | `'searchableJson'` |
 
-> **`Ord` vs `OrdOre`:** prefer `Ord`. The `OrdOre` domains are backed by an ORE operator class the installer creates with the superuser-gated `CREATE OPERATOR CLASS`. Platform support varies — AWS RDS and Aurora allow it; cloud-hosted Supabase does not (the one confirmed platform that refuses it), and there the install bundle skips the ORE opclass and disables the `_ord_ore` domains it cannot support. The two ordering flavours produce different, non-cross-comparable terms (`Ord`/`Search` extract via `eql_v3.ord_term`; `OrdOre` via `eql_v3.ord_term_ore`).
+> **`Ord` vs `OrdOre` — prefer `Ord`.** The `OrdOre` domains are backed by an ORE operator class the installer creates with `CREATE OPERATOR CLASS`, which is superuser-gated in stock PostgreSQL. Platform support varies and is **not** a blanket managed-Postgres rule: AWS RDS and Aurora allow it (their admin role clears the gate despite `rolsuper = f`); cloud-hosted Supabase is the one confirmed platform that refuses it.
+>
+> Where the install role cannot create the class, the bundle skips it **and adds an always-raising `eql_ore_unavailable` CHECK to every `_ord_ore` domain**, so every write to such a column fails loudly rather than producing an index that silently never engages. On those databases an `OrdOre` column is unusable, not merely unindexed — which is why the matrix marks it ⛔ rather than ⚠️.
+>
+> Don't guess which case you are in — ask: `stash eql preflight` predicts it before you install (the `ORE operator class` row), and `stash eql status` / `stash eql verify` report it afterwards. `Ord` is OPE-backed, binds PostgreSQL's native `bytea` btree operator class, and needs no privileges anywhere.
+>
+> The two flavours mint different, non-cross-comparable terms (`Ord`/`Search` extract via `eql_v3.ord_term`; `OrdOre` via `eql_v3.ord_term_ore`), so switching between them means re-encrypting the column.
 
 **Domain families and plaintext types:**