feat(spec,metadata-protocol): a stored filter the record-filter conversion leaves as stored is a TODO in os migrate meta --stored, not silence (#17321)

.changeset/17321-conversion-todo-channel.md

@@ -0,0 +1,50 @@
+---
+"@objectstack/spec": minor
+"@objectstack/metadata-protocol": minor
+---
+
+feat(spec,metadata-protocol): `os migrate meta --stored` lists every stored page filter the record-filter conversion leaves as stored, as a TODO naming the page, the block and why — the ADR-0087 D3 TODO channel (#17321, ruling B item 2)
+
+**Clause-②: yes** — `@objectstack/spec` gains public exports (`CONVERSION_TODO_CODE`,
+`ConversionTodoDetail`, `ConversionTodoNotice`, and the optional `ApplyConversionsOptions.onTodo`
+and `ConversionContext.reportTodo`), and `@objectstack/metadata-protocol` gains
+`StoredMigrationTodo` and `StoredMigrationRow.todos`. No door's accept set moves, and nothing
+that was left as stored before starts converting: every stored body is rewritten exactly as it
+was.
+
+**What was silent.** The D2 conversion `page-component-filter-record-to-rule-array` leaves a
+stored filter as stored wherever no lossless rule-array spelling exists — above all a record
+carrying `$and` / `$or` / `$not`, which is never flattened. It emitted nothing for such a site,
+and `os migrate meta --stored` reads conversion notices as its change signal, so a page whose
+only legacy filter carried a combinator was reported as **already on protocol**.
+
+**What it says now.** Each such site is a structured TODO (code `OS_METADATA_CONVERSION_TODO`)
+carrying its path, the shape left in place, and a reason that names the block (its type, and its
+`id` when it has one) and what blocks the rewrite — the combinator by name, the operator
+(`$null`, `$exists`, an AST `like`), the null or array value, the rule the door would refuse, or
+the inline rows the block renders. The stored pass lists them under their row, whatever the
+row's outcome:
+
+```text
+⚠ 1 row(s) are outside this pass — each row's reason says why:
+  • page/pipeline_board [env-wide] — the conversion chain rewrites nothing here: it left 1 site(s) of this row as stored, …
+      TODO page-component-filter-record-to-rule-array: {"$or":[…]} left as stored at pages[0].regions[0].components[0].properties.filter — On the `object-kanban` block, this filter carries the combinator `$or`: …
+☐ TODO: 1 site(s) in 1 row(s) are left as stored — no conversion can rewrite them without changing what they mean, so no run of this pass will. …
+```
+
+The same list is `rows[].todos` in `--json` and in the `POST /api/v1/meta/_migrate-stored`
+report. A run with no TODO prints exactly what it printed before.
+
+**Outcome and exit code.** A row whose only finding is TODOs has nothing to persist and is now
+reported `skipped` (it was `canonical`). Like every other skip class it does not change the
+run's exit code: no run of this pass can clear it, because the conversion must not flatten a
+combinator — it is the hand rewrite's to decide. A row that also converts something keeps the
+outcome its conversion gives it, with its TODOs listed beside its notices. Measured through the
+write path: on `--apply`, a row whose leftover sits in a block's `properties.filter` or
+`properties.defaultFilters` is rewritten (its lossless filters persist; the metadata API's save
+does not refuse block props by component type), while a leftover in `dataSource.filter` fails
+the save, and the row's TODO says why.
+
+**For code calling the conversion layer.** `onTodo` and `reportTodo` are optional. Only the
+stored-metadata pass passes a sink today; every other seam leaves the site as stored silently,
+exactly as before.

.changeset/17321-record-filter-d2-conversion.md

@@ -1,3 +1,3 @@
 ---
 "@objectstack/spec": minor
 ---
@@ -45,8 +45,11 @@
 stored: the `object-map`, `object-tree`, `object-calendar` and `object-gantt` renderers match
 that filter against their own rows in an in-memory data source that reads the record form but
 excludes every row for a rule array, so a rewrite there would empty the block. The same filter
-on a block that queries an object converts. Such a page keeps loading and rendering unchanged and is refused at its
-`filter` door on its next save — and for a combinator record that refusal no longer renders the
+on a block that queries an object converts. Such a page keeps loading and rendering unchanged,
+and its `filter` door refuses the form: at `dataSource.filter` on the page's next save; at a
+block's `properties.filter` / `properties.defaultFilters` only as the component-props gate's
+advisory finding (`os validate`, `os build`, `os lint`) — a re-save through the metadata API is
+not refused there, measured — and for a combinator record that refusal no longer renders the
 combinator as a field (`{ field: '$or', … }`); it names the combinator and says why no rule
-spells it. `os migrate meta --stored` does not list these rows yet: a row the conversion leaves
-as stored reports there as already on protocol.
+spells it.
+`os migrate meta --stored` lists each such filter left as stored as a TODO under its row.

content/docs/deployment/cli.mdx

@@ -1196,6 +1196,7 @@ can produce, because both supply a live one.
 | Types with no repository write path (`agent`) | Their write path records no history and would force a draft live — a half-write is worse than leaving the row to the read path |
 | Rows that still fail the current schema after conversion | That is a genuine contract violation, not chain-owned history. The write path's rejection is correct; fix the row in Studio |
 | A flow whose rename the conflict guard refused | The old node-type token is a live name something else owns here. Rewriting would clobber that owner, so the row fails loudly naming the token — never a silent skip |
+| A site the conversion chain leaves as stored because no lossless rewrite exists — above all a page filter carrying `$and` / `$or` / `$not` | Flattening a combinator changes which rows the page selects, so it is never done. Each site is printed as a `TODO` line under its row — path, block, and what blocks the rewrite — whatever the row's outcome; a row with nothing but TODOs is reported `skipped`. It does not fail the run, since no run of this pass can clear it: rewrite each site by hand |
 
 **Flows are covered, and cost one extra plugin.** Flow-node conversions carry an
 open-namespace conflict guard that has to consult the *live* executor registry
@@ -1222,8 +1223,8 @@ something the platform can depend on. What running it buys is hygiene (cleaner
 diffs, exports and history from here on, and the recurring boot notices go
 quiet) plus one thing that was previously unobtainable: **you can assert it.**
 A run with nothing left to do exits `0`; a deployment with rows still carrying
-an old dialect exits `1`. So "my metadata is on protocol N" becomes a check
-rather than a belief.
+an old dialect this pass can convert exits `1`. So "my metadata is on protocol N"
+becomes a check rather than a belief.
 
 Note the division of labour with the default mode: `os migrate meta --from N`
 lists the edits **an author's source** needs and reads no database; `--stored`

packages/metadata-protocol/src/index.ts

@@ -164,6 +164,7 @@ export type {
   StoredMigrationOutcome,
   StoredMigrationReport,
   StoredMigrationRow,
+  StoredMigrationTodo,
 } from './stored-migration.js';
 
 export {

packages/metadata-protocol/src/protocol.stored-migration.test.ts

@@ -626,3 +626,180 @@ describe('formatStoredMigrationReport (#4327)', () => {
         expect(text).toMatch(/conditionalRequired → requiredWhen/);
     });
 });
+
+describe('migrateStoredMetadata — a site the chain leaves as stored is a TODO, not silence (#17321)', () => {
+    // Ruling item 2 (decision batch #121 item 4, B): a stored page filter
+    // carrying `$and` / `$or` / `$not` is passed through unchanged and
+    // reported as a structured TODO naming the page/block and the combinator —
+    // `os migrate meta --stored` prints the list, so the operator can answer
+    // "did it convert my row" from that output.
+    //
+    // Before the TODO lane existed, the conversion emitted NO notice for such a
+    // site, this pass reads notices as its change signal, and so a page whose
+    // only legacy filter carried a combinator was counted `canonical` —
+    // "already on protocol" about a row whose next save is refused.
+    const COMBINATOR = { $or: [{ stage: 'open' }, { stage: 'won' }] };
+    const pageRow = (name: string, components: unknown[]) => ({
+        type: 'page',
+        name,
+        metadata: { name, label: 'Pipeline', type: 'app', regions: [{ name: 'main', components }] },
+    });
+    const kanban = (filter: unknown) => ({ type: 'object-kanban', properties: { objectName: 'deal', filter } });
+    const grid = (filter: unknown) => ({ type: 'object-grid', properties: { objectName: 'deal', filter } });
+    const combinatorPage = pageRow('pipeline_board', [kanban(COMBINATOR)]);
+    const losslessPage = pageRow('open_deals', [grid({ stage: 'open' })]);
+    const mixedPage = pageRow('deal_desk', [grid({ stage: 'open' }), kanban(COMBINATOR)]);
+    const canonicalPage = pageRow('won_deals', [grid([{ field: 'stage', operator: 'equals', value: 'won' }])]);
+
+    it('a combinator-only page is listed with a TODO naming its path and the combinator — not counted canonical', async () => {
+        const { engine, tables } = makeStubEngine([combinatorPage]);
+        const before = JSON.stringify(metaRows(tables));
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const report = await protocol.migrateStoredMetadata();
+
+        expect(report.canonical).toBe(0);
+        expect(report.skipped).toBe(1);
+        expect(report.rows).toHaveLength(1);
+        const row = report.rows[0]!;
+        expect(row).toMatchObject({ type: 'page', name: 'pipeline_board', outcome: 'skipped', notices: [] });
+        expect(row.reason).toMatch(/left 1 site\(s\) of this row as stored/);
+        expect(row.todos).toHaveLength(1);
+        const todo = row.todos[0]!;
+        expect(todo.conversionId).toBe('page-component-filter-record-to-rule-array');
+        expect(todo.path).toBe('pages[0].regions[0].components[0].properties.filter');
+        expect(todo.from).toBe(JSON.stringify(COMBINATOR));
+        expect(todo.reason).toContain('the `object-kanban` block');
+        expect(todo.reason).toContain('the combinator `$or`');
+        // A TODO is reporting only: nothing written, and the filter is never flattened.
+        expect(JSON.stringify(metaRows(tables))).toBe(before);
+
+        // The operator reads it off the rendered report: the row, the path, the combinator.
+        const text = formatStoredMigrationReport(report).join('\n');
+        expect(text).toContain('page/pipeline_board [env-wide]');
+        expect(text).toContain(`TODO page-component-filter-record-to-rule-array: ${JSON.stringify(COMBINATOR)} left as stored at pages[0].regions[0].components[0].properties.filter`);
+        expect(text).toContain('`$or`');
+        expect(text).toMatch(/☐ TODO: 1 site\(s\) in 1 row\(s\) are left as stored/);
+        // …and is never told the opposite in the same breath.
+        expect(text).not.toMatch(/already on protocol/);
+    });
+
+    it('does not flip `storedMigrationClean` — a skip class this pass has no lever for, by ruling', async () => {
+        const { engine, tables } = makeStubEngine([combinatorPage]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const preview = await protocol.migrateStoredMetadata();
+        expect(storedMigrationClean(preview)).toBe(true);
+
+        // An apply run writes nothing for it either, and says the same thing.
+        const applied = await protocol.migrateStoredMetadata({ apply: true });
+        expect(applied.rows[0]).toMatchObject({ outcome: 'skipped' });
+        expect(applied.rows[0]!.todos).toHaveLength(1);
+        expect(storedMigrationClean(applied)).toBe(true);
+        expect(historyRows(tables)).toHaveLength(0);
+    });
+
+    it('CONTROL — a losslessly folded filter produces no TODO, and its outcome is unchanged', async () => {
+        const { engine, tables } = makeStubEngine([losslessPage]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const preview = await protocol.migrateStoredMetadata();
+        expect(preview.rows).toHaveLength(1);
+        expect(preview.rows[0]).toMatchObject({ outcome: 'pending', todos: [] });
+        expect(preview.rows[0]!.notices.map((n) => n.path)).toEqual([
+            'pages[0].regions[0].components[0].properties.filter',
+        ]);
+        expect(formatStoredMigrationReport(preview).join('\n')).not.toMatch(/TODO/);
+
+        const applied = await protocol.migrateStoredMetadata({ apply: true });
+        expect(applied.rows[0]).toMatchObject({ outcome: 'rewritten', todos: [] });
+        expect(storedMigrationClean(applied)).toBe(true);
+        const stored = JSON.parse(metaRows(tables)[0]!.metadata);
+        expect(stored.regions[0].components[0].properties.filter).toEqual([
+            { field: 'stage', operator: 'equals', value: 'open' },
+        ]);
+    });
+
+    it('CONTROL — an already-canonical page is counted, never itemised, and reports no TODO', async () => {
+        const { engine } = makeStubEngine([canonicalPage]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const report = await protocol.migrateStoredMetadata();
+
+        expect(report.canonical).toBe(1);
+        expect(report.rows).toHaveLength(0);
+        const text = formatStoredMigrationReport(report).join('\n');
+        expect(text).toMatch(/already on protocol/);
+        expect(text).not.toMatch(/TODO/);
+    });
+
+    it('one row, two filters: the lossless one converts, the combinator one is a TODO on the same row', async () => {
+        const { engine } = makeStubEngine([mixedPage]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const report = await protocol.migrateStoredMetadata();
+
+        // The outcome is what the notice alone makes it — TODOs never move it.
+        expect(report.pending).toBe(1);
+        expect(storedMigrationClean(report)).toBe(false);
+        const row = report.rows[0]!;
+        expect(row.outcome).toBe('pending');
+        expect(row.notices.map((n) => n.path)).toEqual(['pages[0].regions[0].components[0].properties.filter']);
+        expect(row.todos.map((t) => t.path)).toEqual(['pages[0].regions[0].components[1].properties.filter']);
+        expect(row.todos[0]!.reason).toContain('the combinator `$or`');
+
+        const text = formatStoredMigrationReport(report).join('\n');
+        const rowAt = text.indexOf('page/deal_desk');
+        const noticeAt = text.indexOf('page-component-filter-record-to-rule-array: {"stage":"open"} →');
+        const todoAt = text.indexOf('TODO page-component-filter-record-to-rule-array:');
+        // Both nested under the row, the conversion first.
+        expect(rowAt).toBeGreaterThanOrEqual(0);
+        expect(noticeAt).toBeGreaterThan(rowAt);
+        expect(todoAt).toBeGreaterThan(noticeAt);
+    });
+
+    it('MEASURED — the write path judges the two door kinds differently, and the TODO rides on either outcome', async () => {
+        // `properties.filter` sits in the page component's open `properties` bag:
+        // the runtime save door does not parse it by `type` (the props gate is
+        // `@objectstack/lint`'s, advisory). `dataSource.filter` is a declared key
+        // of the strict component schema, so the save door refuses it there.
+        const bindingMixed = pageRow('deal_room', [
+            grid({ stage: 'open' }),
+            { type: 'object-kanban', dataSource: { object: 'deal', filter: COMBINATOR }, properties: { objectName: 'deal' } },
+        ]);
+        // The third door: `defaultFilters` on the grid, the same open bag.
+        const defaultsMixed = pageRow('deal_grid', [
+            { type: 'object-grid', properties: { objectName: 'deal', filter: { stage: 'open' }, defaultFilters: COMBINATOR } },
+        ]);
+        const { engine, tables } = makeStubEngine([mixedPage, bindingMixed, defaultsMixed]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+
+        const report = await protocol.migrateStoredMetadata({ apply: true });
+
+        const props = report.rows.find((r) => r.name === 'deal_desk')!;
+        expect(props.outcome).toBe('rewritten');
+        expect(props.todos.map((t) => t.path)).toEqual(['pages[0].regions[0].components[1].properties.filter']);
+        const binding = report.rows.find((r) => r.name === 'deal_room')!;
+        expect(binding.outcome).toBe('failed');
+        expect(binding.todos.map((t) => t.path)).toEqual(['pages[0].regions[0].components[1].dataSource.filter']);
+        const defaults = report.rows.find((r) => r.name === 'deal_grid')!;
+        expect(defaults.outcome).toBe('rewritten');
+        expect(defaults.todos.map((t) => t.path)).toEqual(['pages[0].regions[0].components[0].properties.defaultFilters']);
+        const storedDefaults = JSON.parse(metaRows(tables).find((r) => r.name === 'deal_grid')!.metadata);
+        expect(storedDefaults.regions[0].components[0].properties.defaultFilters).toEqual(COMBINATOR);
+
+        // The rewritten row persisted its lossless half; the combinator is byte-identical.
+        const stored = JSON.parse(metaRows(tables).find((r) => r.name === 'deal_desk')!.metadata);
+        expect(stored.regions[0].components[0].properties.filter).toEqual([
+            { field: 'stage', operator: 'equals', value: 'open' },
+        ]);
+        expect(stored.regions[0].components[1].properties.filter).toEqual(COMBINATOR);
+
+        // Re-run: what is left of the rewritten row is its TODO — skipped, never canonical.
+        const again = await protocol.migrateStoredMetadata({ apply: true, types: ['page'] });
+        const rerun = again.rows.find((r) => r.name === 'deal_desk')!;
+        expect(rerun.outcome).toBe('skipped');
+        expect(rerun.todos).toHaveLength(1);
+        expect(again.canonical).toBe(0);
+    });
+});