feat(driver-sql,objectql,cli)!: the ADR-0104 file-family column step and its kernel→driver supply (#15989)

.changeset/15989-file-family-column-step.md

@@ -0,0 +1,54 @@
+---
+"@objectstack/driver-sql": minor
+"@objectstack/objectql": minor
+"@objectstack/platform-objects": minor
+"@objectstack/spec": minor
+"@objectstack/cli": minor
+---
+
+feat(driver-sql,objectql,cli)!: the ADR-0104 file-family column step, and the kernel→driver supply that arms it (#15989)
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing authorable moves. No `packages/spec` key, no Zod schema, no authored metadata property, no object definition and no accepted request shape changes its spelling, type or legality in this diff: `DataMigrationFlagSchema` and its `columns_moved_at` member landed under #16185 and are READ here, not edited, and the one `packages/spec` edit is a new exported PREDICATE function over that existing type. So `objectstack migrate meta` has nothing to visit, `spec-changes.json` has nothing to project and the upgrade guide has no row to gain — the ledger's whole subject is authored metadata, and what moves here is a physical column's type plus the encoding of the values inside it, on a deployment whose operator ran a command to move them. ADR-0104's row-data side already has its own declared, operator-run surface (`os migrate files-to-references`), which is not a metadata upgrade. The other four categories are closed on facts: every package here publishes to npm, declares no `private` and ships `dist` in `files[]` (not `unpublished`); no ADR-0087 id is minted in this diff (not `registered`) and none pre-dates the base that would cover it (not `already-registered`); and exported declarations DO change — 13 new declarations reaching a package entry (11 on `@objectstack/driver-sql`'s: the 6 values MEDIA_COLUMN_MOVE_DIALECTS, MEDIA_COLUMN_MOVE_ROLLBACK_NOTES, MEDIA_ID_MOVE_WIDTH, isJsonColumnType, mediaColumnMoveDialect, mediaColumnMovePlan and the 5 types MediaColumnMoveDialect, MediaColumnMoveKind, MediaColumnMovePlan, MediaColumnMoveRefusal, MediaColumnMoveScan; plus recordFileColumnMove on `@objectstack/platform-objects/system` and hasMovedFileColumns on `@objectstack/spec`) and 3 new public methods on exported classes (SqlDriver.planMediaColumnMove, SqlDriver.setFileColumnsMovedResolver, ObjectQL.haveFileColumnsMoved) — so neither `runtime-interface-only` nor `type-surface-only` applies. The `**BREAKING**` banner below is carried rather than dropped, because published storage behaviour of `@objectstack/driver-sql` changes. -->
+
+**BREAKING** on the published storage behaviour of `@objectstack/driver-sql`. A deployment that runs `os migrate files-to-references --apply` now has its media columns **retyped and their values rewritten** into the bare-`sys_file`-id encoding, and its driver writes bare ids from the next boot. This completes the maintainer ruling on #15041 (「15041 应该改为实际 id 保存。选A，其他同意」) whose encoding half shipped in the previous release.
+
+Shipped as `minor` under the repo's launch-window convention, in which `major` is refused by `check-changeset-no-major` and breaking-ness is carried by this banner plus the ADR-0087 disposition rather than by the level.
+
+## The column step
+
+`os migrate files-to-references --apply` gains a further step, run **only after** the backfill and its self-check report zero blocking rows — and it moves nothing at all until three gates pass:
+
+1. the migration's own gate (zero blocking rows);
+2. **every** abort pre-check, across **every** planned column, before a single statement runs;
+3. no refusals — a column the driver could not plan stops the columns it could.
+
+**PostgreSQL** and **SQLite** only. ⛔ MySQL is refused by name and belongs to #17788, where its statement ORDER is settled against a real instance rather than transcribed.
+
+Per column, the shape is read off the column's **physical type**, not off the dialect: a `json` column is retyped (`ALTER … TYPE varchar(2048) USING (col #>> '{}')`), while a column that is already `varchar` — the population `os generate migration --format sql` creates and a JSON-arm driver fills with quoted ids — has its values unquoted in place. SQLite has only the second shape, since it has no json type.
+
+### ⛔ The abort clause is NOT the one the ADR sketched
+
+The #15041 addendum prescribed the retype with nothing in front of it while *requiring* the step to abort "on the first cell that is not a JSON string". Those two sentences contradict each other, and which was wrong was settled by running it. Measured on live PostgreSQL 16.13, `USING (col #>> '{}')` is **accepted** over a row holding an inline metadata blob, because `#>> '{}'` extracts *any* json type as text: the bytes survive, but the column is no longer `json`, so an object becomes a plain string in a column whose declared contents are ids — silently, in a migration that reports success. The director ruling (decision batch #120 item 1) replaced the clause with the pre-check that implements the requirement: `json_typeof(col) IS DISTINCT FROM 'string'` on PostgreSQL, and `json_valid(col) AND json_type(col) <> 'text'` on SQLite, where excluding invalid JSON is what keeps a re-run idempotent over cells a previous run already moved.
+
+Both the destructive form and the guarded one are executed side by side, on one fixture, in this release's own test suite — so the difference stays a measurement rather than a comment.
+
+## The kernel→driver supply seam
+
+`SqlDriverConfig.fileColumnsMoved` shipped last release and no host outside the driver supplied it. It is supplied now: `ObjectQL.registerDriver` hands every driver that has the seam a closure over the new `ObjectQL.haveFileColumnsMoved()`, which reads `sys_migration.columns_moved_at` — and requires the `adr-0104-file-references` flag to be verified **as well**, since the stamp alone would attest a column move with nothing attesting the values inside it.
+
+⭐ **Every way of not knowing still answers "not moved".** The option omitted, a resolver that throws or rejects or answers a non-`true` value, a resolver that never runs because the host never calls `initObjects`, a driver with no such seam, no `sys_migration` object, no row, an unreadable table, a null or empty stamp — all the JSON arm. That is the encoding every deployment in the world is on, and a driver that guessed the other way would write bare ids into a JSON column.
+
+⛔ **A host that names `fileColumnsMoved` in its own config wins**, in either polarity. The engine only ever fills an empty slot, and never contradicts an explicit composition: overruling a declared `false` is precisely the bare-ids-into-a-JSON-column failure this mechanism exists to prevent.
+
+## New published surface
+
+- `@objectstack/spec` — `hasMovedFileColumns(flag)`, the single arbiter of the conjunction above, beside `isDataMigrationFlagVerified` and `authorisesIrreversibleAction`.
+- `@objectstack/objectql` — `ObjectQL.haveFileColumnsMoved()`, sharing one memoized read (and one `invalidateDataMigrationFlags()`) with `isFileReferencesMigrationVerified()`, so the two answers can never come out of one another's date.
+- `@objectstack/platform-objects` — `recordFileColumnMove(engine, migrationId)`, which refuses to stamp a deployment with no verified flag row. `readDataMigrationFlag` now carries `columns_moved_at`; it previously dropped it, which made a moved deployment indistinguishable from an unmoved one to every caller.
+- `@objectstack/driver-sql` — `SqlDriver.setFileColumnsMovedResolver()`, `SqlDriver.planMediaColumnMove()`, and the statement builders `mediaColumnMovePlan` / `mediaColumnMoveDialect` / `isJsonColumnType` with `MEDIA_COLUMN_MOVE_DIALECTS`, `MEDIA_COLUMN_MOVE_ROLLBACK_NOTES` and `MEDIA_ID_MOVE_WIDTH`. The statements live in the package that owns the dialects and measured them; a second copy in the CLI would be a second copy of the clause the ruling got wrong.
+
+## What does NOT change
+
+A deployment that does not run `--apply` is byte-for-byte where it was: the column stays `json`, the write still JSON-encodes, and the read still accepts both encodings. A backfill re-run does not set the stamp and — deliberately — cannot clear it either: `recordDataMigrationRun` omits the key rather than writing a preserved value, so a ledger read that FAILS cannot demote a moved deployment back onto the JSON arm. A partial or failed column step records nothing at all, which leaves such a datastore on the arm that reads both encodings.
+
+`multiple: true` media is untouched on both arms: its value is a list of ids and a JSON column on every deployment.

packages/cli/src/commands/migrate/files-to-references.ts

@@ -19,6 +19,42 @@ import { bootSchemaStack } from '../../utils/schema-migrate.js';
 import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js';
 import { describeOccupancy } from '../../utils/sqlite-occupancy.js';
 import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js';
+import {
+  describeFileColumnMoveRefusal,
+  runFileColumnMove,
+  type FileColumnMoveResult,
+} from '../../utils/file-column-move.js';
+import type { IObjectQLEngine } from '@objectstack/spec/contracts';
+import type { SqlDriverLike } from '../../utils/schema-migrate.js';
+import type { MediaColumnMoveScan, SqlDialectName } from '@objectstack/driver-sql';
+
+/**
+ * What {@link MigrateFilesToReferences.runColumnStep} did, or declined to do.
+ *
+ * `skipped` and `failed` are deliberately separate: every skip is a stated,
+ * non-failing reason (this command's subject is the backfill), and only a
+ * column step that ran and could not finish fails the command — because that
+ * is the one outcome that leaves storage an operator has to be told about.
+ */
+interface ColumnStepOutcome {
+  skipped: 'gate_not_passed' | 'no_sql_driver' | 'no_sql_seam' | 'nothing_to_move' | null;
+  failed: boolean;
+  /** `sys_migration.columns_moved_at` as written, or `null` if it was not written. */
+  stampedAt: string | null;
+  /** Set when the columns moved and RECORDING that failed — a durability failure. */
+  stampError?: string;
+  report: {
+    dialect: SqlDialectName;
+    apply: boolean;
+    blocking: number;
+    outcomes: FileColumnMoveResult['outcomes'];
+    refusals: MediaColumnMoveScan['refusals'];
+    executedStatements: string[];
+    recordable: boolean;
+    /** Carried from the driver, because the renderer cannot `await import`. */
+    rollbackNotes: readonly string[];
+  } | null;
+}
 
 async function confirm(question: string): Promise<boolean> {
   if (!process.stdin.isTTY) return false; // non-interactive → require --yes
@@ -221,6 +257,26 @@ export default class MigrateFilesToReferences extends Command {
         includeUnreferenced: flags['include-unreferenced'],
       });
 
+      // ── The COLUMN step (#15989, the ruling on #15041 step 2) ────────────
+      //
+      // Runs only after the backfill and its self-check reported zero blocking
+      // rows — the ruling's own "abort otherwise", and the reason it lives
+      // here rather than in a command of its own: the gate's verdict is what
+      // authorises it, and this is the only place that verdict exists.
+      //
+      // ⛔ The move and the arm flip are ONE act. Measured on SQLite: after
+      // the columns are converted a JSON-arm driver still READS the migrated
+      // column correctly but its next WRITE re-quotes. So `columns_moved_at`
+      // is stamped in the same block that moved the columns, and only when
+      // every one of them moved.
+      const columnMove = await this.runColumnStep({
+        stack,
+        engine,
+        apply,
+        gatePassed: result.gatePassed,
+        json: flags.json,
+      });
+
       if (flags.json) {
         await emitJson({
           database: stack.dbLabel,
@@ -250,9 +306,11 @@ export default class MigrateFilesToReferences extends Command {
           gatePassed: result.gatePassed,
           gateFailures: result.gateFailures,
           flag: result.flag,
+          columnMove: columnMove.report,
+          columnsMovedAt: columnMove.stampedAt,
           duration: timer.elapsed(),
         });
-        if (!result.gatePassed) this.exit(1);
+        if (!result.gatePassed || columnMove.failed) this.exit(1);
         return;
       }
 
@@ -291,9 +349,11 @@ export default class MigrateFilesToReferences extends Command {
             : 'Fix the records listed above, then re-run (and finally with --apply).',
         );
       }
+      this.renderColumnStep(columnMove);
+
       console.log(chalk.dim(`  ${timer.display()}`));
       console.log('');
-      if (!result.gatePassed) this.exit(1);
+      if (!result.gatePassed || columnMove.failed) this.exit(1);
     } catch (error: any) {
       if (isExitSignal(error)) throw error;
       if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); }
@@ -303,4 +363,172 @@ export default class MigrateFilesToReferences extends Command {
       await stack.shutdown();
     }
   }
+
+  /**
+   * The column step — plan, pre-check, move, stamp (#15989).
+   *
+   * Every early return is a NON-failure with a stated reason: this command's
+   * subject is the backfill, and a deployment whose driver cannot plan a
+   * column move is not a deployment whose backfill failed. The one thing that
+   * fails the command is a column step that was asked to run, ran, and could
+   * not finish — because that leaves storage the operator must be told about.
+   */
+  private async runColumnStep(args: {
+    stack: { driver: SqlDriverLike | null; kernel: unknown };
+    engine: unknown;
+    apply: boolean;
+    gatePassed: boolean;
+    json: boolean;
+  }): Promise<ColumnStepOutcome> {
+    const { stack, apply, gatePassed, json } = args;
+
+    if (!gatePassed) {
+      // ⛔ The ruling's "abort unless backfill + verify report zero blocking".
+      // Not an error of this step's own — the gate already reported why.
+      return { skipped: 'gate_not_passed', failed: false, stampedAt: null, report: null };
+    }
+    if (!stack.driver || typeof stack.driver.planMediaColumnMove !== 'function') {
+      return { skipped: 'no_sql_driver', failed: false, stampedAt: null, report: null };
+    }
+
+    const scan = await stack.driver.planMediaColumnMove();
+    if (scan.plans.length === 0 && scan.refusals.length === 0) {
+      return { skipped: 'nothing_to_move', failed: false, stampedAt: null, report: null };
+    }
+
+    // Lazily, at the point of use — ⛔ never a static value import of a driver
+    // package in a command module (#5726).
+    const { MEDIA_COLUMN_MOVE_ROLLBACK_NOTES } = await import('@objectstack/driver-sql');
+    const { resolveSeedTenancyExec, normalizeRows } = await import('@objectstack/metadata-protocol');
+    const exec = resolveSeedTenancyExec(args.engine as IObjectQLEngine | undefined);
+    // Loud absence, never a silent success. A driver can expose an `execute`
+    // that accepts every statement and performs none (#10677) — and "moved 3
+    // columns" from a seam that ran nothing, followed by a `columns_moved_at`
+    // stamp, is the worst report this command could produce: the driver would
+    // then write bare ids into columns that never moved.
+    const answers = exec
+      ? await exec('select 1 as os_seam_probe')
+          .then((r) => normalizeRows(r).length > 0)
+          .catch(() => false)
+      : false;
+    if (!exec || !answers) {
+      return { skipped: 'no_sql_seam', failed: false, stampedAt: null, report: null };
+    }
+
+    const run = await runFileColumnMove({
+      scan,
+      exec,
+      rows: normalizeRows,
+      apply,
+      onStatement: json ? undefined : (statement: string) => printStep(chalk.dim(statement)),
+    });
+
+    let stampedAt: string | null = null;
+    let stampError: string | undefined;
+    if (run.recordable) {
+      try {
+        const { recordFileColumnMove } = await import('@objectstack/platform-objects/system');
+        const { FILE_REFERENCES_MIGRATION_ID } = await import('@objectstack/spec/system');
+        stampedAt = await recordFileColumnMove(args.engine as any, FILE_REFERENCES_MIGRATION_ID);
+      } catch (error: any) {
+        // The columns MOVED and the ledger does not say so. That is a
+        // durability degradation in the sense AGENTS.md names: the next boot
+        // stays on the JSON arm and re-quotes its writes into a column that
+        // has already been converted. It must fail the command.
+        stampError = error?.message ?? String(error);
+      }
+    }
+
+    const failed =
+      run.outcomes.some((o) => o.status === 'failed') || stampError !== undefined;
+
+    return {
+      skipped: null,
+      failed,
+      stampedAt,
+      stampError,
+      report: {
+        dialect: scan.dialect,
+        rollbackNotes: MEDIA_COLUMN_MOVE_ROLLBACK_NOTES,
+        apply: run.apply,
+        blocking: run.blocking,
+        outcomes: run.outcomes,
+        refusals: run.refusals,
+        executedStatements: run.executedStatements,
+        recordable: run.recordable,
+      },
+    };
+  }
+
+  /** The human-mode half of {@link runColumnStep}. JSON mode reports the same facts. */
+  private renderColumnStep(outcome: ColumnStepOutcome): void {
+    if (outcome.skipped === 'gate_not_passed' || outcome.report === null) {
+      if (outcome.skipped === 'no_sql_driver') {
+        printInfo(
+          'Column step: not applicable — the ADR-0104 file-family column move is a SQL-driver step ' +
+            'and no SQL driver is active here.',
+        );
+      } else if (outcome.skipped === 'no_sql_seam') {
+        printWarning(
+          'Column step: SKIPPED — the active driver exposes no usable raw SQL seam, so the media ' +
+            'columns were neither inspected nor moved. The deployment stays on the JSON encoding.',
+        );
+      } else if (outcome.skipped === 'nothing_to_move') {
+        printInfo('Column step: nothing to move — this datastore declares no single-value media column.');
+      }
+      return;
+    }
+
+    const report = outcome.report;
+    console.log('');
+    console.log(chalk.bold(`Column step · ${report.dialect}`));
+    for (const o of report.outcomes) {
+      const mark =
+        o.status === 'moved' ? chalk.green('✓')
+        : o.status === 'blocked' || o.status === 'failed' ? chalk.red('✗')
+        : chalk.yellow('•');
+      console.log(`${mark} ${chalk.bold(`${o.table}.${o.column}`)}  ${chalk.dim(`(${o.kind})`)}`);
+      console.log(`    ${chalk.cyan(o.statement)}`);
+      if (o.error) console.log(`    ${chalk.red(o.error)}`);
+    }
+    for (const refusal of report.refusals) {
+      printWarning(`${refusal.table}.${refusal.column}: ${refusal.detail}`);
+    }
+
+    const refusal = describeFileColumnMoveRefusal({
+      apply: report.apply,
+      outcomes: report.outcomes,
+      refusals: report.refusals,
+      executedStatements: report.executedStatements,
+      blocking: report.blocking,
+      recordable: report.recordable,
+    });
+    console.log('');
+    if (refusal) {
+      printError(refusal);
+    } else if (!report.apply) {
+      printInfo(
+        `Dry run — every abort pre-check passed and nothing was executed. ${report.outcomes.length} ` +
+          'column(s) would move. Take a backup, then re-run with --apply.',
+      );
+    } else if (outcome.stampError) {
+      printError(
+        `The columns MOVED but recording it failed (${outcome.stampError}). This deployment's ` +
+          'driver will stay on the JSON encoding and re-quote its next write into a column that ' +
+          'has already been converted — re-run this command to record it.',
+      );
+    } else if (outcome.stampedAt) {
+      printSuccess(
+        `Column step complete — ${report.outcomes.length} media column(s) moved to the bare-id ` +
+          `encoding and recorded (sys_migration.columns_moved_at = ${outcome.stampedAt}). The SQL ` +
+          'driver writes bare ids from its next boot, and keeps reading the legacy encoding.',
+      );
+    }
+
+    if (refusal || report.outcomes.some((o) => o.status === 'failed')) {
+      console.log('');
+      console.log(chalk.bold('If it goes wrong:'));
+      for (const note of report.rollbackNotes) console.log(`  ${chalk.dim('·')} ${note}`);
+    }
+  }
 }