diff --git a/codev/projects/1476-tower-extract-runglobalmigrati/status.yaml b/codev/projects/1476-tower-extract-runglobalmigrati/status.yaml new file mode 100644 index 0000000000..b20af978a1 --- /dev/null +++ b/codev/projects/1476-tower-extract-runglobalmigrati/status.yaml @@ -0,0 +1,22 @@ +id: '1476' +title: tower-extract-runglobalmigrati +protocol: air +phase: verified +plan_phases: [] +current_plan_phase: null +gates: + pr: + status: approved + requested_at: '2026-08-17T23:19:15.065Z' + approved_at: '2026-08-17T23:24:10.746Z' +iteration: 1 +build_complete: false +history: [] +started_at: '2026-08-17T23:01:56.616Z' +updated_at: '2026-08-17T23:24:17.107Z' +pr_history: + - phase: pr + pr_number: 1485 + branch: builder/air-1476 + created_at: '2026-08-17T23:19:03.172Z' +pr_ready_for_human: false diff --git a/codev/state/air-1476_thread.md b/codev/state/air-1476_thread.md new file mode 100644 index 0000000000..b377e71fe2 --- /dev/null +++ b/codev/state/air-1476_thread.md @@ -0,0 +1,102 @@ +# air-1476 — Tower: extract runGlobalMigrations(db) + +Protocol: AIR (strict). Issue #1476. + +## What the issue asks + +The global.db migration tests drove hand-maintained *replicas* of the migration blocks that +live inside the private `ensureGlobalDatabase()` path in `db/index.ts`, kept honest by source +guards. Extract the block into `runGlobalMigrations(db)` so production init and tests call the +same runner. + +## Implementation + +- New `packages/codev/src/agent-farm/db/migrations.ts`: `GLOBAL_CURRENT_VERSION` (17) and + `runGlobalMigrations(db, options?)`. The v2→v17 sequence moved over verbatim; only two + seams were added, both defaulted to today's production behavior: + - `options.log` (default `console.log`) — tests collect the per-migration lines instead of + spamming stdout, and can assert which steps ran. + - `options.runDir` (default `~/.codev/run`) — migration v8 renames `shepherd-*.sock` files + on disk. Without this seam, a test driving the real chain would rename a developer's live + sockets. This is the one genuine filesystem side effect in the chain. +- `db/index.ts` now calls `runGlobalMigrations(db)` on the existing-database path and imports + `GLOBAL_CURRENT_VERSION` for the fresh-install marker stamping. Both are re-exported from + `db/index.ts` so no callsite has to learn a new module. + +Note: the runner is only safe on a database that reached its recorded version through +migrations. A fresh GLOBAL_SCHEMA database with no markers would fail at v5 (which selects +`terminal_sessions.project_path`) — which is exactly why `ensureGlobalDatabase` stamps every +marker on the fresh path instead of running the chain. Behavior unchanged; now documented. + +## Tests + +`spec-1313-migration.test.ts` rewritten to drive the real runner (no replicas). Same v15 / v16 +/ v17 coverage as before, plus what only a callable runner makes possible: + +- full v1 → v17 chain on a legacy database, stamping every marker; +- convergence of the migrated database with a fresh `GLOBAL_SCHEMA` install across **all** + tables, columns and indexes (previously only the one table under test); +- v9's `project_path → workspace_path` data carry-over plus v13's architect `role_id` backfill, + asserted on real rows through the table rebuilds no replica reproduced; +- v8's socket rename against an injected run directory, and a missing run directory. + +Because the runner applies every outstanding step, a pre-v15 fixture now walks v15→v16→v17 in +one call — the same thing a real upgrading install does — so the v15 mailbox assertions include +`not_before`. + +Source guards in `send-architect-identity.test.ts` and `bugfix-506-annotator-worktree-cwd.test.ts` +retargeted from `db/index.ts` to `db/migrations.ts` (same intent, new home). + +Out of scope, left alone: `pir-832`, `bugfix-826`, `spec-755` migration tests. Those replicate +migrations of the **retired per-workspace state.db**, which has no production runner to call. + +## CMAP (PR review) + +All three lanes APPROVE, HIGH confidence, zero blocking issues. + +- **gemini** — "Clean extraction of runGlobalMigrations with safe test seams and comprehensive + full-chain test coverage." +- **codex** — "Clean, behavior-preserving extraction with strong real-runner migration coverage." +- **claude** — verified the extraction is a *mechanical* move (diffed it: only the three declared + edits, zero SQL changed) and reproduced the full suite independently. Raised three + non-blocking minors, all now fixed in commit 5a70fbb2: + 1. the `runGlobalMigrations` JSDoc claimed unqualified safety — the marker-less-GLOBAL_SCHEMA + caveat now sits on the function, since it is publicly re-exported; + 2. `pir-832` / `spec-755` / `bugfix-826` still pointed at "db/index.ts's vN block", which now + holds no migrations and whose vN is a *different* migration from the state.db vN they test; + 3. the convergence test compared tables only, so `builders_updated_at` was uncovered — it now + compares triggers too. + + It also noted something I had not: the full-chain marker assertion is a **bidirectional drift + guard on `GLOBAL_CURRENT_VERSION`** — adding a v18 without bumping the constant fails, and so + does bumping it without the migration. Spec 1313 actually shipped that mistake once. + +## Architect CMAP (integration review, risk tier High) + +Unanimous APPROVE across all three lanes, zero blocking. Four non-blocking findings; response +posted as a PR comment. + +- **1 — precondition guard: accepted, implemented** (`235b490c`). The runner now rejects a + marker-less GLOBAL_SCHEMA database at entry (workspace_path-shaped but no v9 marker) with a + named error, instead of dying at v5 on `no such column: project_path`. Unreachable in + production; it exists because the extraction is precisely what made the runner callable from + anywhere. Two tests cover it. +- **3 — pragma fidelity: accepted, implemented** (same commit). The harness now sets production's + full pragma set rather than WAL alone — it matters for v7–v9's DROP + RENAME rebuilds. +- **4 — stale replica comments: already done** in `5a70fbb2`, pushed before the review landed. + The arch.md line is architect/MAINTAIN scope. +- **2 — redundant `GLOBAL_CURRENT_VERSION` source guard: left in place, rebutted.** The new + full-chain marker assertion does cover it better, but deleting an assertion from a Spec 1313 + test changes *that* test's intent rather than this issue's. Offered to drop it either way. + +## Status + +- Implement phase: complete. Build green; full suite green (4861 passed, 48 skipped, 0 failures). +- PR phase: **PR #1485 open**, CMAP fixes pushed, `porch check` green (pr_exists, e2e_tests), + PR recorded via `porch done --pr 1485 --branch builder/air-1476`. +- **pr gate approved by the human** (relayed by the architect); `porch approve 1476 pr` run by me, + then `porch done 1476` → **PROTOCOL COMPLETE**. +- **PR #1485 is deliberately left OPEN and unmerged.** We are not cluesmith/codev maintainers; the + PR is parked for a maintainer to merge. That is why the post-merge steps a builder would normally + run — `porch done --merged`, closing the issue, worktree cleanup — are not done here. Whoever + picks this up after the merge should run them. diff --git a/packages/codev/src/agent-farm/__tests__/bugfix-506-annotator-worktree-cwd.test.ts b/packages/codev/src/agent-farm/__tests__/bugfix-506-annotator-worktree-cwd.test.ts index 4310e772e5..def92bf877 100644 --- a/packages/codev/src/agent-farm/__tests__/bugfix-506-annotator-worktree-cwd.test.ts +++ b/packages/codev/src/agent-farm/__tests__/bugfix-506-annotator-worktree-cwd.test.ts @@ -35,8 +35,10 @@ describe('Bugfix #506: cwd column in terminal_sessions', () => { }); it('migration v12 should add cwd column', () => { + // Issue #1476: the migration chain lives in db/migrations.ts (runGlobalMigrations); + // spec-1313-migration.test.ts drives it for real, this guard pins the v12 statement. const dbSrc = readFileSync( - resolve(import.meta.dirname, '../db/index.ts'), + resolve(import.meta.dirname, '../db/migrations.ts'), 'utf-8', ); expect(dbSrc).toContain('Migration v12'); diff --git a/packages/codev/src/agent-farm/__tests__/bugfix-826-migration.test.ts b/packages/codev/src/agent-farm/__tests__/bugfix-826-migration.test.ts index 87907f988b..a7ed4fa3c9 100644 --- a/packages/codev/src/agent-farm/__tests__/bugfix-826-migration.test.ts +++ b/packages/codev/src/agent-farm/__tests__/bugfix-826-migration.test.ts @@ -84,9 +84,10 @@ describe('Bugfix #826 — Migration v11 (workspace-scoped architect schema)', () } /** - * Run the v11 migration in isolation against the test DBs. Mirrors the - * production code in `db/index.ts` (the v11 block). Kept verbatim so the - * test fails loudly if production drifts from the expected shape. + * Run the v11 migration in isolation against the test DBs. Mirrors the retired + * per-workspace state.db's v11 block — a different numbering space from global.db's + * v11 (`terminal_sessions.label`, in `db/migrations.ts`), with no production runner + * left to call. Kept verbatim so the test fails loudly if the shape drifts. */ function runV11Migration() { const cols = localDb.prepare('PRAGMA table_info(architect)').all() as Array<{ name: string }>; diff --git a/packages/codev/src/agent-farm/__tests__/pir-832-migration.test.ts b/packages/codev/src/agent-farm/__tests__/pir-832-migration.test.ts index 8d9dddf091..216cb110a9 100644 --- a/packages/codev/src/agent-farm/__tests__/pir-832-migration.test.ts +++ b/packages/codev/src/agent-farm/__tests__/pir-832-migration.test.ts @@ -4,7 +4,7 @@ * Migration v12 adds the per-architect conversation `session_id` column so Tower * can resume each architect's prior agent conversation after a restart. These tests * instantiate the prior (post-v11) architect schema by hand, then drive a faithful - * replica of `db/index.ts`'s v12 block and assert the resulting shape — matching the + * replica of the retired state.db's v12 block and assert the resulting shape — matching the * inline-replication convention of `spec-755-migration.test.ts` / `bugfix-826-migration.test.ts`. * Migrations are forward-only by project convention; there is no reverse SQL to test. */ @@ -56,7 +56,12 @@ describe('PIR #832 — architect session_id migration (v12)', () => { } } - /** Faithful replica of the v12 block in db/index.ts (idempotent ALTER + marker). */ + /** + * Faithful replica of the retired per-workspace state.db's v12 block (idempotent + * ALTER + marker). Issue #1476: this is NOT global.db's v12 (`terminal_sessions.cwd`, + * in `db/migrations.ts`) — state.db numbers its migrations in its own space and has + * no production runner left to call, so the replica stays. + */ function runV12Migration(): void { const v12 = db.prepare('SELECT version FROM _migrations WHERE version = 12').get(); if (!v12) { diff --git a/packages/codev/src/agent-farm/__tests__/send-architect-identity.test.ts b/packages/codev/src/agent-farm/__tests__/send-architect-identity.test.ts index b4af14bc8e..56919cec96 100644 --- a/packages/codev/src/agent-farm/__tests__/send-architect-identity.test.ts +++ b/packages/codev/src/agent-farm/__tests__/send-architect-identity.test.ts @@ -229,7 +229,9 @@ describe('Spec 1313 — migration + self-heal source guards', () => { const read = (rel: string) => fs.readFileSync(path.resolve(import.meta.dirname, rel), 'utf-8'); it('db migration v16 is registered, bumps the version, and adds the command column', () => { - const dbSrc = read('../db/index.ts'); + // Issue #1476: the migration chain moved out of db/index.ts into db/migrations.ts + // (`runGlobalMigrations`), which the migration tests now drive directly. + const dbSrc = read('../db/migrations.ts'); // The version constant MUST advance — else a fresh install records only 1..15 // and the v16 block only converges on a later open (the omission #23 flagged). // It now sits at 17 (Spec 1313 round 3 added the not_before mailbox migration); diff --git a/packages/codev/src/agent-farm/__tests__/spec-1313-migration.test.ts b/packages/codev/src/agent-farm/__tests__/spec-1313-migration.test.ts index 3a47d95f8c..9d9a563b40 100644 --- a/packages/codev/src/agent-farm/__tests__/spec-1313-migration.test.ts +++ b/packages/codev/src/agent-farm/__tests__/spec-1313-migration.test.ts @@ -1,127 +1,161 @@ /** - * Spec 1313 — mailbox table migration (v15). + * global.db migrations — driven through the REAL production runner. * - * Migration v15 adds the additive `mailbox` table (mailbox-first delivery). These - * tests instantiate a pre-v15 database by hand, drive a faithful replica of the - * v15 block in `db/index.ts`, and assert the resulting shape — matching the - * inline-replication convention of `pir-832-migration.test.ts` / - * `bugfix-826-migration.test.ts`. Migrations are forward-only by project - * convention; there is no reverse SQL to test. + * Issue #1476: these tests used to drive hand-maintained *replicas* of the + * migration blocks in `db/index.ts`, kept honest by source guards. The blocks now + * live in `db/migrations.ts` as `runGlobalMigrations(db)`, which is what + * `ensureGlobalDatabase()` calls — so every case below exercises the SQL that + * actually ships. Each fixture builds a database at a historical version, calls + * the runner, and asserts the resulting shape. * - * The critical invariant: a freshly-created database (GLOBAL_SCHEMA) and an - * upgraded pre-v15 database must converge on the identical `mailbox` shape. The - * fresh path here exercises the REAL production GLOBAL_SCHEMA, so drift between - * the two definitions fails this test. + * Because the runner applies *every* outstanding step, a pre-v15 fixture walks + * v15 → v16 → v17 in one call — exactly as a real upgrading install does. The + * critical invariant remains: an upgraded database and a freshly-created one + * (GLOBAL_SCHEMA) must converge on an identical shape. Migrations are + * forward-only by project convention; there is no reverse SQL to test. */ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import Database from 'better-sqlite3'; -import { existsSync, mkdirSync, rmSync } from 'node:fs'; +import { existsSync, mkdirSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; import { resolve } from 'node:path'; import { GLOBAL_SCHEMA } from '../db/schema.js'; +import { GLOBAL_CURRENT_VERSION, runGlobalMigrations } from '../db/migrations.js'; -describe('Spec 1313 — mailbox table migration (v15)', () => { - const testDir = resolve(process.cwd(), '.test-spec-1313-migration'); - let db: Database.Database; - let dbPath: string; +/** + * Test harness: a scratch directory holding the database under migration, plus a + * `runDir` standing in for `~/.codev/run` so migration v8's socket rename can + * never touch a developer's live sockets. + */ +function harness(name: string) { + const testDir = resolve(process.cwd(), name); + const state = { + testDir, + runDir: resolve(testDir, 'run'), + db: null as unknown as Database.Database, + logs: [] as string[], + }; beforeEach(() => { if (existsSync(testDir)) rmSync(testDir, { recursive: true }); - mkdirSync(testDir, { recursive: true }); - dbPath = resolve(testDir, 'global.db'); - db = new Database(dbPath); - db.pragma('journal_mode = WAL'); + mkdirSync(state.runDir, { recursive: true }); + state.db = new Database(resolve(testDir, 'global.db')); + // Match production's configurePragmas(): v7–v9 rebuild tables with DROP + RENAME, + // which is exactly the SQL whose behavior depends on `foreign_keys`. + state.db.pragma('journal_mode = WAL'); + state.db.pragma('synchronous = FULL'); + state.db.pragma('busy_timeout = 5000'); + state.db.pragma('foreign_keys = ON'); + state.logs = []; }); afterEach(() => { - db.close(); + state.db.close(); if (existsSync(testDir)) rmSync(testDir, { recursive: true }); }); - /** - * Faithful replica of the v15 block's DDL in `db/index.ts`. Kept verbatim so - * this test fails loudly if the production migration drifts. - */ - const MAILBOX_DDL = ` - CREATE TABLE IF NOT EXISTS mailbox ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - to_agent TEXT NOT NULL, - terminal_id TEXT, - from_agent TEXT, - from_workspace TEXT, - body TEXT NOT NULL, - formatted_message TEXT NOT NULL, - no_enter INTEGER NOT NULL DEFAULT 0, - status TEXT NOT NULL DEFAULT 'held' - CHECK(status IN ('held', 'delivered', 'superseded', 'dismissed')), - reason TEXT CHECK(reason IN ('busy', 'no-profile', 'no-live-pty')), - supersede_key TEXT, - escalated INTEGER NOT NULL DEFAULT 0, - created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL, - resolved_at INTEGER - ); - CREATE INDEX IF NOT EXISTS idx_mailbox_workspace_status ON mailbox(workspace_path, status); - CREATE INDEX IF NOT EXISTS idx_mailbox_agent_drain ON mailbox(workspace_path, to_agent, status); - CREATE INDEX IF NOT EXISTS idx_mailbox_supersede ON mailbox(supersede_key); - `; - - /** - * Reproduce a pre-v15 database: a _migrations table with v1..v14 applied and no - * mailbox table. v15 only creates a new table (it references no other), so no - * other tables are needed to drive it. - */ - function buildPreV15Db(): void { - db.exec(` - CREATE TABLE _migrations ( - version INTEGER PRIMARY KEY, - applied_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - `); - for (let v = 1; v <= 14; v++) { - db.prepare('INSERT INTO _migrations (version) VALUES (?)').run(v); - } - } + /** Drive the real production runner against the fixture database. */ + const migrate = () => + runGlobalMigrations(state.db, { + log: (m) => state.logs.push(m), + runDir: state.runDir, + }); - /** Faithful replica of the v15 block in db/index.ts (idempotent create + marker). */ - function runV15Migration(): void { - const v15 = db.prepare('SELECT version FROM _migrations WHERE version = 15').get(); - if (!v15) { - db.exec(MAILBOX_DDL); - db.prepare('INSERT INTO _migrations (version) VALUES (15)').run(); - } - } - - function tableExists(name: string): boolean { - return !!db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = ?").get(name); - } + const markers = () => + (state.db.prepare('SELECT version FROM _migrations ORDER BY version').all() as Array<{ + version: number; + }>).map((r) => r.version); - function mailboxColumns(): string[] { - return (db.prepare("SELECT name FROM pragma_table_info('mailbox')").all() as Array<{ name: string }>) + const columns = (table: string) => + (state.db.prepare(`SELECT name FROM pragma_table_info(?)`).all(table) as Array<{ name: string }>) .map((c) => c.name) .sort(); - } - function mailboxIndexes(): string[] { - return ( - db - .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='mailbox'") - .all() as Array<{ name: string }> + const indexes = (table: string) => + ( + state.db + .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name = ?") + .all(table) as Array<{ name: string }> ) .map((i) => i.name) - .filter((n) => !n.startsWith('sqlite_')) // drop the implicit PK index + .filter((n) => !n.startsWith('sqlite_')) // drop implicit PK/UNIQUE indexes .sort(); + + const tableExists = (name: string) => + !!state.db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = ?").get(name); + + const seedMarkers = (through: number) => { + state.db.exec(`CREATE TABLE IF NOT EXISTS _migrations ( + version INTEGER PRIMARY KEY, + applied_at TEXT NOT NULL DEFAULT (datetime('now')) + );`); + for (let v = 1; v <= through; v++) { + state.db.prepare('INSERT OR IGNORE INTO _migrations (version) VALUES (?)').run(v); + } + }; + + return { state, migrate, markers, columns, indexes, tableExists, seedMarkers }; +} + +/** The post-v14 terminal_sessions shape: label + cwd, no `command` (that is v16). */ +const PRE_V16_TERMINAL_SESSIONS_DDL = ` + CREATE TABLE IF NOT EXISTS terminal_sessions ( + id TEXT PRIMARY KEY, + workspace_path TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), + role_id TEXT, + pid INTEGER, + shellper_socket TEXT, + shellper_pid INTEGER, + shellper_start_time INTEGER, + label TEXT, + cwd TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); +`; + +/** The post-v15 mailbox shape: no `not_before` (that is v17). */ +const PRE_V17_MAILBOX_DDL = ` + CREATE TABLE IF NOT EXISTS mailbox ( + id TEXT PRIMARY KEY, + workspace_path TEXT NOT NULL, + to_agent TEXT NOT NULL, + terminal_id TEXT, + from_agent TEXT, + from_workspace TEXT, + body TEXT NOT NULL, + formatted_message TEXT NOT NULL, + no_enter INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL DEFAULT 'held' + CHECK(status IN ('held', 'delivered', 'superseded', 'dismissed')), + reason TEXT CHECK(reason IN ('busy', 'no-profile', 'no-live-pty')), + supersede_key TEXT, + escalated INTEGER NOT NULL DEFAULT 0, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + resolved_at INTEGER + ); +`; + +describe('Spec 1313 — mailbox table migration (v15) via runGlobalMigrations', () => { + const h = harness('.test-spec-1313-migration'); + + /** A pre-v15 database: markers through v14 and the post-v14 terminal_sessions. */ + function buildPreV15Db(): void { + h.seedMarkers(14); + h.state.db.exec(PRE_V16_TERMINAL_SESSIONS_DDL); } it('creates the mailbox table on a pre-v15 database', () => { buildPreV15Db(); - expect(tableExists('mailbox')).toBe(false); + expect(h.tableExists('mailbox')).toBe(false); - runV15Migration(); + h.migrate(); - expect(tableExists('mailbox')).toBe(true); - expect(mailboxColumns()).toEqual( + expect(h.tableExists('mailbox')).toBe(true); + // v15 creates the table; the same call then walks v16/v17, so `not_before` + // (v17) is present too — precisely the shape a real upgrade lands on. + expect(h.columns('mailbox')).toEqual( [ 'body', 'created_at', @@ -131,6 +165,7 @@ describe('Spec 1313 — mailbox table migration (v15)', () => { 'from_workspace', 'id', 'no_enter', + 'not_before', 'reason', 'resolved_at', 'status', @@ -145,8 +180,8 @@ describe('Spec 1313 — mailbox table migration (v15)', () => { it('creates the drain and supersede indexes', () => { buildPreV15Db(); - runV15Migration(); - expect(mailboxIndexes()).toEqual([ + h.migrate(); + expect(h.indexes('mailbox')).toEqual([ 'idx_mailbox_agent_drain', 'idx_mailbox_supersede', 'idx_mailbox_workspace_status', @@ -155,26 +190,31 @@ describe('Spec 1313 — mailbox table migration (v15)', () => { it('records v15 in _migrations and is idempotent on re-run', () => { buildPreV15Db(); - runV15Migration(); - expect(() => runV15Migration()).not.toThrow(); + h.migrate(); + const afterFirstRun = h.state.logs.length; + expect(() => h.migrate()).not.toThrow(); - const markers = db.prepare('SELECT COUNT(*) AS n FROM _migrations WHERE version = 15').get() as { + const count = h.state.db.prepare('SELECT COUNT(*) AS n FROM _migrations WHERE version = 15').get() as { n: number; }; - expect(markers.n).toBe(1); - expect(tableExists('mailbox')).toBe(true); + expect(count.n).toBe(1); + expect(h.tableExists('mailbox')).toBe(true); + // The second run applies nothing — every marker is already stamped, so it logs nothing. + expect(h.state.logs.length).toBe(afterFirstRun); }); it('a held row round-trips through the migrated table with its defaults', () => { buildPreV15Db(); - runV15Migration(); + h.migrate(); - db.prepare( - `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, created_at, updated_at) - VALUES ('m1', '/ws/a', 'spir-1313', 'raw', 'formatted', 1000, 1000)` - ).run(); + h.state.db + .prepare( + `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, created_at, updated_at) + VALUES ('m1', '/ws/a', 'spir-1313', 'raw', 'formatted', 1000, 1000)` + ) + .run(); - const row = db.prepare("SELECT * FROM mailbox WHERE id = 'm1'").get() as { + const row = h.state.db.prepare("SELECT * FROM mailbox WHERE id = 'm1'").get() as { status: string; reason: string | null; no_enter: number; @@ -190,9 +230,9 @@ describe('Spec 1313 — mailbox table migration (v15)', () => { it('the status CHECK constraint rejects an unknown status', () => { buildPreV15Db(); - runV15Migration(); + h.migrate(); expect(() => - db + h.state.db .prepare( `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, status, created_at, updated_at) VALUES ('bad', '/ws/a', 'x', 'b', 'f', 'bogus', 1, 1)` @@ -202,20 +242,13 @@ describe('Spec 1313 — mailbox table migration (v15)', () => { }); it('a fresh install (GLOBAL_SCHEMA) converges on the identical mailbox shape as the migration', () => { - // Migrated shape = the FULL mailbox migration chain a pre-v15 database really walks: - // v15 CREATEs the table, then v17 (Spec 1313 round 3) ADDs `not_before`. The live - // GLOBAL_SCHEMA already carries `not_before` in its base CREATE, so the chain must apply - // v17 too or this convergence assertion (correctly) fails — which is exactly what caught - // the round-3 base-schema/migration drift. buildPreV15Db(); - runV15Migration(); - db.exec(`ALTER TABLE mailbox ADD COLUMN not_before INTEGER`); // v17 add-column (see the v17 block below) - const migratedCols = mailboxColumns(); - const migratedIdx = mailboxIndexes(); + h.migrate(); + const migratedCols = h.columns('mailbox'); + const migratedIdx = h.indexes('mailbox'); // Fresh shape: a brand-new database created from the REAL production GLOBAL_SCHEMA. - const freshPath = resolve(testDir, 'fresh.db'); - const fresh = new Database(freshPath); + const fresh = new Database(resolve(h.state.testDir, 'fresh.db')); try { fresh.exec(GLOBAL_SCHEMA); const freshCols = ( @@ -240,107 +273,73 @@ describe('Spec 1313 — mailbox table migration (v15)', () => { }); }); -describe('Spec 1313 — command column migration (v16)', () => { - const testDir = resolve(process.cwd(), '.test-spec-1313-v16-migration'); - let db: Database.Database; - - beforeEach(() => { - if (existsSync(testDir)) rmSync(testDir, { recursive: true }); - mkdirSync(testDir, { recursive: true }); - db = new Database(resolve(testDir, 'global.db')); - db.pragma('journal_mode = WAL'); - }); - afterEach(() => { - db.close(); - if (existsSync(testDir)) rmSync(testDir, { recursive: true }); - }); - - /** The pre-v16 terminal_sessions shape (v15 schema: label + cwd, NO command). */ - const PRE_V16_TERMINAL_SESSIONS_DDL = ` - CREATE TABLE IF NOT EXISTS terminal_sessions ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), - role_id TEXT, - pid INTEGER, - shellper_socket TEXT, - shellper_pid INTEGER, - shellper_start_time INTEGER, - label TEXT, - cwd TEXT, - created_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - `; +describe('Spec 1313 — command column migration (v16) via runGlobalMigrations', () => { + const h = harness('.test-spec-1313-v16-migration'); function buildPreV16Db(): void { - db.exec(`CREATE TABLE _migrations (version INTEGER PRIMARY KEY, applied_at TEXT NOT NULL DEFAULT (datetime('now')));`); - for (let v = 1; v <= 15; v++) db.prepare('INSERT INTO _migrations (version) VALUES (?)').run(v); - db.exec(PRE_V16_TERMINAL_SESSIONS_DDL); + h.seedMarkers(15); + h.state.db.exec(PRE_V16_TERMINAL_SESSIONS_DDL); + h.state.db.exec(PRE_V17_MAILBOX_DDL); // v15 already applied on this fixture } - /** - * Faithful replica of the v16 block in db/index.ts: PRAGMA-gated (only ALTER - * when the column is genuinely absent, so a real failure surfaces instead of - * being marked migrated), then the version marker. - */ - function runV16Migration(): void { - const v16 = db.prepare('SELECT version FROM _migrations WHERE version = 16').get(); - if (!v16) { - const hasCommand = (db.prepare(`PRAGMA table_info(terminal_sessions)`).all() as Array<{ name: string }>) - .some((c) => c.name === 'command'); - if (!hasCommand) db.exec(`ALTER TABLE terminal_sessions ADD COLUMN command TEXT`); - db.prepare('INSERT INTO _migrations (version) VALUES (16)').run(); - } - } - - const termCols = () => - (db.prepare("SELECT name FROM pragma_table_info('terminal_sessions')").all() as Array<{ name: string }>) - .map((c) => c.name).sort(); - it('adds the command column to a pre-v16 terminal_sessions and records v16', () => { buildPreV16Db(); - expect(termCols()).not.toContain('command'); + expect(h.columns('terminal_sessions')).not.toContain('command'); - runV16Migration(); + h.migrate(); - expect(termCols()).toContain('command'); - expect(db.prepare('SELECT version FROM _migrations WHERE version = 16').get()).toBeTruthy(); + expect(h.columns('terminal_sessions')).toContain('command'); + expect(h.state.db.prepare('SELECT version FROM _migrations WHERE version = 16').get()).toBeTruthy(); // The healed column round-trips a value (what reconcile persists for identity). - db.prepare(`INSERT INTO terminal_sessions (id, workspace_path, type, command) VALUES ('t', '/ws', 'architect', 'claude')`).run(); - expect((db.prepare("SELECT command FROM terminal_sessions WHERE id='t'").get() as { command: string }).command).toBe('claude'); + h.state.db + .prepare( + `INSERT INTO terminal_sessions (id, workspace_path, type, command) VALUES ('t', '/ws', 'architect', 'claude')` + ) + .run(); + expect( + (h.state.db.prepare("SELECT command FROM terminal_sessions WHERE id='t'").get() as { command: string }) + .command + ).toBe('claude'); }); it('is idempotent: re-running does not throw, double-add, or duplicate the marker', () => { buildPreV16Db(); - runV16Migration(); - expect(() => runV16Migration()).not.toThrow(); - const markers = db.prepare('SELECT COUNT(*) AS n FROM _migrations WHERE version = 16').get() as { n: number }; - expect(markers.n).toBe(1); - expect(termCols().filter((c) => c === 'command')).toHaveLength(1); + h.migrate(); + expect(() => h.migrate()).not.toThrow(); + const count = h.state.db.prepare('SELECT COUNT(*) AS n FROM _migrations WHERE version = 16').get() as { + n: number; + }; + expect(count.n).toBe(1); + expect(h.columns('terminal_sessions').filter((c) => c === 'command')).toHaveLength(1); }); it('the PRAGMA gate skips the ALTER when the column already exists (fresh-install shape)', () => { // Simulate a fresh install: GLOBAL_SCHEMA already created `command`, but the // v16 marker was not yet stamped. The gate must NOT attempt a duplicate ALTER. - db.exec(`CREATE TABLE _migrations (version INTEGER PRIMARY KEY, applied_at TEXT NOT NULL DEFAULT (datetime('now')));`); - for (let v = 1; v <= 15; v++) db.prepare('INSERT INTO _migrations (version) VALUES (?)').run(v); - db.exec(PRE_V16_TERMINAL_SESSIONS_DDL.replace('cwd TEXT,', 'cwd TEXT,\n command TEXT,')); - expect(termCols()).toContain('command'); + h.seedMarkers(15); + h.state.db.exec(PRE_V16_TERMINAL_SESSIONS_DDL.replace('cwd TEXT,', 'cwd TEXT,\n command TEXT,')); + h.state.db.exec(PRE_V17_MAILBOX_DDL); + expect(h.columns('terminal_sessions')).toContain('command'); - expect(() => runV16Migration()).not.toThrow(); - expect(db.prepare('SELECT version FROM _migrations WHERE version = 16').get()).toBeTruthy(); + expect(() => h.migrate()).not.toThrow(); + expect(h.state.db.prepare('SELECT version FROM _migrations WHERE version = 16').get()).toBeTruthy(); }); it('a fresh install (GLOBAL_SCHEMA) has the command column, matching the migrated shape', () => { buildPreV16Db(); - runV16Migration(); - const migratedCols = termCols(); + h.migrate(); + const migratedCols = h.columns('terminal_sessions'); - const fresh = new Database(resolve(testDir, 'fresh.db')); + const fresh = new Database(resolve(h.state.testDir, 'fresh.db')); try { fresh.exec(GLOBAL_SCHEMA); - const freshCols = (fresh.prepare("SELECT name FROM pragma_table_info('terminal_sessions')").all() as Array<{ name: string }>) - .map((c) => c.name).sort(); + const freshCols = ( + fresh.prepare("SELECT name FROM pragma_table_info('terminal_sessions')").all() as Array<{ + name: string; + }> + ) + .map((c) => c.name) + .sort(); expect(freshCols).toContain('command'); expect(freshCols).toEqual(migratedCols); } finally { @@ -349,122 +348,86 @@ describe('Spec 1313 — command column migration (v16)', () => { }); }); -describe('Spec 1313 round 3 — mailbox not_before column migration (v17)', () => { - const testDir = resolve(process.cwd(), '.test-spec-1313-v17-migration'); - let db: Database.Database; - - beforeEach(() => { - if (existsSync(testDir)) rmSync(testDir, { recursive: true }); - mkdirSync(testDir, { recursive: true }); - db = new Database(resolve(testDir, 'global.db')); - db.pragma('journal_mode = WAL'); - }); - afterEach(() => { - db.close(); - if (existsSync(testDir)) rmSync(testDir, { recursive: true }); - }); - - /** The pre-v17 mailbox shape (v15 schema: no `not_before`). */ - const PRE_V17_MAILBOX_DDL = ` - CREATE TABLE IF NOT EXISTS mailbox ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - to_agent TEXT NOT NULL, - terminal_id TEXT, - from_agent TEXT, - from_workspace TEXT, - body TEXT NOT NULL, - formatted_message TEXT NOT NULL, - no_enter INTEGER NOT NULL DEFAULT 0, - status TEXT NOT NULL DEFAULT 'held' - CHECK(status IN ('held', 'delivered', 'superseded', 'dismissed')), - reason TEXT CHECK(reason IN ('busy', 'no-profile', 'no-live-pty')), - supersede_key TEXT, - escalated INTEGER NOT NULL DEFAULT 0, - created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL, - resolved_at INTEGER - ); - `; +describe('Spec 1313 round 3 — mailbox not_before column migration (v17) via runGlobalMigrations', () => { + const h = harness('.test-spec-1313-v17-migration'); function buildPreV17Db(): void { - db.exec(`CREATE TABLE _migrations (version INTEGER PRIMARY KEY, applied_at TEXT NOT NULL DEFAULT (datetime('now')));`); - for (let v = 1; v <= 16; v++) db.prepare('INSERT INTO _migrations (version) VALUES (?)').run(v); - db.exec(PRE_V17_MAILBOX_DDL); - } - - /** - * Faithful replica of the v17 block in db/index.ts: PRAGMA-gated (only ALTER when the - * column is genuinely absent, so a real failure surfaces instead of being marked migrated), - * then the version marker. Mirrors v16's pattern — a blanket try/catch would let a real - * ALTER failure be recorded as "migrated" and every subsequent mailbox insert would fail. - */ - function runV17Migration(): void { - const v17 = db.prepare('SELECT version FROM _migrations WHERE version = 17').get(); - if (!v17) { - const hasNotBefore = (db.prepare(`PRAGMA table_info(mailbox)`).all() as Array<{ name: string }>) - .some((c) => c.name === 'not_before'); - if (!hasNotBefore) db.exec(`ALTER TABLE mailbox ADD COLUMN not_before INTEGER`); - db.prepare('INSERT INTO _migrations (version) VALUES (17)').run(); - } + h.seedMarkers(16); + h.state.db.exec(PRE_V17_MAILBOX_DDL); } - const mailboxCols = () => - (db.prepare("SELECT name FROM pragma_table_info('mailbox')").all() as Array<{ name: string }>) - .map((c) => c.name).sort(); - it('adds the not_before column to a pre-v17 mailbox and records v17', () => { buildPreV17Db(); - expect(mailboxCols()).not.toContain('not_before'); + expect(h.columns('mailbox')).not.toContain('not_before'); - runV17Migration(); + h.migrate(); - expect(mailboxCols()).toContain('not_before'); - expect(db.prepare('SELECT version FROM _migrations WHERE version = 17').get()).toBeTruthy(); + expect(h.columns('mailbox')).toContain('not_before'); + expect(h.state.db.prepare('SELECT version FROM _migrations WHERE version = 17').get()).toBeTruthy(); // The healed column round-trips a due time (what a `--delay` row persists) and defaults null. - db.prepare( - `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, not_before, created_at, updated_at) - VALUES ('d', '/ws', 'spir-1313', 'b', 'f', 5000, 1000, 1000)` - ).run(); - expect((db.prepare("SELECT not_before FROM mailbox WHERE id='d'").get() as { not_before: number }).not_before).toBe(5000); - db.prepare( - `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, created_at, updated_at) - VALUES ('n', '/ws', 'spir-1313', 'b', 'f', 1000, 1000)` - ).run(); - expect((db.prepare("SELECT not_before FROM mailbox WHERE id='n'").get() as { not_before: number | null }).not_before).toBeNull(); + h.state.db + .prepare( + `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, not_before, created_at, updated_at) + VALUES ('d', '/ws', 'spir-1313', 'b', 'f', 5000, 1000, 1000)` + ) + .run(); + expect( + (h.state.db.prepare("SELECT not_before FROM mailbox WHERE id='d'").get() as { not_before: number }) + .not_before + ).toBe(5000); + h.state.db + .prepare( + `INSERT INTO mailbox (id, workspace_path, to_agent, body, formatted_message, created_at, updated_at) + VALUES ('n', '/ws', 'spir-1313', 'b', 'f', 1000, 1000)` + ) + .run(); + expect( + (h.state.db.prepare("SELECT not_before FROM mailbox WHERE id='n'").get() as { + not_before: number | null; + }).not_before + ).toBeNull(); }); it('is idempotent: re-running does not throw, double-add, or duplicate the marker', () => { buildPreV17Db(); - runV17Migration(); - expect(() => runV17Migration()).not.toThrow(); - const markers = db.prepare('SELECT COUNT(*) AS n FROM _migrations WHERE version = 17').get() as { n: number }; - expect(markers.n).toBe(1); - expect(mailboxCols().filter((c) => c === 'not_before')).toHaveLength(1); + h.migrate(); + expect(() => h.migrate()).not.toThrow(); + const count = h.state.db.prepare('SELECT COUNT(*) AS n FROM _migrations WHERE version = 17').get() as { + n: number; + }; + expect(count.n).toBe(1); + expect(h.columns('mailbox').filter((c) => c === 'not_before')).toHaveLength(1); }); it('the PRAGMA gate skips the ALTER when not_before already exists (fresh-install shape)', () => { // Fresh install: GLOBAL_SCHEMA already created `not_before`, but the v17 marker was not // yet stamped. The gate must NOT attempt a duplicate ALTER (which SQLite would reject). - db.exec(`CREATE TABLE _migrations (version INTEGER PRIMARY KEY, applied_at TEXT NOT NULL DEFAULT (datetime('now')));`); - for (let v = 1; v <= 16; v++) db.prepare('INSERT INTO _migrations (version) VALUES (?)').run(v); - db.exec(PRE_V17_MAILBOX_DDL.replace('escalated INTEGER NOT NULL DEFAULT 0,', 'escalated INTEGER NOT NULL DEFAULT 0,\n not_before INTEGER,')); - expect(mailboxCols()).toContain('not_before'); + h.seedMarkers(16); + h.state.db.exec( + PRE_V17_MAILBOX_DDL.replace( + 'escalated INTEGER NOT NULL DEFAULT 0,', + 'escalated INTEGER NOT NULL DEFAULT 0,\n not_before INTEGER,' + ) + ); + expect(h.columns('mailbox')).toContain('not_before'); - expect(() => runV17Migration()).not.toThrow(); - expect(db.prepare('SELECT version FROM _migrations WHERE version = 17').get()).toBeTruthy(); + expect(() => h.migrate()).not.toThrow(); + expect(h.state.db.prepare('SELECT version FROM _migrations WHERE version = 17').get()).toBeTruthy(); }); it('a fresh install (GLOBAL_SCHEMA) has not_before, matching the migrated shape', () => { buildPreV17Db(); - runV17Migration(); - const migratedCols = mailboxCols(); + h.migrate(); + const migratedCols = h.columns('mailbox'); - const fresh = new Database(resolve(testDir, 'fresh.db')); + const fresh = new Database(resolve(h.state.testDir, 'fresh.db')); try { fresh.exec(GLOBAL_SCHEMA); - const freshCols = (fresh.prepare("SELECT name FROM pragma_table_info('mailbox')").all() as Array<{ name: string }>) - .map((c) => c.name).sort(); + const freshCols = ( + fresh.prepare("SELECT name FROM pragma_table_info('mailbox')").all() as Array<{ name: string }> + ) + .map((c) => c.name) + .sort(); expect(freshCols).toContain('not_before'); expect(freshCols).toEqual(migratedCols); } finally { @@ -472,3 +435,169 @@ describe('Spec 1313 round 3 — mailbox not_before column migration (v17)', () = } }); }); + +/** + * Whole-chain coverage — only possible now that the runner is callable (Issue #1476). + * A replica test could assert one block at a time; driving the real function walks a + * v1-era database through every step, including the table rebuilds (v7/v8/v9) that no + * replica reproduced. + */ +describe('Issue #1476 — the full v1 → v17 chain through the real runner', () => { + const h = harness('.test-issue-1476-full-chain'); + + /** A v1-era database: the marker table with v1 applied and no content tables. */ + function buildLegacyV1Db(): void { + h.seedMarkers(1); + } + + const tables = (db: Database.Database) => + ( + db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'") + .all() as Array<{ name: string }> + ) + .map((t) => t.name) + .sort(); + + it('walks a v1 database to the current version, stamping every marker', () => { + buildLegacyV1Db(); + + h.migrate(); + + const expected = Array.from({ length: GLOBAL_CURRENT_VERSION }, (_, i) => i + 1); + expect(h.markers()).toEqual(expected); + expect(h.state.logs.length).toBeGreaterThan(0); + }); + + const triggers = (db: Database.Database) => + (db.prepare("SELECT name FROM sqlite_master WHERE type='trigger'").all() as Array<{ name: string }>) + .map((t) => t.name) + .sort(); + + it('converges on the same tables, columns, indexes and triggers as a fresh GLOBAL_SCHEMA install', () => { + buildLegacyV1Db(); + h.migrate(); + + const fresh = new Database(resolve(h.state.testDir, 'fresh.db')); + try { + fresh.exec(GLOBAL_SCHEMA); + + expect(tables(h.state.db)).toEqual(tables(fresh)); + // The builders_updated_at trigger is defined in both v14 and GLOBAL_SCHEMA. + expect(triggers(h.state.db)).toEqual(triggers(fresh)); + + for (const table of tables(fresh)) { + const freshCols = ( + fresh.prepare('SELECT name FROM pragma_table_info(?)').all(table) as Array<{ name: string }> + ) + .map((c) => c.name) + .sort(); + const freshIdx = ( + fresh + .prepare("SELECT name FROM sqlite_master WHERE type='index' AND tbl_name = ?") + .all(table) as Array<{ name: string }> + ) + .map((i) => i.name) + .filter((n) => !n.startsWith('sqlite_')) + .sort(); + + expect({ table, cols: h.columns(table) }).toEqual({ table, cols: freshCols }); + expect({ table, idx: h.indexes(table) }).toEqual({ table, idx: freshIdx }); + } + } finally { + fresh.close(); + } + }); + + it('migration v9 carries legacy project_path rows over to workspace_path', () => { + // A post-v8 database: project_path-era tables carrying real rows. + h.seedMarkers(8); + h.state.db.exec(` + CREATE TABLE terminal_sessions ( + id TEXT PRIMARY KEY, + project_path TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), + role_id TEXT, + pid INTEGER, + shellper_socket TEXT, + shellper_pid INTEGER, + shellper_start_time INTEGER, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + CREATE TABLE file_tabs ( + id TEXT PRIMARY KEY, + project_path TEXT NOT NULL, + file_path TEXT NOT NULL, + created_at INTEGER NOT NULL + ); + CREATE TABLE known_projects ( + project_path TEXT PRIMARY KEY, + name TEXT NOT NULL, + last_launched_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + INSERT INTO terminal_sessions (id, project_path, type, role_id) VALUES ('t1', '/ws/legacy', 'architect', NULL); + INSERT INTO file_tabs (id, project_path, file_path, created_at) VALUES ('f1', '/ws/legacy', 'a.md', 1); + INSERT INTO known_projects (project_path, name) VALUES ('/ws/legacy', 'legacy'); + `); + + h.migrate(); + + expect(h.columns('terminal_sessions')).toContain('workspace_path'); + const session = h.state.db.prepare("SELECT workspace_path, role_id FROM terminal_sessions WHERE id='t1'").get() as { + workspace_path: string; + role_id: string; + }; + expect(session.workspace_path).toBe('/ws/legacy'); + expect(session.role_id).toBe('main'); // v13 backfill of legacy architect rows + expect( + (h.state.db.prepare("SELECT workspace_path FROM file_tabs WHERE id='f1'").get() as { + workspace_path: string; + }).workspace_path + ).toBe('/ws/legacy'); + expect( + (h.state.db.prepare("SELECT name FROM known_workspaces WHERE workspace_path='/ws/legacy'").get() as { + name: string; + }).name + ).toBe('legacy'); + expect(h.tableExists('known_projects')).toBe(false); + }); + + it('migration v8 renames shellper sockets in the injected run directory only', () => { + buildLegacyV1Db(); + writeFileSync(resolve(h.state.runDir, 'shepherd-abc.sock'), ''); + writeFileSync(resolve(h.state.runDir, 'unrelated.txt'), ''); + + h.migrate(); + + expect(readdirSync(h.state.runDir).sort()).toEqual(['shellper-abc.sock', 'unrelated.txt']); + }); + + it('refuses a marker-less fresh GLOBAL_SCHEMA database with a named error', () => { + // The runner is now callable from anywhere, so the one shape it cannot handle must + // fail diagnosably rather than dying at v5 on `no such column: project_path`. + h.state.db.exec(GLOBAL_SCHEMA); + h.state.db.prepare('DELETE FROM _migrations').run(); + + expect(() => h.migrate()).toThrow(/no v9 migration marker/); + // Nothing was applied — the guard fires before any step runs. + expect(h.markers()).toEqual([]); + }); + + it('accepts a fresh GLOBAL_SCHEMA database once its markers are stamped (the production path)', () => { + h.state.db.exec(GLOBAL_SCHEMA); + for (let v = 1; v <= GLOBAL_CURRENT_VERSION; v++) { + h.state.db.prepare('INSERT OR IGNORE INTO _migrations (version) VALUES (?)').run(v); + } + + expect(() => h.migrate()).not.toThrow(); + expect(h.state.logs).toEqual([]); // every step already marked; nothing re-runs + }); + + it('a missing run directory does not fail the chain', () => { + buildLegacyV1Db(); + rmSync(h.state.runDir, { recursive: true }); + + expect(() => h.migrate()).not.toThrow(); + expect(h.markers()).toContain(8); + }); +}); diff --git a/packages/codev/src/agent-farm/__tests__/spec-755-migration.test.ts b/packages/codev/src/agent-farm/__tests__/spec-755-migration.test.ts index 383dddec32..232cf9e94b 100644 --- a/packages/codev/src/agent-farm/__tests__/spec-755-migration.test.ts +++ b/packages/codev/src/agent-farm/__tests__/spec-755-migration.test.ts @@ -9,7 +9,7 @@ * These tests instantiate the prior schema by hand, then drive the project's * actual `_migrations`-versioned migration code paths and assert the resulting * shape. Migration paths are forward-only by project convention (see plan and - * `db/index.ts` v3/v4 precedent) — there is no reverse SQL to test. + * `db/migrations.ts` v3/v4 precedent) — there is no reverse SQL to test. */ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; @@ -68,7 +68,9 @@ describe('Spec 755 — Multi-architect migration', () => { /** * Run the v9 migration block in isolation against the test DB. Mirrors the - * production code in `db/index.ts`. Keeping a copy here lets the test + * retired per-workspace state.db's v9 block — a different numbering space from + * global.db's v9 (`project_path` → `workspace_path`, in `db/migrations.ts`), with + * no production runner left to call. Keeping a copy here lets the test * assert behavior without importing the full `getDb()` setup (which would * pull in workspace config, env detection, etc.). */ diff --git a/packages/codev/src/agent-farm/db/index.ts b/packages/codev/src/agent-farm/db/index.ts index 2173bb1895..aad4843f66 100644 --- a/packages/codev/src/agent-farm/db/index.ts +++ b/packages/codev/src/agent-farm/db/index.ts @@ -6,11 +6,11 @@ */ import Database from 'better-sqlite3'; -import { existsSync, mkdirSync, readdirSync, renameSync } from 'node:fs'; -import { homedir } from 'node:os'; -import { resolve, dirname, join } from 'node:path'; +import { existsSync, mkdirSync } from 'node:fs'; +import { resolve, dirname } from 'node:path'; import { AGENT_FARM_DIR } from '../lib/tower-client.js'; import { GLOBAL_SCHEMA } from './schema.js'; +import { GLOBAL_CURRENT_VERSION, runGlobalMigrations } from './migrations.js'; // Singleton instance. Issue #1118: there is now a single user-global database // (~/.agent-farm/global.db). getDb() and getGlobalDb() both return it; the @@ -141,9 +141,6 @@ function ensureGlobalDatabase(): Database.Database { const db = new Database(dbPath); configurePragmas(db); - // Current migration version — bump when adding new migrations - const GLOBAL_CURRENT_VERSION = 17; - // Detect fresh vs existing database by checking if content tables exist. // On existing databases, GLOBAL_SCHEMA must NOT run because it references column names // (workspace_path) that don't exist until migration v9 renames them from project_path. @@ -166,456 +163,10 @@ function ensureGlobalDatabase(): Database.Database { return db; } - // Existing database: only run migrations (skip GLOBAL_SCHEMA to avoid column name conflicts) - // Ensure _migrations table exists for tracking - db.exec(`CREATE TABLE IF NOT EXISTS _migrations ( - version INTEGER PRIMARY KEY, - applied_at TEXT NOT NULL DEFAULT (datetime('now')) - )`); - - // Migration v2: No-op (previously added columns to port_allocations, now removed by Spec 0098) - const v2 = db.prepare('SELECT version FROM _migrations WHERE version = 2').get(); - if (!v2) { - db.prepare('INSERT INTO _migrations (version) VALUES (2)').run(); - } - - // Migration v3: Add terminal_sessions table (Spec 0090 TICK-001) - const v3 = db.prepare('SELECT version FROM _migrations WHERE version = 3').get(); - if (!v3) { - // Create terminal_sessions table if it doesn't exist - db.exec(` - CREATE TABLE IF NOT EXISTS terminal_sessions ( - id TEXT PRIMARY KEY, - project_path TEXT NOT NULL, - type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), - role_id TEXT, - pid INTEGER, - tmux_session TEXT, - created_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_project ON terminal_sessions(project_path); - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); - `); - db.prepare('INSERT INTO _migrations (version) VALUES (3)').run(); - console.log('[info] Created terminal_sessions table (Spec 0090 TICK-001)'); - } - - // Migration v4: Add file_tabs table (Spec 0099 Phase 4) - const v4 = db.prepare('SELECT version FROM _migrations WHERE version = 4').get(); - if (!v4) { - db.exec(` - CREATE TABLE IF NOT EXISTS file_tabs ( - id TEXT PRIMARY KEY, - project_path TEXT NOT NULL, - file_path TEXT NOT NULL, - created_at INTEGER NOT NULL - ); - CREATE INDEX IF NOT EXISTS idx_file_tabs_project ON file_tabs(project_path); - `); - db.prepare('INSERT INTO _migrations (version) VALUES (4)').run(); - console.log('[info] Created file_tabs table (Spec 0099 Phase 4)'); - } - - // Migration v5: Add known_projects table for persistent project registry - const v5 = db.prepare('SELECT version FROM _migrations WHERE version = 5').get(); - if (!v5) { - db.exec(` - CREATE TABLE IF NOT EXISTS known_projects ( - project_path TEXT PRIMARY KEY, - name TEXT NOT NULL, - last_launched_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - `); - // Seed from existing terminal_sessions so current projects appear immediately - db.exec(` - INSERT OR IGNORE INTO known_projects (project_path, name, last_launched_at) - SELECT DISTINCT project_path, '', datetime('now') FROM terminal_sessions; - `); - db.prepare('INSERT INTO _migrations (version) VALUES (5)').run(); - console.log('[info] Created known_projects table'); - } - - // Migration v6: Add shepherd columns to terminal_sessions (Spec 0104) - const v6 = db.prepare('SELECT version FROM _migrations WHERE version = 6').get(); - if (!v6) { - const cols = ['shepherd_socket TEXT', 'shepherd_pid INTEGER', 'shepherd_start_time INTEGER']; - for (const col of cols) { - try { - db.exec(`ALTER TABLE terminal_sessions ADD COLUMN ${col}`); - } catch { - // Column already exists (fresh install ran updated schema) - } - } - db.prepare('INSERT INTO _migrations (version) VALUES (6)').run(); - console.log('[info] Added shepherd columns to terminal_sessions (Spec 0104)'); - } - - // Migration v7: Drop tmux_session column from terminal_sessions (Spec 0104 Phase 4) - const v7 = db.prepare('SELECT version FROM _migrations WHERE version = 7').get(); - if (!v7) { - // SQLite table-rebuild pattern to drop the tmux_session column - try { - db.exec(` - CREATE TABLE IF NOT EXISTS terminal_sessions_new ( - id TEXT PRIMARY KEY, - project_path TEXT NOT NULL, - type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), - role_id TEXT, - pid INTEGER, - shepherd_socket TEXT, - shepherd_pid INTEGER, - shepherd_start_time INTEGER, - created_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - INSERT OR IGNORE INTO terminal_sessions_new - SELECT id, project_path, type, role_id, pid, shepherd_socket, shepherd_pid, shepherd_start_time, created_at - FROM terminal_sessions; - DROP TABLE terminal_sessions; - ALTER TABLE terminal_sessions_new RENAME TO terminal_sessions; - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_project ON terminal_sessions(project_path); - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); - `); - } catch { - // Table may already be in the correct schema (fresh install) - } - db.prepare('INSERT INTO _migrations (version) VALUES (7)').run(); - console.log('[info] Dropped tmux_session column from terminal_sessions (Spec 0104)'); - } - - // Migration v8: Rename shepherd_* columns to shellper_* (Spec 0106) - const v8 = db.prepare('SELECT version FROM _migrations WHERE version = 8').get(); - if (!v8) { - try { - db.exec(` - CREATE TABLE IF NOT EXISTS terminal_sessions_new ( - id TEXT PRIMARY KEY, - project_path TEXT NOT NULL, - type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), - role_id TEXT, - pid INTEGER, - shellper_socket TEXT, - shellper_pid INTEGER, - shellper_start_time INTEGER, - created_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - INSERT OR IGNORE INTO terminal_sessions_new - SELECT id, project_path, type, role_id, pid, shepherd_socket, shepherd_pid, shepherd_start_time, created_at - FROM terminal_sessions; - DROP TABLE terminal_sessions; - ALTER TABLE terminal_sessions_new RENAME TO terminal_sessions; - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_project ON terminal_sessions(project_path); - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); - UPDATE terminal_sessions SET shellper_socket = REPLACE(shellper_socket, 'shepherd-', 'shellper-') - WHERE shellper_socket LIKE '%shepherd-%'; - `); - } catch { - // Table may already be in the correct schema (fresh install) - } - // Rename physical socket files on disk - try { - const runDir = join(homedir(), '.codev', 'run'); - if (existsSync(runDir)) { - const files = readdirSync(runDir); - for (const file of files) { - if (file.startsWith('shepherd-') && file.endsWith('.sock')) { - const newName = file.replace('shepherd-', 'shellper-'); - try { - renameSync(join(runDir, file), join(runDir, newName)); - } catch { - // Skip files that can't be renamed (missing, permissions, etc.) - } - } - } - } - } catch { - // Skip if run directory doesn't exist or can't be read - } - db.prepare('INSERT INTO _migrations (version) VALUES (8)').run(); - console.log('[info] Renamed shepherd columns to shellper in terminal_sessions (Spec 0106)'); - } - - // Migration v9: Rename project_path → workspace_path in all tables (Spec 0112) - // Note: Fresh installs never reach here (handled above), so old column names are guaranteed. - // Wrapped in a transaction for atomicity — all three renames succeed or none do. - const v9 = db.prepare('SELECT version FROM _migrations WHERE version = 9').get(); - if (!v9) { - const migrate = db.transaction(() => { - // 1. Rename terminal_sessions.project_path → workspace_path - db.exec(` - CREATE TABLE IF NOT EXISTS terminal_sessions_new ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), - role_id TEXT, - pid INTEGER, - shellper_socket TEXT, - shellper_pid INTEGER, - shellper_start_time INTEGER, - created_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - INSERT OR IGNORE INTO terminal_sessions_new - SELECT id, project_path, type, role_id, pid, shellper_socket, shellper_pid, shellper_start_time, created_at - FROM terminal_sessions; - DROP TABLE terminal_sessions; - ALTER TABLE terminal_sessions_new RENAME TO terminal_sessions; - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_workspace ON terminal_sessions(workspace_path); - CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); - `); - - // 2. Rename file_tabs.project_path → workspace_path - db.exec(` - CREATE TABLE IF NOT EXISTS file_tabs_new ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - file_path TEXT NOT NULL, - created_at INTEGER NOT NULL - ); - INSERT OR IGNORE INTO file_tabs_new - SELECT id, project_path, file_path, created_at - FROM file_tabs; - DROP TABLE file_tabs; - ALTER TABLE file_tabs_new RENAME TO file_tabs; - CREATE INDEX IF NOT EXISTS idx_file_tabs_workspace ON file_tabs(workspace_path); - `); - - // 3. Rename known_projects → known_workspaces with project_path → workspace_path - db.exec(` - CREATE TABLE IF NOT EXISTS known_workspaces ( - workspace_path TEXT PRIMARY KEY, - name TEXT NOT NULL, - last_launched_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - INSERT OR IGNORE INTO known_workspaces (workspace_path, name, last_launched_at) - SELECT project_path, name, last_launched_at FROM known_projects; - DROP TABLE IF EXISTS known_projects; - `); - - db.prepare('INSERT INTO _migrations (version) VALUES (9)').run(); - }); - migrate(); - console.log('[info] Renamed project_path → workspace_path in global tables (Spec 0112)'); - } - - // Migration v10: Add cron_tasks table (Spec 399) - const v10 = db.prepare('SELECT version FROM _migrations WHERE version = 10').get(); - if (!v10) { - db.exec(` - CREATE TABLE IF NOT EXISTS cron_tasks ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - task_name TEXT NOT NULL, - last_run INTEGER, - last_result TEXT, - last_output TEXT, - enabled INTEGER NOT NULL DEFAULT 1, - UNIQUE(workspace_path, task_name) - ); - `); - db.prepare('INSERT INTO _migrations (version) VALUES (10)').run(); - console.log('[info] Created cron_tasks table (Spec 399)'); - } - - // Migration v11: Add label column to terminal_sessions (Spec 468) - const v11 = db.prepare('SELECT version FROM _migrations WHERE version = 11').get(); - if (!v11) { - try { - db.exec(`ALTER TABLE terminal_sessions ADD COLUMN label TEXT`); - } catch { - // Column may already exist from a fresh install - } - db.prepare('INSERT INTO _migrations (version) VALUES (11)').run(); - console.log('[info] Added label column to terminal_sessions (Spec 468)'); - } - - // Migration v12: Add cwd column to terminal_sessions (Bugfix #506) - const v12 = db.prepare('SELECT version FROM _migrations WHERE version = 12').get(); - if (!v12) { - try { - db.exec(`ALTER TABLE terminal_sessions ADD COLUMN cwd TEXT`); - } catch { - // Column may already exist from a fresh install - } - db.prepare('INSERT INTO _migrations (version) VALUES (12)').run(); - console.log('[info] Added cwd column to terminal_sessions (Bugfix #506)'); - } - - // Migration v13: Backfill terminal_sessions.role_id for legacy architect rows (Spec 755) - // Pre-v13 rows for architects always stored role_id as NULL because there was only - // ever one architect per workspace. Multi-architect support requires the name to be - // present in role_id so reconnect can re-key the in-memory map. The idempotent - // backfill sets role_id = 'main' for legacy rows; subsequent architect rows write - // their explicit name and are unaffected. - const v13 = db.prepare('SELECT version FROM _migrations WHERE version = 13').get(); - if (!v13) { - db.prepare(` - UPDATE terminal_sessions - SET role_id = 'main' - WHERE type = 'architect' AND role_id IS NULL - `).run(); - db.prepare('INSERT INTO _migrations (version) VALUES (13)').run(); - console.log('[info] Backfilled architect role_id with \'main\' (Spec 755)'); - } - - // Migration v14: Absorb the retired state.db tables (Issue #1118). - // Creates architect/builders/utils/annotations in global.db at their final - // shape. architect/utils/annotations move as-is; builders is RESHAPED with a - // workspace_path column + composite PK (workspace_path, id) so the same - // builder id can exist in multiple workspaces. Idempotent via - // `CREATE TABLE IF NOT EXISTS`. The one-time data migration of legacy - // state.db files is a separate, marker-gated step run at Tower boot - // (db/consolidate.ts) — NOT here — so opening global.db never moves data. - const v14 = db.prepare('SELECT version FROM _migrations WHERE version = 14').get(); - if (!v14) { - db.exec(` - CREATE TABLE IF NOT EXISTS architect ( - workspace_path TEXT NOT NULL, - id TEXT NOT NULL, - pid INTEGER NOT NULL, - port INTEGER NOT NULL, - cmd TEXT NOT NULL, - started_at TEXT NOT NULL DEFAULT (datetime('now')), - terminal_id TEXT, - session_id TEXT, - PRIMARY KEY (workspace_path, id) - ); - CREATE INDEX IF NOT EXISTS idx_architect_workspace ON architect(workspace_path); - - CREATE TABLE IF NOT EXISTS builders ( - workspace_path TEXT NOT NULL, - id TEXT NOT NULL, - name TEXT NOT NULL, - port INTEGER NOT NULL DEFAULT 0, - pid INTEGER NOT NULL DEFAULT 0, - status TEXT NOT NULL DEFAULT 'spawning' - CHECK(status IN ('spawning', 'implementing', 'blocked', 'pr', 'complete')), - phase TEXT NOT NULL DEFAULT '', - worktree TEXT NOT NULL, - branch TEXT NOT NULL, - type TEXT NOT NULL DEFAULT 'spec' - CHECK(type IN ('spec', 'task', 'protocol', 'shell', 'worktree', 'bugfix', 'pir')), - task_text TEXT, - protocol_name TEXT, - issue_number TEXT, - terminal_id TEXT, - spawned_by_architect TEXT, - started_at TEXT NOT NULL DEFAULT (datetime('now')), - updated_at TEXT NOT NULL DEFAULT (datetime('now')), - PRIMARY KEY (workspace_path, id) - ); - CREATE INDEX IF NOT EXISTS idx_builders_status ON builders(status); - CREATE INDEX IF NOT EXISTS idx_builders_port ON builders(port); - CREATE TRIGGER IF NOT EXISTS builders_updated_at - AFTER UPDATE ON builders - FOR EACH ROW - BEGIN - UPDATE builders SET updated_at = datetime('now') - WHERE workspace_path = NEW.workspace_path AND id = NEW.id; - END; - - CREATE TABLE IF NOT EXISTS utils ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL, - port INTEGER NOT NULL DEFAULT 0, - pid INTEGER NOT NULL DEFAULT 0, - terminal_id TEXT, - started_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - - CREATE TABLE IF NOT EXISTS annotations ( - id TEXT PRIMARY KEY, - file TEXT NOT NULL, - port INTEGER NOT NULL DEFAULT 0, - pid INTEGER NOT NULL DEFAULT 0, - parent_type TEXT NOT NULL CHECK(parent_type IN ('architect', 'builder', 'util')), - parent_id TEXT, - started_at TEXT NOT NULL DEFAULT (datetime('now')) - ); - `); - db.prepare('INSERT INTO _migrations (version) VALUES (14)').run(); - console.log('[info] Absorbed state.db tables into global.db (Issue #1118)'); - } - - // Migration v15: Add mailbox table (Spec 1313 — mailbox-first delivery). - // Additive new table: every `afx send` is persisted here before the send - // response returns, so nothing is lost to a Tower crash/restart/shutdown. - // Rows address AGENTS (to_agent), not PTYs, so a respawned terminal drains its - // predecessor's mail. No rows to migrate — the retired SendBuffer was in-memory. - // Idempotent via CREATE TABLE / CREATE INDEX IF NOT EXISTS (fresh installs - // already created it from GLOBAL_SCHEMA and reach the marker as a no-op). - const v15 = db.prepare('SELECT version FROM _migrations WHERE version = 15').get(); - if (!v15) { - db.exec(` - CREATE TABLE IF NOT EXISTS mailbox ( - id TEXT PRIMARY KEY, - workspace_path TEXT NOT NULL, - to_agent TEXT NOT NULL, - terminal_id TEXT, - from_agent TEXT, - from_workspace TEXT, - body TEXT NOT NULL, - formatted_message TEXT NOT NULL, - no_enter INTEGER NOT NULL DEFAULT 0, - status TEXT NOT NULL DEFAULT 'held' - CHECK(status IN ('held', 'delivered', 'superseded', 'dismissed')), - reason TEXT CHECK(reason IN ('busy', 'no-profile', 'no-live-pty')), - supersede_key TEXT, - escalated INTEGER NOT NULL DEFAULT 0, - created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL, - resolved_at INTEGER - ); - CREATE INDEX IF NOT EXISTS idx_mailbox_workspace_status ON mailbox(workspace_path, status); - CREATE INDEX IF NOT EXISTS idx_mailbox_agent_drain ON mailbox(workspace_path, to_agent, status); - CREATE INDEX IF NOT EXISTS idx_mailbox_supersede ON mailbox(supersede_key); - `); - db.prepare('INSERT INTO _migrations (version) VALUES (15)').run(); - console.log('[info] Created mailbox table (Spec 1313)'); - } - - // Migration v16: Add command column to terminal_sessions (Spec 1313). - // The render-gate resolves an agent's classifier profile from its launch - // command (PtySession.command). Shellper-backed sessions were created with - // command: '' and the profile fell back to reading `.builder-start.sh` — - // which only builder worktrees have. Architects run in the workspace root - // (no launch script), so they never resolved and every `afx send architect` - // held `no-profile`. Persisting the command lets the reconcile/reconnect - // paths restore identity after a Tower restart, so architects resolve - // directly and survive restart (builders keep the launch-script backstop). - // Mirrors the label (v11) / cwd (v12) column adds. - const v16 = db.prepare('SELECT version FROM _migrations WHERE version = 16').get(); - if (!v16) { - // Only skip the ALTER when the column genuinely exists already (fresh install - // ran GLOBAL_SCHEMA). A blanket try/catch would let a REAL alter failure be - // recorded as "migrated" — and since saveTerminalSession's INSERT now names - // `command`, every future write would then fail against a table missing it. - const hasCommand = (db.prepare(`PRAGMA table_info(terminal_sessions)`).all() as Array<{ name: string }>) - .some((c) => c.name === 'command'); - if (!hasCommand) { - db.exec(`ALTER TABLE terminal_sessions ADD COLUMN command TEXT`); - } - db.prepare('INSERT INTO _migrations (version) VALUES (16)').run(); - console.log('[info] Added command column to terminal_sessions (Spec 1313 restart-safe render-gate identity)'); - } - - // Migration v17: Add not_before column to mailbox (Spec 1313 round 3 — durable `--delay`). - // `afx send --delay` now persists its row at REQUEST time with not_before = now + delay*1000 - // and defers delivery through the render gate, so a delayed send survives a Tower restart - // (the conscious reversal of Spec 1307's drop-on-restart semantics). A row is deliverable - // only when `not_before IS NULL OR not_before <= now`; null means deliver-ASAP (every - // pre-round-3 row). PRAGMA-gated ADD COLUMN mirroring v16 — a blanket try/catch would let a - // real ALTER failure be recorded as "migrated" and every subsequent mailbox insert (which - // now names not_before) would then fail against a table missing it. Do NOT edit v15 in place: - // dev machines on this branch already applied it, so the column must arrive as its own step. - const v17 = db.prepare('SELECT version FROM _migrations WHERE version = 17').get(); - if (!v17) { - const hasNotBefore = (db.prepare(`PRAGMA table_info(mailbox)`).all() as Array<{ name: string }>) - .some((c) => c.name === 'not_before'); - if (!hasNotBefore) { - db.exec(`ALTER TABLE mailbox ADD COLUMN not_before INTEGER`); - } - db.prepare('INSERT INTO _migrations (version) VALUES (17)').run(); - console.log('[info] Added not_before column to mailbox (Spec 1313 durable --delay)'); - } + // Existing database: only run migrations (skip GLOBAL_SCHEMA to avoid column + // name conflicts). The migration sequence itself lives in db/migrations.ts so + // production init and the migration tests drive the same runner (Issue #1476). + runGlobalMigrations(db); return db; } @@ -623,6 +174,8 @@ function ensureGlobalDatabase(): Database.Database { // Re-export types and utilities export { LOCAL_SCHEMA, GLOBAL_SCHEMA } from './schema.js'; export { withRetry } from './errors.js'; +export { GLOBAL_CURRENT_VERSION, runGlobalMigrations } from './migrations.js'; +export type { GlobalMigrationOptions } from './migrations.js'; export type { DbArchitect, DbBuilder, diff --git a/packages/codev/src/agent-farm/db/migrations.ts b/packages/codev/src/agent-farm/db/migrations.ts new file mode 100644 index 0000000000..e7e311b6e9 --- /dev/null +++ b/packages/codev/src/agent-farm/db/migrations.ts @@ -0,0 +1,542 @@ +/** + * global.db forward-only migrations (Issue #1476). + * + * The migration sequence used to live inline inside the private + * `ensureGlobalDatabase()` path in `db/index.ts`, so migration tests could only + * drive hand-maintained *replicas* of it. This module is the single source of + * truth both production init and the tests call: + * + * - `ensureGlobalDatabase()` calls `runGlobalMigrations(db)` on an existing + * database (fresh installs get the final shape from GLOBAL_SCHEMA and stamp + * every marker up to `GLOBAL_CURRENT_VERSION` instead). + * - `__tests__/spec-1313-migration.test.ts` builds a database at a historical + * version and calls the very same function, so the SQL under test is the SQL + * that ships. + * + * The runner takes an explicit `db` handle (matching `db/mailbox.ts` / + * `db/consolidate.ts`) — it never opens a connection, reads a singleton, or sets + * pragmas. Every step is idempotent and gated on its `_migrations` marker, so + * calling it repeatedly on the same handle is a no-op. Migrations are + * forward-only by project convention; there is no reverse SQL. + */ + +import type Database from 'better-sqlite3'; +import { existsSync, readdirSync, renameSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; + +/** Current migration version — bump when adding new migrations. */ +export const GLOBAL_CURRENT_VERSION = 17; + +export interface GlobalMigrationOptions { + /** + * Sink for per-migration progress lines. Defaults to `console.log`, which is + * what the production open path wants; tests pass a collector (or a no-op) to + * keep output clean and to assert which steps actually ran. + */ + log?: (message: string) => void; + /** + * Directory holding shellper sockets, whose files migration v8 renames. + * Defaults to `~/.codev/run` — the production location. Tests point it at a + * temp directory so driving the real runner never touches a developer's live + * sockets. + */ + runDir?: string; +} + +/** + * Run every outstanding global.db migration against `db`. + * + * Safe to call on any existing database that reached its recorded version through + * migrations: each step checks its own `_migrations` marker first, so applied steps + * are skipped and the call converges on the `GLOBAL_CURRENT_VERSION` shape. + * + * NOT for a fresh database. A GLOBAL_SCHEMA-shaped database with no markers would run + * v5 against the long-renamed `terminal_sessions.project_path` — which is why + * `ensureGlobalDatabase()` stamps every marker on the fresh path instead of running + * the chain. That shape is rejected at entry with a named error rather than an opaque + * mid-chain SQLite failure. + * + * @throws if the database carries the post-v9 `workspace_path` shape without the v9 + * marker to match (a fresh schema that never walked the chain). + */ +export function runGlobalMigrations( + db: Database.Database, + options: GlobalMigrationOptions = {} +): void { + const log = options.log ?? ((message: string) => console.log(message)); + const runDir = options.runDir ?? join(homedir(), '.codev', 'run'); + + // Ensure _migrations table exists for tracking + db.exec(`CREATE TABLE IF NOT EXISTS _migrations ( + version INTEGER PRIMARY KEY, + applied_at TEXT NOT NULL DEFAULT (datetime('now')) + )`); + + // Precondition. Steps v5/v7/v8/v9 read and rebuild the project_path-era tables, so + // the chain only makes sense on a database that reached its recorded version through + // migrations. A fresh GLOBAL_SCHEMA database with no markers would instead die at v5 + // with an opaque `no such column: project_path` — so name that shape here. Production + // never reaches this (ensureGlobalDatabase stamps every marker on the fresh path); it + // exists because the runner is now callable from anywhere. + const terminalSessions = db + .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name='terminal_sessions'") + .get(); + if (terminalSessions) { + const renamed = (db.prepare(`PRAGMA table_info(terminal_sessions)`).all() as Array<{ name: string }>) + .some((c) => c.name === 'workspace_path'); + const v9Marker = db.prepare('SELECT version FROM _migrations WHERE version = 9').get(); + if (renamed && !v9Marker) { + throw new Error( + 'runGlobalMigrations: refusing to run on a database whose terminal_sessions is already ' + + 'workspace_path-shaped but has no v9 migration marker. This is a fresh GLOBAL_SCHEMA ' + + 'database that never walked the chain — stamp markers 1..GLOBAL_CURRENT_VERSION instead ' + + '(see ensureGlobalDatabase), or the chain will fail mid-way on renamed columns.' + ); + } + } + + // Migration v2: No-op (previously added columns to port_allocations, now removed by Spec 0098) + const v2 = db.prepare('SELECT version FROM _migrations WHERE version = 2').get(); + if (!v2) { + db.prepare('INSERT INTO _migrations (version) VALUES (2)').run(); + } + + // Migration v3: Add terminal_sessions table (Spec 0090 TICK-001) + const v3 = db.prepare('SELECT version FROM _migrations WHERE version = 3').get(); + if (!v3) { + // Create terminal_sessions table if it doesn't exist + db.exec(` + CREATE TABLE IF NOT EXISTS terminal_sessions ( + id TEXT PRIMARY KEY, + project_path TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), + role_id TEXT, + pid INTEGER, + tmux_session TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_project ON terminal_sessions(project_path); + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); + `); + db.prepare('INSERT INTO _migrations (version) VALUES (3)').run(); + log('[info] Created terminal_sessions table (Spec 0090 TICK-001)'); + } + + // Migration v4: Add file_tabs table (Spec 0099 Phase 4) + const v4 = db.prepare('SELECT version FROM _migrations WHERE version = 4').get(); + if (!v4) { + db.exec(` + CREATE TABLE IF NOT EXISTS file_tabs ( + id TEXT PRIMARY KEY, + project_path TEXT NOT NULL, + file_path TEXT NOT NULL, + created_at INTEGER NOT NULL + ); + CREATE INDEX IF NOT EXISTS idx_file_tabs_project ON file_tabs(project_path); + `); + db.prepare('INSERT INTO _migrations (version) VALUES (4)').run(); + log('[info] Created file_tabs table (Spec 0099 Phase 4)'); + } + + // Migration v5: Add known_projects table for persistent project registry + const v5 = db.prepare('SELECT version FROM _migrations WHERE version = 5').get(); + if (!v5) { + db.exec(` + CREATE TABLE IF NOT EXISTS known_projects ( + project_path TEXT PRIMARY KEY, + name TEXT NOT NULL, + last_launched_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + `); + // Seed from existing terminal_sessions so current projects appear immediately + db.exec(` + INSERT OR IGNORE INTO known_projects (project_path, name, last_launched_at) + SELECT DISTINCT project_path, '', datetime('now') FROM terminal_sessions; + `); + db.prepare('INSERT INTO _migrations (version) VALUES (5)').run(); + log('[info] Created known_projects table'); + } + + // Migration v6: Add shepherd columns to terminal_sessions (Spec 0104) + const v6 = db.prepare('SELECT version FROM _migrations WHERE version = 6').get(); + if (!v6) { + const cols = ['shepherd_socket TEXT', 'shepherd_pid INTEGER', 'shepherd_start_time INTEGER']; + for (const col of cols) { + try { + db.exec(`ALTER TABLE terminal_sessions ADD COLUMN ${col}`); + } catch { + // Column already exists (fresh install ran updated schema) + } + } + db.prepare('INSERT INTO _migrations (version) VALUES (6)').run(); + log('[info] Added shepherd columns to terminal_sessions (Spec 0104)'); + } + + // Migration v7: Drop tmux_session column from terminal_sessions (Spec 0104 Phase 4) + const v7 = db.prepare('SELECT version FROM _migrations WHERE version = 7').get(); + if (!v7) { + // SQLite table-rebuild pattern to drop the tmux_session column + try { + db.exec(` + CREATE TABLE IF NOT EXISTS terminal_sessions_new ( + id TEXT PRIMARY KEY, + project_path TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), + role_id TEXT, + pid INTEGER, + shepherd_socket TEXT, + shepherd_pid INTEGER, + shepherd_start_time INTEGER, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + INSERT OR IGNORE INTO terminal_sessions_new + SELECT id, project_path, type, role_id, pid, shepherd_socket, shepherd_pid, shepherd_start_time, created_at + FROM terminal_sessions; + DROP TABLE terminal_sessions; + ALTER TABLE terminal_sessions_new RENAME TO terminal_sessions; + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_project ON terminal_sessions(project_path); + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); + `); + } catch { + // Table may already be in the correct schema (fresh install) + } + db.prepare('INSERT INTO _migrations (version) VALUES (7)').run(); + log('[info] Dropped tmux_session column from terminal_sessions (Spec 0104)'); + } + + // Migration v8: Rename shepherd_* columns to shellper_* (Spec 0106) + const v8 = db.prepare('SELECT version FROM _migrations WHERE version = 8').get(); + if (!v8) { + try { + db.exec(` + CREATE TABLE IF NOT EXISTS terminal_sessions_new ( + id TEXT PRIMARY KEY, + project_path TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), + role_id TEXT, + pid INTEGER, + shellper_socket TEXT, + shellper_pid INTEGER, + shellper_start_time INTEGER, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + INSERT OR IGNORE INTO terminal_sessions_new + SELECT id, project_path, type, role_id, pid, shepherd_socket, shepherd_pid, shepherd_start_time, created_at + FROM terminal_sessions; + DROP TABLE terminal_sessions; + ALTER TABLE terminal_sessions_new RENAME TO terminal_sessions; + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_project ON terminal_sessions(project_path); + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); + UPDATE terminal_sessions SET shellper_socket = REPLACE(shellper_socket, 'shepherd-', 'shellper-') + WHERE shellper_socket LIKE '%shepherd-%'; + `); + } catch { + // Table may already be in the correct schema (fresh install) + } + // Rename physical socket files on disk + try { + if (existsSync(runDir)) { + const files = readdirSync(runDir); + for (const file of files) { + if (file.startsWith('shepherd-') && file.endsWith('.sock')) { + const newName = file.replace('shepherd-', 'shellper-'); + try { + renameSync(join(runDir, file), join(runDir, newName)); + } catch { + // Skip files that can't be renamed (missing, permissions, etc.) + } + } + } + } + } catch { + // Skip if run directory doesn't exist or can't be read + } + db.prepare('INSERT INTO _migrations (version) VALUES (8)').run(); + log('[info] Renamed shepherd columns to shellper in terminal_sessions (Spec 0106)'); + } + + // Migration v9: Rename project_path → workspace_path in all tables (Spec 0112) + // Note: Fresh installs never reach here (handled by the caller), so old column + // names are guaranteed. + // Wrapped in a transaction for atomicity — all three renames succeed or none do. + const v9 = db.prepare('SELECT version FROM _migrations WHERE version = 9').get(); + if (!v9) { + const migrate = db.transaction(() => { + // 1. Rename terminal_sessions.project_path → workspace_path + db.exec(` + CREATE TABLE IF NOT EXISTS terminal_sessions_new ( + id TEXT PRIMARY KEY, + workspace_path TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('architect', 'builder', 'shell')), + role_id TEXT, + pid INTEGER, + shellper_socket TEXT, + shellper_pid INTEGER, + shellper_start_time INTEGER, + created_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + INSERT OR IGNORE INTO terminal_sessions_new + SELECT id, project_path, type, role_id, pid, shellper_socket, shellper_pid, shellper_start_time, created_at + FROM terminal_sessions; + DROP TABLE terminal_sessions; + ALTER TABLE terminal_sessions_new RENAME TO terminal_sessions; + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_workspace ON terminal_sessions(workspace_path); + CREATE INDEX IF NOT EXISTS idx_terminal_sessions_type ON terminal_sessions(type); + `); + + // 2. Rename file_tabs.project_path → workspace_path + db.exec(` + CREATE TABLE IF NOT EXISTS file_tabs_new ( + id TEXT PRIMARY KEY, + workspace_path TEXT NOT NULL, + file_path TEXT NOT NULL, + created_at INTEGER NOT NULL + ); + INSERT OR IGNORE INTO file_tabs_new + SELECT id, project_path, file_path, created_at + FROM file_tabs; + DROP TABLE file_tabs; + ALTER TABLE file_tabs_new RENAME TO file_tabs; + CREATE INDEX IF NOT EXISTS idx_file_tabs_workspace ON file_tabs(workspace_path); + `); + + // 3. Rename known_projects → known_workspaces with project_path → workspace_path + db.exec(` + CREATE TABLE IF NOT EXISTS known_workspaces ( + workspace_path TEXT PRIMARY KEY, + name TEXT NOT NULL, + last_launched_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + INSERT OR IGNORE INTO known_workspaces (workspace_path, name, last_launched_at) + SELECT project_path, name, last_launched_at FROM known_projects; + DROP TABLE IF EXISTS known_projects; + `); + + db.prepare('INSERT INTO _migrations (version) VALUES (9)').run(); + }); + migrate(); + log('[info] Renamed project_path → workspace_path in global tables (Spec 0112)'); + } + + // Migration v10: Add cron_tasks table (Spec 399) + const v10 = db.prepare('SELECT version FROM _migrations WHERE version = 10').get(); + if (!v10) { + db.exec(` + CREATE TABLE IF NOT EXISTS cron_tasks ( + id TEXT PRIMARY KEY, + workspace_path TEXT NOT NULL, + task_name TEXT NOT NULL, + last_run INTEGER, + last_result TEXT, + last_output TEXT, + enabled INTEGER NOT NULL DEFAULT 1, + UNIQUE(workspace_path, task_name) + ); + `); + db.prepare('INSERT INTO _migrations (version) VALUES (10)').run(); + log('[info] Created cron_tasks table (Spec 399)'); + } + + // Migration v11: Add label column to terminal_sessions (Spec 468) + const v11 = db.prepare('SELECT version FROM _migrations WHERE version = 11').get(); + if (!v11) { + try { + db.exec(`ALTER TABLE terminal_sessions ADD COLUMN label TEXT`); + } catch { + // Column may already exist from a fresh install + } + db.prepare('INSERT INTO _migrations (version) VALUES (11)').run(); + log('[info] Added label column to terminal_sessions (Spec 468)'); + } + + // Migration v12: Add cwd column to terminal_sessions (Bugfix #506) + const v12 = db.prepare('SELECT version FROM _migrations WHERE version = 12').get(); + if (!v12) { + try { + db.exec(`ALTER TABLE terminal_sessions ADD COLUMN cwd TEXT`); + } catch { + // Column may already exist from a fresh install + } + db.prepare('INSERT INTO _migrations (version) VALUES (12)').run(); + log('[info] Added cwd column to terminal_sessions (Bugfix #506)'); + } + + // Migration v13: Backfill terminal_sessions.role_id for legacy architect rows (Spec 755) + // Pre-v13 rows for architects always stored role_id as NULL because there was only + // ever one architect per workspace. Multi-architect support requires the name to be + // present in role_id so reconnect can re-key the in-memory map. The idempotent + // backfill sets role_id = 'main' for legacy rows; subsequent architect rows write + // their explicit name and are unaffected. + const v13 = db.prepare('SELECT version FROM _migrations WHERE version = 13').get(); + if (!v13) { + db.prepare(` + UPDATE terminal_sessions + SET role_id = 'main' + WHERE type = 'architect' AND role_id IS NULL + `).run(); + db.prepare('INSERT INTO _migrations (version) VALUES (13)').run(); + log('[info] Backfilled architect role_id with \'main\' (Spec 755)'); + } + + // Migration v14: Absorb the retired state.db tables (Issue #1118). + // Creates architect/builders/utils/annotations in global.db at their final + // shape. architect/utils/annotations move as-is; builders is RESHAPED with a + // workspace_path column + composite PK (workspace_path, id) so the same + // builder id can exist in multiple workspaces. Idempotent via + // `CREATE TABLE IF NOT EXISTS`. The one-time data migration of legacy + // state.db files is a separate, marker-gated step run at Tower boot + // (db/consolidate.ts) — NOT here — so opening global.db never moves data. + const v14 = db.prepare('SELECT version FROM _migrations WHERE version = 14').get(); + if (!v14) { + db.exec(` + CREATE TABLE IF NOT EXISTS architect ( + workspace_path TEXT NOT NULL, + id TEXT NOT NULL, + pid INTEGER NOT NULL, + port INTEGER NOT NULL, + cmd TEXT NOT NULL, + started_at TEXT NOT NULL DEFAULT (datetime('now')), + terminal_id TEXT, + session_id TEXT, + PRIMARY KEY (workspace_path, id) + ); + CREATE INDEX IF NOT EXISTS idx_architect_workspace ON architect(workspace_path); + + CREATE TABLE IF NOT EXISTS builders ( + workspace_path TEXT NOT NULL, + id TEXT NOT NULL, + name TEXT NOT NULL, + port INTEGER NOT NULL DEFAULT 0, + pid INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL DEFAULT 'spawning' + CHECK(status IN ('spawning', 'implementing', 'blocked', 'pr', 'complete')), + phase TEXT NOT NULL DEFAULT '', + worktree TEXT NOT NULL, + branch TEXT NOT NULL, + type TEXT NOT NULL DEFAULT 'spec' + CHECK(type IN ('spec', 'task', 'protocol', 'shell', 'worktree', 'bugfix', 'pir')), + task_text TEXT, + protocol_name TEXT, + issue_number TEXT, + terminal_id TEXT, + spawned_by_architect TEXT, + started_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY (workspace_path, id) + ); + CREATE INDEX IF NOT EXISTS idx_builders_status ON builders(status); + CREATE INDEX IF NOT EXISTS idx_builders_port ON builders(port); + CREATE TRIGGER IF NOT EXISTS builders_updated_at + AFTER UPDATE ON builders + FOR EACH ROW + BEGIN + UPDATE builders SET updated_at = datetime('now') + WHERE workspace_path = NEW.workspace_path AND id = NEW.id; + END; + + CREATE TABLE IF NOT EXISTS utils ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + port INTEGER NOT NULL DEFAULT 0, + pid INTEGER NOT NULL DEFAULT 0, + terminal_id TEXT, + started_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + + CREATE TABLE IF NOT EXISTS annotations ( + id TEXT PRIMARY KEY, + file TEXT NOT NULL, + port INTEGER NOT NULL DEFAULT 0, + pid INTEGER NOT NULL DEFAULT 0, + parent_type TEXT NOT NULL CHECK(parent_type IN ('architect', 'builder', 'util')), + parent_id TEXT, + started_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + `); + db.prepare('INSERT INTO _migrations (version) VALUES (14)').run(); + log('[info] Absorbed state.db tables into global.db (Issue #1118)'); + } + + // Migration v15: Add mailbox table (Spec 1313 — mailbox-first delivery). + // Additive new table: every `afx send` is persisted here before the send + // response returns, so nothing is lost to a Tower crash/restart/shutdown. + // Rows address AGENTS (to_agent), not PTYs, so a respawned terminal drains its + // predecessor's mail. No rows to migrate — the retired SendBuffer was in-memory. + // Idempotent via CREATE TABLE / CREATE INDEX IF NOT EXISTS (fresh installs + // already created it from GLOBAL_SCHEMA and reach the marker as a no-op). + const v15 = db.prepare('SELECT version FROM _migrations WHERE version = 15').get(); + if (!v15) { + db.exec(` + CREATE TABLE IF NOT EXISTS mailbox ( + id TEXT PRIMARY KEY, + workspace_path TEXT NOT NULL, + to_agent TEXT NOT NULL, + terminal_id TEXT, + from_agent TEXT, + from_workspace TEXT, + body TEXT NOT NULL, + formatted_message TEXT NOT NULL, + no_enter INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL DEFAULT 'held' + CHECK(status IN ('held', 'delivered', 'superseded', 'dismissed')), + reason TEXT CHECK(reason IN ('busy', 'no-profile', 'no-live-pty')), + supersede_key TEXT, + escalated INTEGER NOT NULL DEFAULT 0, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + resolved_at INTEGER + ); + CREATE INDEX IF NOT EXISTS idx_mailbox_workspace_status ON mailbox(workspace_path, status); + CREATE INDEX IF NOT EXISTS idx_mailbox_agent_drain ON mailbox(workspace_path, to_agent, status); + CREATE INDEX IF NOT EXISTS idx_mailbox_supersede ON mailbox(supersede_key); + `); + db.prepare('INSERT INTO _migrations (version) VALUES (15)').run(); + log('[info] Created mailbox table (Spec 1313)'); + } + + // Migration v16: Add command column to terminal_sessions (Spec 1313). + // The render-gate resolves an agent's classifier profile from its launch + // command (PtySession.command). Shellper-backed sessions were created with + // command: '' and the profile fell back to reading `.builder-start.sh` — + // which only builder worktrees have. Architects run in the workspace root + // (no launch script), so they never resolved and every `afx send architect` + // held `no-profile`. Persisting the command lets the reconcile/reconnect + // paths restore identity after a Tower restart, so architects resolve + // directly and survive restart (builders keep the launch-script backstop). + // Mirrors the label (v11) / cwd (v12) column adds. + const v16 = db.prepare('SELECT version FROM _migrations WHERE version = 16').get(); + if (!v16) { + // Only skip the ALTER when the column genuinely exists already (fresh install + // ran GLOBAL_SCHEMA). A blanket try/catch would let a REAL alter failure be + // recorded as "migrated" — and since saveTerminalSession's INSERT now names + // `command`, every future write would then fail against a table missing it. + const hasCommand = (db.prepare(`PRAGMA table_info(terminal_sessions)`).all() as Array<{ name: string }>) + .some((c) => c.name === 'command'); + if (!hasCommand) { + db.exec(`ALTER TABLE terminal_sessions ADD COLUMN command TEXT`); + } + db.prepare('INSERT INTO _migrations (version) VALUES (16)').run(); + log('[info] Added command column to terminal_sessions (Spec 1313 restart-safe render-gate identity)'); + } + + // Migration v17: Add not_before column to mailbox (Spec 1313 round 3 — durable `--delay`). + // `afx send --delay` now persists its row at REQUEST time with not_before = now + delay*1000 + // and defers delivery through the render gate, so a delayed send survives a Tower restart + // (the conscious reversal of Spec 1307's drop-on-restart semantics). A row is deliverable + // only when `not_before IS NULL OR not_before <= now`; null means deliver-ASAP (every + // pre-round-3 row). PRAGMA-gated ADD COLUMN mirroring v16 — a blanket try/catch would let a + // real ALTER failure be recorded as "migrated" and every subsequent mailbox insert (which + // now names not_before) would then fail against a table missing it. Do NOT edit v15 in place: + // dev machines on this branch already applied it, so the column must arrive as its own step. + const v17 = db.prepare('SELECT version FROM _migrations WHERE version = 17').get(); + if (!v17) { + const hasNotBefore = (db.prepare(`PRAGMA table_info(mailbox)`).all() as Array<{ name: string }>) + .some((c) => c.name === 'not_before'); + if (!hasNotBefore) { + db.exec(`ALTER TABLE mailbox ADD COLUMN not_before INTEGER`); + } + db.prepare('INSERT INTO _migrations (version) VALUES (17)').run(); + log('[info] Added not_before column to mailbox (Spec 1313 durable --delay)'); + } +}