[srbaker]

Closes #178.

The engine pin moves to 4.0.0.a8. A database GemStone 4.0.0.a5 wrote is now upgraded in place with Grail's scripts/upgrade_engine.sh --in-place (GemTalk/Grail#1867), so users no longer have to delete it.

How it behaves

ensureRunning checks the extent with copydbf -i after the shared-memory step and before the processes start. It checks once per window and extent, not on every cell.

• Upgradable (a pair in ENGINE_UPGRADES, currently a5 → a8), when someone asked for the database (Start, a notebook cell, the Shell, Run or Debug File):
  • A modal asks first. It names the backup (db/data/extent0.dbf.pre-4.0.0.a5) and says that a failure restores it.
  • GemDB stops the stone and listener with the previous engine, under the stone lock. After an extension update the old stone is usually still running, and a8's stopstone cannot log in to an a5 stone (error 4044, measured). If a stone is still listed, GemDB does not run the script.
  • It runs the bundled script with GEMSTONE set to the new engine and GemDB's GEMSTONE_GLOBAL_DIR, under a temporary stone named gemdbupgrade. The latest upgrade_engine: line shows as progress, and the full output goes to the GemDB log.
  • On success, it checks that copydbf reads a8. It then removes the Grail and MCP stamps, so the start continues and reinstalls both without asking: the upgrade removed their compiled methods. A notification says where the backup is.
  • On failure, the script has already restored the extent. GemDB re-reads its version and says whether the database is back as it was, or else points at the backup. A modal error with Show Log follows, and the database is not started.
  • Declined: nothing is touched and nothing starts.
• Upgradable, but nobody asked (auto-start, an MCP agent): GemDB does not upgrade. It shows a warning with an Upgrade Database… button that runs Start, which asks. The GemDB Shell's cold start meets the pre-start guard, which says how to get the upgrade.
• Refused as before: an older engine with no measured pair, or an unreadable version, still gets the "delete it" message. It now also names what GemDB can upgrade from. A database written by a newer engine is refused without the advice to delete it.
• Setup (prepareFiles) lets an upgradable database through, because setup touches no database.
• New databaseStarted outcomes: upgradeNeeded, upgradeDeclined, upgradeFailed, documented in docs/telemetry.md.

The long comment on assertDatabaseMatchesEngine, the CLAUDE.md section on engine upgrades, docs/automation-boundary.md (the upgrade is now under "Asked") and the CHANGELOG are updated. bundle-grail.sh and check-vsix.sh now require scripts/upgrade_engine.sh.

Testing

• npm test: 1316 passed. New coverage: version ordering and judging; the guard's messages; the upgrade steps against a fake world (declined, no disk room, stop order under the lock, never running the script while the stone is up, the restored and unconfirmed failure messages, a zero exit that the extent does not confirm); running the script itself (the arguments of the --in-place interface, and the reason taken from its last line); and how ensureRunning handles each trigger.
• lint, format:check, typecheck, typecheck:strict and lint:toolchain are clean.
• New local-only integration test, src/__integration__/engineUpgrade.test.ts. It needs GEMDB_TEST_UPGRADE_EXTENT, an a5 extent, and skips without it. It runs the real script from the bundled grail/ on a copy, in the fixture's own global dir. 4 of 4 passed with the a8 engine and Grail at 5d0333f, on a GemDB-shaped database (createDatabase's config, the a5 key, extent pregrown to 10240 MB): a decline leaves the database alone; a failure (a wrong from-version) restores it byte-for-byte and leaves nothing running; the upgrade itself took 45s, after which the database reads a8, the backup equals the original, nothing is left running, and the stamps are gone.
• By hand, end to end, with a separate global dir and stone names of its own:
  1. An a5 stone was run on a copy of an a5 Grail database and stopped, leaving a5 tranlogs.
  2. The bundled script upgraded the copy (50s).
  3. An a8 stone started on it normally, without -N.
  4. resources/install-grail.sh installed Grail 0c5e215 (exit 0), with "Recompiling 5826 Grail methods an engine upgrade removed".
  5. Python ran: a class, sum, a dict holding a tuple and None, and Unicode upper().
• Not run: the full npm run test:integration. It needs the pinned engine under ~/GemDB, which I did not touch.

Remaining

• PINNED_GRAIL_REF is Grail main, 29833cd (61d18f6): #1867 (the engine upgrade), #1888 (redeploy of dataclass defaults) and #1886 (compile() in an except SyntaxError: handler), all merged. The brain-freeze gate (ci/upgrade.sh: move a reader's GemDB to a new release and prove nothing was lost brain-freeze#101) reruns on this build.
• Release CI runs the gitleaks scan of the .vsix. The new script carries no password literal, but the payload has not been scanned yet.

🤖 Generated with Claude Code