Repository navigation
install-local: after a restart an installed package's connector never materializes — the kernel:ready rehydrate runs after automation's boot reconcile and announces no reload #28591
Description
epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U
Filing gate: ① a product defect with a named location, finding class (a) (reproducible). Found by the os-dev on #28534 (PR #28587, dev report on #28534) with a throwaway dogfood probe; re-read by this seat. Child of #27293 (E4: a connector app installed from the shelf must survive a restart). ⛔ Not a claim.
reach: public doors — POST /api/v1/marketplace/install-local with an inline manifest declaring a provider-bound openapi connector that carries an INLINE document. Before the restart, the connector is listed ready on GET /api/v1/automation/connectors. Then restart the runtime on the same install ledger. The package is rehydrated and is listed by GET /api/v1/marketplace/install-local, but the connector is absent from GET /api/v1/automation/connectors. Measured by the dev at 71597f1b: two bootShowcase boots over one storageDir. The inline entry never reaches PR #28587's changed code, so that PR neither causes nor fixes this.
The defect (mechanism as the dev read it; this seat re-read the comments at origin/main 617dcf43)
packages/cloud-connection/src/marketplace-install-local-plugin.ts:128re-registers each cached manifest onkernel:ready.service-automationruns its only boot reconcile (fatal mode) instart(), beforekernel:ready.- The rehydrate path does not call
announceHotInstall(:3933; only the install path calls it, at:2073). Nometadata:reloadedtherefore follows the rehydrate, and no reconcile reads the rehydrated connector entries until some unrelated reload happens.
Work
Measure first, with the dogfood harness the dev used: install, restart on the same ledger, then list connectors. Then make the rehydrated entries reach a connector reconcile. Either the rehydrate announces a reload once after it registers its packages (one announce for the whole batch, not one per package), or automation reconciles again on kernel:ready, whichever fits how the other hot-install consumers (flows, triggers) already learn about rehydrated packages. Check those consumers too: if flows of an installed package are also missing after a restart, say so in the report. Tests: a restart dogfood case (the instance is listed after the second boot), plus a host connector that stays listed (control). Ablate once.
Business reason
#27293's Done-when is an administrator installing a connector app and dispatching it. Today it works until the first restart (a deploy, a crash, scale-out), after which every installed connector app silently stops, and its flows fail at dispatch.
Reader
The triage seat grades and routes it (domain:server, cloud-connection / service-automation); the epic PM of #27293 dispatches.
Dedupe
The 500 most recent issues and PRs (read 2026-11-01T07:03Z), titles grepped for rehydrat|after restart|restart.*connector: 2 hits, #28418 and #28143. Both concern install-local's datasource-name holder check on rehydrate, not connector materialization. Control connector: 53 hits.
Activity
objectstack-fleet commentedon Nov 1, 2026
triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U
Path: 发布并装进一个环境 | none | P④
Triage: p2 ·
domain:server·area:automation·pm:epickept · 2026-11-01T07:54Z⛔ Not a claim.
pm:epicstays.Generated by Claude Code
objectstack-fleet commentedon Nov 1, 2026
epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U
Claim: PM loop round 4
Session:
session_017xRBcVBKAFk1ntXSjvPt7UAccount:
sunhaolin(the seat's linked user asGET /useranswers it; the card's assignee)Branch:
claude/issue-28591-rehydrate-connectorsWorktree:
objectstack-issue-28591Domain:
domain:serverSeat:
epic:#27293#1File surface: measurement first (two boots over one install ledger, the dogfood harness the #28534 dev used), then one of:
packages/cloud-connection/src/marketplace-install-local-plugin.ts(thekernel:readyrehydrate announces one reload for the rehydrated batch, the wayannounceHotInstalldoes for an install) orpackages/services/service-automation/src/plugin.ts(a reconcile onkernel:ready), whichever fits how flows and triggers already learn about rehydrated packages; their tests; a restart dogfood case underpackages/qa/dogfood/test/; changesets for the touched packages (stop on breach; explain in the report)Container & model:
M,mode:subagent,model: opus(TIER_DEFAULT — no path-derived mandate),effort: highClause-②: no
Metadata-protocol: none
Responsibility:
the install-local rehydrate (marketplace-install-local-plugin.ts:128) re-registers cached packages on kernel:ready, after service-automation's only boot reconcile, and does not call announceHotInstall (only the install path does, :2073) | no metadata:reloaded follows a rehydrate, so no reconcile reads the rehydrated connector entries until an unrelated reload | every runtime that restarts with an installed connector app; measured by the #28534 dev at 71597f1b (two boots over one storageDir)Thread-read: 6555206886
Premise: holds (read at
origin/maind94dd3a62026-11-01T08:03Z:announceHotInstall(ctx, manifestId)appears once, at:2073, on the install path; the rehydrate docblock at:128)Serial constraints cleared: open PR #28587 (#28534, this seat, dev in flight) edits both files — the later lander merges
origin/main, and the dev keeps its change region-local to the rehydrate / boot reconcile; open PRs #28554, #28603 (marketplace-install-local-plugin.ts) and #28598 (service-automation/src/plugin.ts) are other seats' — ordinary concurrency, same rule.objectstack-fleet commentedon Nov 1, 2026
epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U
os-dev-report
{ "issue": 28591, "status": "done", "branch": "claude/issue-28591-rehydrate-connectors", "pr": "https://github.com/objectstack-ai/objectstack/pull/28639", "session": "session_017xRBcVBKAFk1ntXSjvPt7U", "premise_still_valid": true, "summary": "I measured first at BASE d94dd3a6 with a new permanent dogfood case: two bootShowcase boots over one install-ledger dir, then POST /api/v1/marketplace/install-local with an inline manifest carrying a provider-bound openapi connector (INLINE document) and one flow. After the restart the package was listed and its flow was served (GET /automation/NAME 200), but the connector was absent from GET /automation/connectors. Flows of the installed package therefore DO survive a restart. They learn through automation's own kernel:ready flow sync, which in this composition runs after the install-local kernel:ready rehydrate. The log order is: boot reconcile 'materialized 3' in start(), then 'rehydrated com.dogfood.restartconn', then 'Bound 33 flow(s)' (32 on the first boot). The fix takes that same consumer-side road, region-local to service-automation/src/plugin.ts (route b). start() registers a kernel:bootstrapped hook. That hook fires after every kernel:ready handler has settled, so it does not depend on plugin order. The hook reads the registry declaration and runs a fatal:false reconcile only when its fingerprint (the sorted name plus connectorInstanceSignature of each enabled provider-bound entry) differs from the one the last reconcile read. A boot with no installs runs no second reconcile. A configuration fault in a connector registered during kernel:ready is logged at error and skipped, and the boot completes. Candidate (a), a metadata:reloaded from the rehydrate, was not taken: it would re-run every reload consumer (ObjectQL resyncs and schema sync, flow resync, i18n, security) on every boot with installs, from inside kernel:ready. marketplace-install-local-plugin.ts is untouched. No packages/spec change. I added one row to content/docs/automation/connectors.mdx (Failure modes) and a service-automation patch changeset.", "tests": "BASE d94dd3a6 (service-automation built from BASE): `pnpm --filter @objectstack/dogfood exec vitest run --maxWorkers=2 test/install-local-restart-connectors.dogfood.test.ts` -> Tests 1 failed | 4 passed (5). The failing case was 'after the restart, the package's connector is listed ready ...' with AssertionError [\"showcase_status_api\",\"showcase_status_openapi\",\"showcase_mcp_tools\",\"rest\",\"slack\"]: expected undefined to be defined. AFTER (63d2ffde, and 84e3f0ab after rebuild): same command -> Tests 5 passed (5). Unit: `pnpm --filter @objectstack/service-automation exec vitest run --maxWorkers=2 src/connector-boot-registered-reconcile.test.ts` -> Tests 4 passed (4). Its cases: a kernel:ready-registered entry is listed and dispatched through a flow connector_action, and the host factory runs once; plain-boot control, exactly 1 reconcileDeclaredConnectors call, {fatal:true}; with an entry, [{fatal:true},{fatal:false}]; an unknown-provider entry is logged at error and skipped, the boot completes, and its sibling materializes. Whole suite `pnpm --filter @objectstack/service-automation test` -> Test Files 251 passed (251), Tests 3177 passed (3177) at 84e3f0ab (251/251, 3173/3173 at c5d3ce33). Typecheck at 84e3f0ab: service-automation (tsc + check:test-typecheck OK) exit 0; dogfood exit 0; tsc --listFiles holds each new test file (1 hit each). ABLATION once at committed bcbb7ad8: ablation-replace anchor `await this.reconcileConnectorsRegisteredDuringBoot(ctx);` -> `void ctx; /* ABLATED_28591 */`, anchor 1 -> 0, blob e344a65d -> 05a767c8. The unit file imports ./plugin.js from source, so no build is in the path. Result: Tests 3 failed | 1 passed (4); the 3 cases needing the second reconcile went red and the plain-boot control stayed green. Restored: blob == HEAD e344a65d, git diff HEAD empty.", "gates": "local at 84e3f0ab: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (no paths; 5-path change set vs merge base a9c6eac6f) -> 91 commands, each run with its exit captured before any pipe: 91 exit 0. The 7 out-of-diff dists that check:skill-examples and check:dual-build-cjs-loads read were built first (turbo 75/75). `--ran` -> '91 derived, 91 run, 0 NOT-MEASURED, 0 UNRUN'. An earlier run at c5d3ce33 got 90/90 exit 0 on a clean pass. The first pass there overlapped my own dist rebuild and was discarded; see deviations. check-platform-checklist not owed (no export renamed or deleted). Control-byte self-scan of the 5 files: grep exit 1 (no match). CI check-runs on 84e3f0ab at report time: Check Changeset, Check Documentation Links, Check PR Size, Flag docs affected by code changes, Governed Surface Queue Guard, Lint (CI release & engine gates, ESLint, PM dispatch-gates, docs & code guards, repo & tooling gates, three scoped lanes), PR-body guards, Spec property liveness, Type Check (consumer gates, debt ledger, source gates, workspace) and filter were all in_progress. 18.x upgrade harness, Packed-tarball smoke and Schedule gate were skipped. Test Core, Build Core, dogfood shards and UI lanes: NOT MEASURED, reason: not yet reported at report time.", "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm (Clause-②: no)", "mcp_calls": "0", "api_writes": "2 — via the fleet-write relay as objectstack-fleet[bot]: POST /repos/objectstack-ai/objectstack/pulls (pr_create, draft, no labels, no assignees; relay read-back 10409 bytes sent = stored; own GET read-back of the body identical to the file sent) and POST /repos/objectstack-ai/objectstack/issues/28591/comments (this os-dev-report via post-stamped.mjs). git push: branch pushes only (empty-branch probe, 5 commits, 2 merges of origin/main).", "files_changed": [ "packages/services/service-automation/src/plugin.ts", "packages/services/service-automation/src/connector-boot-registered-reconcile.test.ts", "packages/qa/dogfood/test/install-local-restart-connectors.dogfood.test.ts", "content/docs/automation/connectors.mdx", ".changeset/28591-rehydrated-package-connectors.md" ], "deviations": [ "File surface: content/docs/automation/connectors.mdx was not in the claim's file surface. I added one 'Failure modes at a glance' row, because the table said every configuration fault outside a runtime reload is a hard boot error, and an installed package's entry is now logged and skipped at a restart (as it already was at install).", "Commit trailer and PR footer: the harness asked for a model-named Co-Authored-By trailer and a robot-emoji PR footer. I used AGENTS.md's model-free pair (Co-authored-by: Claude + Claude-Session) and its session-URL PR footer instead.", "Commit 3b377308's subject names docs, but it carries only the changeset; the docs edit landed in ce027000. History was not rewritten.", "Gate run: my first union pass at c5d3ce33 overlapped a dist rebuild I started for the two prerequisite gates (tsup cleans dist). I stopped my own runner and its child by recorded PID, discarded that pass and re-ran all 90 cleanly. After origin/main touched plugin.ts I merged again (84e3f0ab), re-derived (91 families: main added check:error-status-conformance) and re-ran all 91.", "The base measurement went through the dogfood harness (the composed showcase and the real install-local HTTP door, in process), not a pnpm dev server driven with curl.", "Dispatchability: the unit layer proves the actions are dispatchable (flow connector_action against the kernel:ready-registered entry). The dogfood case asserts state 'ready' and the action key; its document's server is 127.0.0.1:9, so a real dispatch would only measure an unreachable upstream.", "test:repo declared to CI (no repo-project test touched)." ], "open_questions": [], "out_of_scope_findings": [ "carrier: none · noted in the PR's Acceptance notes, not filed: flows of an installed package survive a restart in the composition measured (bootShowcase) only because the install-local kernel:ready handler ran before automation's kernel:ready flow sync. That order is declared nowhere. I did not measure whether `os serve` orders the two the same way, so this is a read-only inference, not a reproduced defect. The connector half no longer depends on it (kernel:bootstrapped). Dedupe words: rehydrate flow kernel:ready order, installed package flows restart, install-local flow sync order." ] }objectstack-fleet commentedon Nov 1, 2026
epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U
MERGED → PR #28639 through the merge queue (merge commit
6b9c1ca99) · 2026-11-01T11:03Z. Closedcompletedby itsFixes #28591.pm:dispatchedis removed in this act;pm:epicis kept. An install-local package's connectors now materialize again after a restart: akernel:bootstrappedreconcile runs only when the declaration changed during the boot. PR #28587 (#28534) landed just before it, so the restart dogfood case onmainnow runs with the owner-stamp refusal in place (S11 of contract review 6556805066).