Skip to content

🤖 refactor: replace the bash-monitor wake outbox with a level-triggered reconciler - #4029

Merged
ibetitsmike merged 45 commits into
mainfrom
mike/wake-reconciler
Sep 1, 2026
Merged

🤖 refactor: replace the bash-monitor wake outbox with a level-triggered reconciler#4029
ibetitsmike merged 45 commits into
mainfrom
mike/wake-reconciler

Conversation

@ibetitsmike

Copy link
Copy Markdown
Contributor

Summary

Replaces the edge-triggered bash-monitor wake outbox (bashMonitorWakeStore.ts, 3.5k LOC production + 5.5k LOC tests, plus ~1.9k wake lines inside workspaceService.ts) with a level-triggered BashMonitorWakeReconciler. One durable watermark comparison replaces five separate failure-recovery mechanisms; process events only poke the reconciler and are never the correctness mechanism. Net −13,596 LOC across production and tests.

Background

Card 1 of the 2026-08-31 architecture review. The wake store persisted every matched-line/settlement event as a durable wake record and kept exact-delivery correct through staged 2PC clears with tombstones, staged-clear heartbeat refreshes, deferred temp-file recovery timers, quarantine dirs with stranded-prune recovery, and snapshot rollback/restore tokens, with three lock maps and clear-commit retry timers leaking into workspaceService.ts. Edge-triggered wake-delivery hardening was already rejected once in favor of level-triggered reconciliation (#3990, terminalAttentionStore); this module was the remaining instance of that pattern.

Implementation

Durable state shrinks to two small things:

  • Registry rows (bashMonitorRegistryStore.ts, existing): lifetime extended from "until monitor retires" to "until all signals delivered"; rows gain an optional terminal summary (written at settlement) and a lost summary (written on runtime monitor failure), so a restart can deliver a real settlement/runtime-failure notice instead of a generic monitor-lost.
  • A per-workspace watermark file (atomic temp+rename): per process generation, matchedThroughOffset delivered, terminal delivered, lost delivered.

reconcile(workspaceId) (single-flight per workspace) derives outstanding signals from live BackgroundProcessManager state past the watermark: matched lines beyond max(watermark, shownThroughOffset), undelivered settlements (wakeOnExit), and registry rows without a live monitor. It composes one synthetic turn (prompt format unchanged), sends when the owner is idle, advances the watermark on acceptance, then deletes fully-delivered dead rows. All events (monitor:match/armed/stopped, output:shown, startup, history clear) reduce to scheduleReconcile pokes. Dispatches report in-flight vs deferred so a busy-owner deferral clears dispatch dedupe state and the post-idle re-poke redelivers.

History clear becomes a watermark bump (pre-bump, clear, post-bump; fails toward cleared); truncations no longer touch wake state. pendingWakeKind for the UI derives from the same reconciler snapshot. Legacy wake-record files are deleted best-effort once per workspace, never parsed.

Intentional behavior change: matched-line content that was never delivered before an app shutdown is not replayed after restart; the workspace still wakes with a settlement-summary/monitor-lost notice naming the process and filter.

Validation

  • make static-check, typecheck, and the four touched suites green on this head (reconciler 12, backgroundProcessManager 127, registry 8, workspaceService 401).
  • Two adversarial remote dogfood UAT rounds against pushed heads. Round 1 found two real defects: a wake owed while the owner was busy was never delivered on idle (dispatch-dedupe zombie), and settlement payloads replayed already-delivered lines / duplicated the settle marker. Both fixed with red-green tests. Round 2 verified both fixes (busy→idle delivery 4/4 scenarios, settlement payloads exact) plus a full regression sweep: task_stop silence, natural-exit settlement, restart produces exactly one monitor-lost wake with no replay, history clear no-replay with fresh post-clear wakes, 12-line burst coalescing.
  • One UAT scenario was unreachable through the real UI: clearing history while a wake is pending is blocked by the pre-existing "Cannot truncate history while a turn is active" gate (present on main, workspaceService.ts:12890); the guarded invariant (clear consumes pending signals durably; post-clear matches wake) is covered by reconciler unit tests and a workspaceService wiring test.

Risks

Wake delivery is intricate timing-sensitive logic feeding agent turns; the regression surface is background-bash monitor wakes (line-match, settlement, monitor-lost), the pending-wake UI indicator, and history-clear interaction. Mitigations: the behavioral invariants are pinned by tests on the reconcile() surface (busy deferral redelivery, watermark dedup across reconstruction, cancellation silence, maxEvents retirement, generation reuse, clear pre/post consumption, tail frontier filtering), and two UAT rounds exercised the real UI end to end. The restart-content change above is the one deliberate semantic difference.


Generated with xum • Model: anthropic:claude-fable-5 • Thinking: xhigh • Cost: $120.95

@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: f20d1fa739

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Comment thread src/node/services/workspaceService.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f20d1fa739

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/bashMonitorWakeReconciler.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts Outdated
Comment thread src/node/services/backgroundProcessManager.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts
Comment thread src/node/services/bashMonitorWakeReconciler.ts
Comment thread src/node/services/bashMonitorWakeReconciler.ts
Comment thread src/node/services/bashMonitorRegistryStore.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts
Comment thread src/node/services/workspaceService.ts Outdated
@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a75438b78f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/workspaceService.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts
@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 247619523e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/workspaceService.ts Outdated
Comment thread src/node/services/backgroundProcessManager.ts
Comment thread src/node/services/bashMonitorRegistryStore.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts Outdated
@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b234d7379f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/bashMonitorWakeReconciler.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts
@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 433f3c1b39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/bashMonitorWakeReconciler.ts
Comment thread src/node/services/workspaceService.ts Outdated
Comment thread src/node/services/bashMonitorWakeReconciler.ts
@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 59ff2bd1ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

This comment has been minimized.

@ibetitsmike
ibetitsmike added this pull request to the merge queue Sep 1, 2026
Merged via the queue into main with commit 1a6db60 Sep 1, 2026
19 of 20 checks passed
@ibetitsmike
ibetitsmike deleted the mike/wake-reconciler branch September 1, 2026 01:38
@mux-bot mux-bot Bot mentioned this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant