Skip to content

fix(work): harden durable recovery and TUI controls - #792

Open
XuPeng-SH wants to merge 13 commits into
matrixorigin:mainfrom
XuPeng-SH:feat/durable-work-recovery-verified-publish
Open

XuPeng-SH wants to merge 13 commits into
matrixorigin:mainfrom
XuPeng-SH:feat/durable-work-recovery-verified-publish

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

A durable Work could still feel unusable at the boundaries: a fresh TUI conversation could not start /work start without a throwaway chat turn, a full Work board could cover the command palette after /work, and Web could reject valid executor labels such as MacBook Pro as if they were protocol identities. Multi-session recovery also had a workspace-fence race: delayed cleanup from one turn could clear the activity marker for a later turn under the same retained writer, and that later turn could not renew because the writer still carried its original acquisition cursor.

Change

  • Make Session execution and checkout ownership explicit, serialized, and ABA-safe across CLI, Server, Web, and Edge paths.
  • Revalidate the exact workspace claim after reservation and durable run start, before model/tool side effects; classify writer, cursor, authority, claim-lost, and workspace-busy failures separately.
  • Key workspace activity by the per-turn reservation identity and writer epoch. Renewal validates the caller reservation against the locked active reservation and canonical head, while writer acquisition cursor identity remains stable across retained turns. Release of a writer retires the whole Session authority under the same lock.
  • Repair derived conversation-log state from a validated canonical journal and quarantine corrupt derived state instead of poisoning future turns.
  • Keep new Sessions attached after rejection and make recovery explicit; ordinary input clears stale startup hints, and TUI errors explain that no model/tool ran and how to retry or open a new Session.
  • Allow /work start <goal> to bootstrap the durable Session it needs, while reusing an existing Session without extra transport or auth work.
  • Keep the TUI composer and slash palette visible when the inline Work board is large, and resync popup ownership at native-command and Work-completion boundaries so the next /… command is immediately discoverable.
  • Namespace Web card keys so execution and patch cards for one branch never share a React key.
  • Treat executor names, display names, and hostnames as human-readable labels while retaining strict validation for protocol identities.
  • Document the user journey for multi-client observation, serialized writes, idle transfer, and stale-owner recovery.

Validation

  • make format-check
  • make lint
  • make check
  • make test-offline (baseline Rust workspace, runtime e2e, SDK, Web, and build gates passed before the final TUI/SDK delta)
  • make test-contract (16 contract tests passed)
  • TUI targeted tests: slash integration 21 passed, task-board draw 17 passed, fresh Work session 2 passed
  • SDK npm run typecheck, work-contract tests (65 passed), and npm run build
  • Full local core online lane: 59 runtime + 11 turn-core + 386 services tests passed
  • Session authority DB integration: all 7 live MatrixOne tests passed, including retained-writer R1→R2 renewal, tampered-cursor fencing, and stale-cleanup protection
  • Dynamic Edge spawn E2E passed with the CI-aligned event synchronization budget
  • DeepSeek Flash harness isolation run: 13/17 scenarios passed; 4 remaining failures are model/task-quality outcomes, with no runtime corruption or checkout collision
  • Separate SDK workspace request types from runtime event bindings. Web now sends the strict canonical kind/display_name/root/source/authority shape; Edge requests use root plus source.kind=edge_path and never send event-only cwd or fallback_policy.
  • Add a shared workspace-binding contract fixture consumed by Web route tests and Rust deny_unknown_fields tests, plus SDK consumer typecheck assertions that reject event-only request fields.

@XuPeng-SH
XuPeng-SH force-pushed the feat/durable-work-recovery-verified-publish branch from c1304bd to d880cda Compare September 16, 2026 08:18
@XuPeng-SH XuPeng-SH changed the title fix(session): make concurrent recovery user-safe fix(work): harden durable recovery and TUI controls Sep 16, 2026
@XuPeng-SH
XuPeng-SH force-pushed the feat/durable-work-recovery-verified-publish branch from 437a8fb to 7d8558f Compare September 16, 2026 12:36

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep-reviewed 7d8558f17d347646f1f3aef18477157c6f28cd0e. This is the connected user's own PR, so I am submitting COMMENT. One P2 concurrency finding should be addressed before merging; see inline. Upgrades and migrations are outside this review's acceptance scope.

The product direction is useful: a fresh /work start can establish its Session without a throwaway chat turn; the composer/palette gets priority over a large board; executor display labels no longer masquerade as protocol identifiers; rejected work stays attached to the actual Session and tells the user no model/tool ran. Quarantining a corrupt derived CSL only when a canonical journal continuation exists is also the right authority boundary.

I inspected the workspace activity claim lifecycle and transfer checks, retained writer/reservation identities, admission reserve/renew/release and materialized capacity accounting, recovery publication and effect-frontier checks, CSL repair, harness worktree isolation, and the TUI/SDK changes. In particular, the recovery publisher rechecks its canonical anchors transactionally, does not publish caller-authored manifests, and blocks unresolved invocation effects. Its user-facing assessment correctly says workspace/run content is not captured and restore is unavailable. This is a verified Work/conversation boundary, not yet portable workspace recovery or a complete restore experience; keep that distinction in the product wording.

The remaining problem is that the new activity fence is per writer lease, while its clearing API is used as per-turn cleanup. Different reservations under a retained writer share the same cleanup identity, so the advertised stale-cleanup protection does not hold across those turns.

Validation independently executed: 47 Python schema tests passed; git diff --check against the PR base passed. I also extracted the production cleanup UPDATE and checked its predicate in an in-memory SQLite fixture: an old cleanup with the retained writer's lease/epoch clears the row representing a newer reservation under that same writer. This confirms the predicate collision, not an end-to-end Rust/MatrixOne concurrency reproduction. cargo/rustc and local SDK dependencies are unavailable, so I did not run Rust, live DB, SDK, model, or PTY tests. At inspection, GitHub Static Checks passed and Test Suite was still running. Head rechecked unchanged before submission.

active_execution_expires_at_ms = NULL, updated_at = NOW(6)
WHERE isolation_domain = ? AND owner_user_id = ?
AND session_id = ? AND branch_id = ?
AND active_execution_id = ? AND active_execution_generation = ?",

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fence activity cleanup by the turn reservation, not only the writer lease

The pair in this predicate is not unique to an execution: reserve_turn generates a new reservation_id but copies lease_id and writer_epoch from the retained writer, and installs those latter two values as the activity identity. commit_turn clears the reservation while retaining that writer. Therefore R1 and a later R2 under the same writer L/epoch E both use (L, E) here.

A delayed R1 clear_execution_workspace_activity after R2 reserves will erase R2's live marker. The public cleanup method locks the head but never checks its active reservation, and runtime settlement invokes this cleanup separately after commit_turn and WAL retirement (including the AlreadyApplied result). At minimum, R2's subsequent claim verification can now reject a valid admission; in the pre-Run/slot window this also removes the activity evidence intended to prevent another Session taking the checkout. This conditional UPDATE is safe against a new writer, but not a new turn under the same writer.

Use reservation_id (or another per-execution generation) consistently for activation, verification, renewal, and cleanup, or explicitly reject stale per-turn cleanup against the current reservation. Add a DB regression that reserves/commits R1, reserves R2 using the same live writer, invokes R1 cleanup, and proves R2 still verifies and another Session still cannot take the claim. The current activity test covers distinct controllers but not this retained-writer case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant