Skip to content

feat(work): harden cross-surface Work continuity - #790

Open
XuPeng-SH wants to merge 15 commits into
matrixorigin:mainfrom
XuPeng-SH:feat/durable-work-session-workspace-recovery
Open

XuPeng-SH wants to merge 15 commits into
matrixorigin:mainfrom
XuPeng-SH:feat/durable-work-session-workspace-recovery

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make one durable Work usable across TUI, Web, Server, and Edge without silently resuming or forking a Session.

  • TUI /work opens an owner-scoped catalog and asks the user to Observe Work or Continue this Work. Observe is read-only; Continue is an explicit Work-scoped turn and never mutates the current chat Session.
  • Work attachments carry an explicit surface identity (cli, tui, web, app, server, or edge). The SDK, Server, TUI, Web, and tests use the same strict contract.
  • Work continuation consumes SSE lifecycle facts incrementally. Tool, approval, prompt, waiting, pause, artifact, usage, and response progress are visible while a turn runs.
  • Partial output and request identity survive stream interruption. Unknown 5xx responses stay retryable, retries reuse the same request and attachment identity, and the first resolved branch remains pinned even if another client changes the delivery branch.
  • Root Run scoping prevents descendant Runs from settling the Work early. Durable text_done replaces a provisional streamed prefix so replayed answers are complete and never duplicated. Final outcomes are reliably delivered even when progress queues are full; late progress cannot overwrite a terminal state.
  • /work save remains an honest logical checkpoint boundary for Work/branch/session facts. It does not claim to snapshot code, data, credentials, artifacts, pending effects, or an active Run.

User journey

  1. Start or promote a Work from TUI or Web. TUI requires an existing durable Session and never creates a hidden one.
  2. Open the same Work from another authorized TUI or Web surface. The catalog, plan, activity, transcript, execution placement, and freshness are bounded Server projections.
  3. Observe without taking control. A reader does not acquire the branch writer lease.
  4. Explicitly Continue. A controller is admitted through the canonical Session/Run path; conflicts and stale bases give a safe refresh or takeover action. TUI displays lifecycle progress and a final Work-scoped response while leaving the active chat Session unchanged.
  5. If the stream is uncertain, inspect the Work and retry the exact request id. No blind second Run is created.
  6. Change the next-turn Edge only between settled Runs and only after the target workspace is verified clean and matching. Active Run migration and automatic workspace transfer remain outside this release.
  7. Review the task graph, evidence, artifacts, Done-when criteria, and selected branch before delivery.

Architecture and limits

The Server owns Work, branch, Session, Run, controller, and provider-binding facts. Surface clients are projections and explicit command issuers; they do not grow separate lifecycle or checkpoint state machines. Edge replacement changes future provider selection, not an in-flight Server-owned agent loop.

Workspace snapshots/materialization, MatrixOne data portability, artifact location transfer, active Run recovery after Server loss, and inline TUI approval/prompt interaction require separate contracts. The current release reports these boundaries instead of implying that a logical recovery point can restore them.

Verification

  • cargo fmt --all, git diff --check, targeted cargo check, and cargo clippy -p astra-cli --all-targets -- -D warnings passed.
  • cargo test -p astra-cli --lib: 5,053 passed, 9 ignored.
  • Work continuation regression set: 6 passed, including queue saturation, branch pinning, 5xx unknown outcome, root/child lifecycle scoping, text_done replay, partial output, and cleanup failure.
  • cargo test -p astra-thin-client --lib: 186 passed; cargo test -p astra-server-types --lib: 167 passed.
  • SDK typecheck/build and contract tests passed (322 passed, 13 skipped); Web typecheck, 586 Vitest tests, and production Next build passed.
  • MatrixOne Work DB integration tests remain opt-in in this environment (20 tests discovered, 20 skipped without ASTRA_TEST_DB_IT=1).
  • GPT-6 medium review approved the final frozen diff with no remaining P0/P1/P2 findings.

Compatibility

The current contract intentionally requires the explicit surface field and does not retain old request shapes or compatibility aliases.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 50f51a7 from the storage, validation, lifecycle-cleanup, harness UX, and failure-path perspectives. This is my own PR, so I am submitting COMMENT; the two inline P2 findings should be fixed before merging.

The preparing-only boundary is a sound scope choice: persisted caller input does not become a verified restore point, capability assessment does not promise restore/continuation, and branch deletion includes the new records. I am not treating the explicitly deferred ready publisher, content upload, or restore UI as missing functionality in this PR.

Validation actually executed: python3 -m unittest discover -s scripts/schema -p 'test_*.py' — 47 tests, 44 passed, 3 failed, all exposing the missing work_recovery_points inventory registration. The relative executable-path regression is source analysis, not an executed Rust reproduction. This environment has neither cargo nor rustc, so I could not independently run the Rust suites or MatrixOne integration tests. GitHub reported Static Checks successful while Test Suite was still in progress when inspected; the local schema failures remain reproducible despite that green check.

Additional test-quality gap: the new repository unit test checks request-hash inequality, but the two DB integration tests do not exercise identical request replay, changed-payload request reuse, or concurrent replay. Please test these through record_preparing and assert returned identity, conflict behavior, and row count; a hash helper test alone does not verify transactional idempotency. Existing owner-isolation and branch-cleanup integration coverage is useful.

"work_runtime_event_outbox",
WORK_RUNTIME_EVENT_OUTBOX_CREATE_SQL,
),
("work_recovery_points", WORK_RECOVERY_POINTS_CREATE_SQL),

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Register the new table in the closed-world schema inventory

Adding this production table without updating scripts/schema/schema_inventory.py::TABLE_METADATA and the expected Work table set in scripts/schema/test_schema_inventory.py breaks the existing schema contract suite. On this exact commit, python3 -m unittest discover -s scripts/schema -p 'test_*.py' runs 47 tests and fails three: test_every_production_table_has_semantic_metadata (one unclassified table), test_storage_and_work_metadata_are_explicit_and_closed_world, and test_work_schema_source_and_exact_table_set; both set failures identify work_recovery_points. Add its semantic ownership/retention/rebuildability metadata and update the expected table set, then rerun the suite.

command
.env("NO_PROXY", "localhost,127.0.0.1")
.env("no_proxy", "localhost,127.0.0.1")
.current_dir(probe_workspace)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Resolve the executable before changing the probe's working directory

resolve_astra_bin in main.rs returns explicit --astra-bin paths and ASTRA_BIN values unchanged after checking them in the caller's directory. Thus a supported invocation with --astra-bin ./target/debug/astra (or the equivalent ASTRA_BIN) passes binary/server checks, but this new cwd makes Command::new(astra_bin) resolve that relative executable under the empty temporary directory on Unix. Every model then fails with spawn failed: No such file or directory, even though the binary/model is healthy. Canonicalize/absolutize the executable against the original cwd before constructing subprocesses, including the retry/registration paths. Add a public preflight regression using a relative executable and assert that it runs successfully with the disposable workspace as cwd.

@XuPeng-SH

Copy link
Copy Markdown
Collaborator Author

Addressed the review findings in 731faa2fd:

  • registered work_recovery_points in the closed-world schema inventory and updated its authority/retention metadata;
  • added MatrixOne integration coverage for exact replay identity, changed-payload request conflict, concurrent replay, and the one-row idempotency invariant;
  • reran schema inventory (47/47), targeted Clippy (-D warnings), relevant unit suites, and the ignored MatrixOne recovery-point integration tests (2/2 passed locally).

The stage remains preparing-only: there is still no ready publisher or restore claim.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 731faa2, including the delta from 50f51a7 and the surrounding persistence, manifest validation, cleanup, and preflight paths. This is my own PR, so I am submitting COMMENT.

The schema-inventory finding from review #790 (review) is fixed: I independently reran python3 -m unittest discover -s scripts/schema -p 'test_*.py' and all 47 tests passed. git diff --check e215c032 HEAD also passed. GitHub Test Suite, Static Checks, and PR Title are green for this head.

The previous P2 relative-executable finding remains unresolved. Neither resolve_astra_bin nor the preflight subprocess setup changed: --astra-bin ./target/debug/astra / relative ASTRA_BIN is still returned unchanged and passed to a command whose cwd is the new temporary workspace. Please resolve the executable against the original cwd before changing subprocess cwd. I am retaining the original finding rather than opening a duplicate inline thread. This is source-confirmed; I have not claimed an executed Rust reproduction.

The new repository test now meaningfully covers identical replay, changed-payload request reuse, and simultaneous replays of an existing record. One remaining coverage distinction: both concurrent calls happen after the row has already been inserted, so they do not exercise concurrent first admission. Add a separate test with a fresh request/recovery-point identity, start both record_preparing calls before any insert, and assert the intended results plus a single row by request identity. This is a test gap, not proof that the transaction implementation fails.

I also inspected the workspace portability boundary; an additional non-blocking validation concern is inline. The preparing-only scope remains appropriate and does not yet promise a usable restore point.

Validation limits: cargo/rustc are unavailable here, so I did not independently run Rust or MatrixOne tests. I ran a small temporary-directory filesystem experiment for the symlink-resolution example in the inline comment; that verifies OS path resolution, not execution of the Rust validator or an Astra restore flow.

return false;
}
let mut components = path.split('/').collect::<Vec<_>>();
components.pop();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validation boundary to address before ready publication/materialization (non-blocking for this preparing-only scope): this checks target components lexically and does not resolve other snapshot symlinks. For a snapshot with a -> . and b -> a/../outside, both targets pass this helper by inspection, but following b resolves to the workspace's parent/outside because a resolves to the workspace root before .. is applied. I confirmed the latter with real symlinks in a temporary directory. Please add a manifest-level chained-symlink regression and either validate the complete link graph with bounded cycle detection or explicitly document this as lexical validation while requiring containment enforcement at the materializer. The current helper must not be treated as proof that all links stay inside the workspace. There is no restore path in this PR, so this is not a claim of a currently exploitable restore vulnerability.

@XuPeng-SH
XuPeng-SH force-pushed the feat/durable-work-session-workspace-recovery branch from 731faa2 to 505de37 Compare September 16, 2026 15:01
@XuPeng-SH XuPeng-SH changed the title feat(work): persist owner-scoped recovery captures feat(work): harden cross-surface Work continuity Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant