[NancyKwongPerplexity]

Summary

Merging this makes an HTTP (SSE) reattach to a native process session replay the output the guest still retains (up to 4 MiB per process), instead of losing it. resync_required is sent only when the stream cannot continue exactly from where the client is. After evicted output, the stream continues from the oldest output still retained rather than jumping to the live tail, including against a process that keeps writing.

Before this change, on main:

• Reattach without Last-Event-ID. sse.rs never sent a cursor. An exited process came back as only process_started and process_exited, with no stdout and no resync_required. A running one followed the live tail.
• Reattach with Last-Event-ID. The stream only renumbered behind resync_required(cursor_replay_unavailable), and missed output was dropped.
• Backend reconnect after a reattach. It reopened at a cursor counted from that attach's own frames, so it replayed stale output. It also always sent resync_required(process_stream_reconnected).

What changes:

Event ids carry the guest journal position. Process SSE ids are tagged integers that a resume decodes, so any data-plane replica can serve it.

• Layout: bit 52 is a tag, the low 16 bits are always zero, then a 2-bit frame counter and a 34-bit position field. Every id is below 2^53, so a JavaScript Number (or any double) holds it exactly. The field's top bit marks blocks of ids without a position; those blocks sort above every position.
• Only frames that move the position carry an id: stdout, stderr and resync_required, which acknowledges a gap. process_started, process_exited, backend_reconnecting/backend_reconnected, error and closed carry none, so an EventSource keeps the id of the last output it received. Resumes and backend reconnects against an idle process therefore spend no frames at its position. Output and every gap that keeps the position move it, so a position has at most two ids (the unsent start and the resync_required that begins a replay for an id without a position). File-watch ids are unchanged.
• Never misread: ids from earlier data planes, and ids an earlier plane counted on from by +1, fail to decode, so they are never read as a position. The exception is a count of exactly k × 65,536 frames (see Rollout).
• Strictly increasing: ids never repeat or go back, including across a resume. Every id a resumed stream sends is above Last-Event-ID.
  • A position too large for the field (more than 2^33 - 1 output chunks), running out of the 4 frames at one position, a lower position, or a jump to the live tail move the stream to a fresh block with no position.
  • A stream that has to count the output from the start again numbers its frames in a fresh no-position block above Last-Event-ID. This covers a resume from an id without a position, a cursor past the output, and a foreign id on the attach that starts the process. For an untagged id, positional ids already sort above it.
• 400: a Last-Event-ID at or above 2^53, or too close to the end of the id space to number a later frame above it, is refused. Only a forged id can be that high. The refusal is counted as an attach result (invalid_last_event_id).

Resume behavior:

• Without Last-Event-ID, the stream replays from the start of the output.
• With Last-Event-ID, it continues at the position the id carries, with no resync. That includes an attach that finds the process already started but its binding never persisted.
• From an id without a position, it replays the retained output behind resync_required(cursor_replay_unavailable).
• A backend reconnect reopens at the position the client has reached. It sends resync_required(process_stream_reconnected) only when the stream has no known position.
• Evicted output produces resync_required(cursor_out_of_window) once, then the oldest output the guest still retains, with ids that keep their position, so a resume from the gap's id is exact. The guest starts the stream there itself (see Guest), so this holds against a process that keeps writing: the reattach becomes the guest's primary reader at that output, and nothing written afterwards can evict it before it is read. A fresh attach to a process that wrote more than the window gets the retained output this way, not only its exit.
• Fallbacks to the live tail. A guest older than this change does not say where its retained output starts, and the stream follows the live tail instead, as it did at this pull request's previous head. (On main, a lag in the middle of a stream ended it with an error.) A stream already moved to the oldest retained output that falls behind again before any output, which can happen when another reader holds the primary place, also follows the live tail rather than chasing the next oldest output.
• The attach that starts the process counts its output from the start, behind cursor_out_of_window, even if the client sent another stream's Last-Event-ID.
• One process_started per response. It is sent once, even when the first backend stream fails before reporting the process. A gap is reported once until output follows it, so a cursor past the output whose start is also evicted sends one cursor_out_of_window.
• Window-miss reopens. After 3 reopens in a row without output, a cursor past the output, or a live tail that keeps falling behind, ends the stream with a retryable error. Evicted output never does. A reopen refused as a stale route (for example an expired route token) goes through the same route refresh as a backend reconnect. Each gap is logged with its cause (cursor_ahead, output_evicted, foreign_last_event_id).

Guest (process.rs, constants in common/core):

• A connect whose cursor is past the output the process has written used to wait for output to reach it, skipping everything before (or got only the exit). It now ends with OUT_OF_RANGE, tagged space-error-reason: process-output-cursor-ahead.
• ConnectProcessRequest.resume_from_oldest_retained (new, internal proto) starts a cursor before the oldest retained output at that output. The guest chooses the start under the journal lock that also claims the primary reader's place, and reports it in ProcessConnected.output_cursor (new). The native data plane sets it on every connect; ECW streams do not. Sandbox-manager and the spaced host client forward both fields.
• Without the flag, the RESOURCE_EXHAUSTED that ends a lagged stream is tagged space-error-reason: process-output-lagged, with space-process-output-first-sequence naming the oldest retained cursor; the data plane then reopens there. It treats only a tagged status, or an untagged one with the older guest's lag message, as evicted output. Any other RESOURCE_EXHAUSTED (a hop's refusal of an oversized chunk, an HTTP/2 reset, "replay state was evicted") ends the stream with a retryable error instead of a jump to the tail.
• All of this is documented on ConnectProcessRequest and ProcessConnected in operations.proto. The node simulator fails a cursor past the output as the guest does, instead of clamping it, and reports the start it connects at.

Docs. The OpenAPI text for GET /v1/sessions/{session_id}/events and its Last-Event-ID parameter (public_docs.rs, schema_contract.rs) now states:

• echo ids exactly; they increase but are not consecutive, and stay below 2^53;
• on a process stream only stdout, stderr and resync_required carry ids;
• ids carry the output position only for a process's first 2^33 - 1 output chunks;
• an id is only valid for the session whose stream sent it;
• without Last-Event-ID a stream starts at the beginning of the output; evicted output is followed by cursor_out_of_window and the oldest output still retained;
• ids that do not resume exactly: one from an earlier SPACE version, one altered in transit, and one sent after the stream replayed the output from its start behind cursor_out_of_window or continued with live output;
• every id a resumed process stream sends is above Last-Event-ID, and when resync_required is sent.

The DataPlaneAttachInfo.events_url schema text (public/api), the drain diagram, the API catalog, first-sandbox.md, troubleshooting.md and architecture.md are updated to match.

No public schema, enum or migration changes: Last-Event-ID stays int64, and cursor_out_of_window was already a published resync reason. The two new proto fields are on internal node and guest messages. m1_conformance.py and smoke-spaced-rest.sh now assert increasing (not consecutive) ids, that only output and resync_required carry them, and no resync or duplicated output on a lossless resume. The data-plane restart scenario takes its reconnect cursor from the first output instead of process_started. File-watch streams, WebSocket sessions and ECW are unchanged. This changes documented native-API behavior and targets the next 0.9 release.

Validation

Rebased onto main at 9b1985d.

At this head (329fcb8):

• cargo fmt --check passes.
• cargo clippy --all-features --all-targets -- -D warnings on pplx-space-data-plane-server, pplx-space-api, pplx-space-spaced-guest-execution, pplx-space-spaced-guest-server, pplx-space-sbx-mgr, pplx-space-spaced-host-client, pplx-space-spaced-host-server, pplx-space-spaced-host-cli, pplx-space-node-api-sim, pplx-space-common-core and pplx-space-public-api passes.
• cargo nextest run --no-fail-fast --config 'profile.test.package.pplx-space-api.opt-level=0' --all-features on those crates except the two guest crates, with Postgres and Redis in Docker: 4236 passed, 1 failed, 2 skipped. The failure (spaced-host-cli read_run_and_connect_preserve_bytes_without_waiting_for_process_exit) gets PermissionDenied spawning a guest child as an unprivileged user; it passes run as root.
• The guest crates spawn child processes that need root on the validation host, so their test binaries were run as root: pplx-space-spaced-guest-execution 168 passed; pplx-space-spaced-guest-server it 52 passed, 1 failed. The failure, meta::bridge_guest_snapshot_restore_discards_recorded_diagnostics, reads process-wide diagnostics and passes on its own; it does not touch process streams.
• shim/proto was regenerated with the pinned protoc 35.0, protoc-gen-go v1.36.11 and protoc-gen-go-grpc v1.6.1; go build ./... and go vet ./proto/... pass, and a second go generate leaves no diff.

Tests:

• Integration tests in native/tests.rs, against a fake guest journal that asserts every connect asks for the oldest retained output:
  • reattach after an early drop; Last-Event-ID resume; lossless backend reconnect after a resume;
  • eviction where the guest reports that the stream starts at its oldest retained output: one gap, no reopen, and an exact resume from the gap's id;
  • eviction through the lag status, reopened at the reported first sequence;
  • evicted_output_that_falls_behind_again_follows_the_live_tail: a fake whose oldest retained output moves past every cursor it is connected at, with and without the reported start. The stream gets tail output and no error. It fails at the previous head;
  • eviction on an older guest (live tail, then cursor_replay_unavailable on resume);
  • an untagged RESOURCE_EXHAUSTED that is not a gap;
  • cursor-ahead followed by an evicted start, reported once, for both guest behaviors;
  • the reopen cap: a start past the output too, a guest that keeps moving the start past the output, and a live tail that keeps falling behind end with an error; output between misses resets it;
  • a window-miss reopen refused as a stale route, recovered through refresh;
  • a backend failure before the first start, still sending one process_started;
  • a Last-Event-ID past the output, an untagged one, a foreign one on the starting attach, AlreadyExists recovery, and the 400 with no backend opened;
  • 300 resumes, and 300 backend reconnects, against an idle process: the next output still carries the position (idle_resumes_keep_positional_ids, idle_backend_reconnects_keep_positional_ids).
  • Every test that reads a process stream to its close also checks that only output and resync_required carry ids.
• Unit tests in native/sse/tests.rs, next to the now-private layout: the replay start at the layout edges, every following id staying above Last-Event-ID, ids increasing through frame saturation and lost positions, mangled ids rejected, every id printing back unchanged through a double, and positions up to 2^33 - 1 and none after, with the docs' example id decoding to position 1. A const check keeps the layout below 2^53.
• Guest: lagged_cursor_from_oldest_retained_starts_there_and_holds_it (the start is the oldest retained output, and later output waits for the reader instead of evicting it), cursor_past_the_output_fails_instead_of_waiting, lagged_cursor_reports_the_oldest_retained_output, and bridge_guest_process_connect_replays_from_output_cursor now checks the reported start. Sandbox-manager: process_event_to_spaced_keeps_the_connected_output_cursor. Spaced host client: the connect request keeps the flag. Simulator: connect_replays_the_journal_from_the_cursor_only expects the cursor-ahead failure and the reported start.

Mutation checks, one at a time, each restored and verified. All 15 Rust mutations made for this round failed at least one test, and the REST smoke's new check failed its own mutation on Kind: the guest ignoring the flag; the guest not claiming the primary place at the start it chose; the guest, sandbox-manager or simulator dropping the reported start; the host client dropping the flag; the data plane not setting it; the reported start ignored; the reopen at the oldest not marked (the chase then never ends); output not clearing that mark; an error instead of the tail after a reopen at the oldest falls behind again (both a broad and a narrow version); the tail and cursor-ahead reopens uncapped; and 8 frame bits. Earlier rounds' 18 mutations are unaffected by this round's changes. Sandbox-manager forwarding the flag has no unit test (neither does its existing cursor); the Kind run below covers it: without the flag, a busy reattach would reopen and follow the live tail, and it did neither.

End-to-end on a Kind cluster, recreated with infra/space/kind/scripts/bootstrap.sh for each column, Firecracker, every SPACE image and seed:latest built from that checkout: 25561e4 (this pull request's previous head) and 329fcb8 (this head). The data-plane and worker images are certified clean at 329fcb8. The seed publish image is labelled 329fcb8948a4-dirty-… only because I regenerated shim/proto, byte-identical, while it was building. Its guest comes from the certified Rust artifact image.

Reattach to a process that keeps writing after it has passed the window. The command runs i=0; while [ $i -lt 100000000 ]; do i=$((i+1)); echo "line $i"; done. The client reads to line 100, drops the stream for 2 s, then reattaches and reads for 10 s. Three attempts each with and without Last-Event-ID:

	previous head (25561e4)	this head
frames after attached	process_started, cursor_out_of_window, error (retryable, "process output stream fell behind retained output buffer") in 6 of 6	process_started, cursor_out_of_window, then stdout from the first frame, in 6 of 6
output in 10 s	none	690,337 to 1,187,199 lines, contiguous, no error
ids	—	positional to the end, above Last-Event-ID
data-plane log	3 gap continuations (reopens) per attempt	1 per attempt (from: Some(0) to the reported start), no reopen

On main the same reattach follows the live tail without a resync. That last row is from reading main's code, not from a run.

Also at this head, rerun after the id layout change. In every row there are no error frames, one process_started, ids only on output and resync_required, and ids that increase, are all above Last-Event-ID, and are below 2^53. Each command printed 1000 numbered lines over about 10 s unless noted; the client dropped after 100 lines and reattached 2 s later:

Case	Result
(a) reattach without Last-Event-ID	all 1000 lines, byte-exact, no resync
(b) reattach with Last-Event-ID	first + reattach byte-exact, no duplicates, no resync
(c) as (b), then delete the sandbox-manager pods mid-stream	one backend_reconnecting/backend_reconnected, no resync, byte-exact
A real id after a JavaScript Number() round trip (4503599640477696, unchanged, position 50)	exact resume: byte-exact, no resync
About 8.6 MB written while disconnected (200,000 lines), fresh attach	one cursor_out_of_window, then lines 164740..200000 contiguous to END; last id positional
Resume from that cursor_out_of_window frame's id	no resync, the same retained output again
Forged Last-Event-ID far past a running process's output	cursor_out_of_window and all 6 lines from the start
Untagged Last-Event-ID: 1500000	cursor_replay_unavailable, retained output replayed
A bit-62 id	400 last-event-id is past every event id this stream can send
300 resumes (each dropped after process_started) against a process idle between two outputs, in 0.88 s	none of those frames has an id; the next output carries position 2, frame 0

Typed smokes: scripts/tests/smoke-kind-typed.sh native with SPACE_SMOKE_KIND_ADMIN_TOKEN_FILE set, 26 passed, 0 failed. That includes native.m1_generated_sdk_conformance, native_recovery.command_survives_data_plane_restart, native_recovery.attached_stream_drains_through_data_plane_restart, native_concurrency.slow_reader, native_concurrency.independent_commands, native_contract.permanent_command_error and native_contract.token_scope. scripts/tests/smoke-spaced-rest.sh passes, including its Last-Event-ID resume stage and the new check of which frames carry ids; that check fails when stdout ids are treated as unexpected.

Not validated:

• WebSocket process sessions (unchanged).
• A guest from a template built before this change (cursor past the output, untagged lag status, no reported start).
• An old data plane, or an old node, running against the new guest.

Severity and risk

Severity: medium. A client that reattached to a process after a disconnect silently lost the output it missed. When main did send resync_required, the output was gone anyway. A backend reconnect after a reattach could silently duplicate output.

Risk:

• Id format. Process SSE ids change format (for example 4503599627632640 instead of 1, 2, 3), and only output and resync_required frames carry one. Clients that echo the last id received, as EventSource does, or parse it into a JavaScript Number, are unaffected. Clients that expected consecutive ids, or an id on every frame, must accept increasing ones on some frames.
• Repeated output. A reattach without Last-Event-ID, or from an id without a position, replays retained output from the start, which can repeat output the client already has. The second case is flagged by resync_required.
• Lost positions. After a stream has replayed from the start for an id without a position, or followed the live tail, its later ids carry no position, so each later resume replays behind resync_required.
• Accepted limitation: 2^33 - 1 output chunks. Ids carry a position only for a process's first 8,589,934,591 output chunks (about 99 days at 1,000 separate writes a second). After that every resume replays the retained output behind resync_required. Documented in the OpenAPI text, first-sandbox.md and troubleshooting.md.
• Ids are not bound to a session. A position id resumes any process stream it is sent to; the docs say an id is only valid for the session whose stream sent it.
• Errors instead of a jump. An unclassified mid-stream RESOURCE_EXHAUSTED now ends the stream with a retryable error (the client reattaches with Last-Event-ID) instead of reporting a gap and skipping to the tail.
• Backpressure. A reattach that starts at the oldest retained output becomes the guest's primary reader there, so a slow reattached client throttles a process with a full buffer until it catches up, as an attached reader already does.
• Rollback. Reverting this pull request restores main's behavior. Ids issued by this version are then treated by the older data plane as plain counters, and get its usual live-tail resume behind resync_required.

Rollout

Old and new data planes together. Nothing resumes at a wrong position, with one exception:

• A new plane never reads an old plane's id, or an id an old plane counted on from, as a position. It replays retained output behind resync_required(cursor_replay_unavailable), numbered above that id.
• The exception: an old plane that counts exactly k × 65,536 frames on from a new id produces an id that decodes as a position. Resuming from it duplicates output silently, or skips some if the count carries into the field.
• An old plane treats a new id as a counter, as it does today.

New guest with an old data plane. An old data plane does not decode ids or set resume_from_oldest_retained. It sends no cursor on reattach, and on a backend reconnect a cursor counted from its own attach, which trails the real position, so it should not send a cursor past the output; this pairing was not run. The tagged lag status keeps the old code and message, so an old data plane treats it as before.

Old node components. Sandbox-manager, the spaced host and node-api each decode and re-encode the connect request and the process events, so a component older than this change drops the flag and the reported start together. The guest then ends a lagged stream with the tagged status, and the data plane reopens at the reported first sequence. Against a busy process that reopen can fall behind again, and the stream then follows the live tail.

Old guest with a new data plane. The guest change reaches a sandbox only through a template built with this version.

• Evicted output: the older guest's untagged lag status is recognized by its message, and the stream follows the live tail.
• A Last-Event-ID past the output: an older guest waits until the process writes past that cursor, skipping the output before it, or until it exits. Only a forged or foreign id gets there.

Not covered

No issue has been opened for these yet.

• Two-phase rollout. Shipping id decoding first and turning on the new ids once every replica decodes them would remove the k × 65,536 exception above.
• Older-guest release check. Run the old-guest check from docs/versioning.md against a retained 0.1.x template before relying on exact positions there. Older guests replay a retried Start from the output tail (ecw_process/stream.rs), so on the native lost-binding path positions would be counted from 0 on a stream that begins at the tail, and a later positional resume would duplicate output silently.
• Session-bound ids. Ids carry no session; deriving the guard bits from the session id would make a foreign id fail to decode instead of relying on the documented rule.
• Backpressure docs. The public docs do not yet say that a reattached client can throttle a process with a full buffer.
• Smoke coverage. Neither smoke checks that output written while disconnected is replayed: M1 prints its marker before the disconnect and the REST smoke resumes after exit. A command that keeps printing across the disconnect, like the Kind case above, would cover it.
• A retried Start on the new guest still replays from cursor 0 without the flag, so when that output was evicted it goes through the lag status and one reopen.
• The native WebSocket process bridge reconnects without a cursor and always sends resync_required(process_stream_reconnected).
• cursor_out_of_window still covers eviction, cursor-ahead and a foreign id on the starting attach; only the logs tell them apart.

AI assistance: Claude Code drafted the code, tests, and this description; the author reviewed them.

[ac-pplx]

Thanks. This is a solid change. I traced the id encoding through every load, first_after, next and restart path. Ids stay strictly increasing and above Last-Event-ID, data-plane positions match the guest journal's sequences, and space-error-reason survives the node hop. I found no blockers. There are three small fixes and two design questions I'd like answered before merge. The rest can be follow-ups.

Before merge

1. A backend failure before the first start event drops the response's only process_started.

   • Code: native/sse.rs:456 presets process_id for a bound process. The reconnect arm sets reopened = true (:782), so the reopened stream's first Connected is filtered (:610, :643-647).
   • Scenario: a bound attach whose first backend stream fails with Unavailable, or a refreshable route rejection, after open but before its Connected message. The client gets attached, backend_reconnecting, backend_reconnected and output, but never process_started. On main the start was repeated, not lost. Clients that wait for it, such as space_smoke/m1_conformance.py:701-703, never run their process control.
   • Fix: record "this response already sent process_started" when the frame is pushed, and drop only later ones. That also covers the window-miss and cursor-ahead reopens. A test that fails the stream before its first Connected would pin it.

2. Some public text is now wrong or stale.

   • crates/libs/public/api/src/lib.rs:1879-1880: the DataPlaneAttachInfo.events_url doc, which is published schema text, still says streams "do not replay missed output".
   • docs/space/diagrams/data-plane-sigterm-drain.mmd:19 and its .svg still say "missed output not replayed". That contradicts the rewritten docs/architecture.md:243.
   • crates/libs/api/src/routes/public_docs.rs:68 says an attach without Last-Event-ID "starts from the oldest retained output". It actually starts at position 0 (sse.rs:408-413). Once position 0 is evicted, the guest answers lagged and the stream sends cursor_out_of_window, then the live tail (sse.rs:688-699). For a completed process that means no stdout at all, even with output still retained. Suggested wording: "starts from the beginning of the output; if that is no longer retained, cursor_out_of_window and then live output".
   • The same text says ids "exceed the integers a double represents exactly". Every id's set bits lie in bits 16 to 62, so a double holds each one exactly; src/tests.rs:3654-3656 asserts this. What breaks is decimal printing above 2^53. Suggested wording: "exceed 2^53, so a JavaScript Number may print them back differently".

3. A test and some comments still describe the dropped contracts.

   • src/tests.rs:3662-3687 still asserts the last + 1 resume for "process", and native/auth.rs:97-101 still justifies MAX_LAST_EVENT_ID by it. Both are file-watch only now.
   • src/tests.rs:3534-3541 talks about "decimal boundaries" (999_999, 1_000_000) from the first commit's decimal encoding, which this PR no longer uses. Probing the real layout edges would be better, for example PROCESS_EVENT_ID_TAG - 1 and the last id below the first no-position block.
   • sse.rs:277-284 says a +1-counted id never decodes, without the k × 65,536 exception that the Rollout section names.

Questions before merge

Neither is a correctness problem. I'd still like an answer before merge, because the SSE id layout and the guest change are both costly to revise once they ship.

4. Should process ids fit under 2^53?

   • The Last-Event-ID schema is still integer/int64 (schema_contract.rs:117-126), while the text calls ids opaque strings.
   • Generators and JS clients that map int64 to number print a different decimal. Every resume then replays the retained output behind cursor_replay_unavailable, and later ids stay in no-position blocks (sse.rs:384-394), so such a client never resumes exactly. The generated-SDK smoke uses Python, so CI cannot see this.
   • Earlier data planes issue small counters, so a tag at bit 52 would separate them just as well. That leaves 16 guard bits and 36 bits for frame plus field, and the +1 guard keeps working.
   • If you prefer the current layout, please list this as an accepted limitation in the PR and the docs.

5. Could the guest report its oldest retained sequence in this same change?

   • After the first eviction the stream sets its cursor to None (sse.rs:694-699), and Unknown ids never return to positions. A fresh attach to a process that has written more than 4 MiB lands in the same state, because guest cursor(Some(0)) with first_sequence > 0 returns lagged (process.rs:1228).
   • On long, chatty processes, where reattach matters most, each later short disconnect then loses output the guest still retains. It is flagged by a resync and is no worse than main.
   • Adding first_sequence as metadata on the lag status would let the data plane reopen there and keep positional ids. This guest change already needs a template rollout, so doing it later costs another one. A follow-up is fine if you'd rather keep the scope.

Follow-ups

• The window-miss reopen bypasses route refresh.
  • Code: sse.rs:700-716 reconnects with state.client and the stream's existing route token, which has an expires_at (state.rs:97-101).
  • Scenario: a long-lived stream whose token has expired falls behind the journal. The reopen is refused as a refreshable rejection, and the stream ends with a retryable error instead of continuing.
  • Fix: send a refreshable failure through the existing refresh path at :717, and add that case to the eviction test.
• Eviction is detected by a bare status code.
  • is_output_window_miss (sse.rs:812-820) treats any mid-stream RESOURCE_EXHAUSTED other than the limiter's as eviction.
  • That also matches spaced-host's oversized-chunk refusal (crates/libs/spaced/host/server/src/process.rs:28-36), the guest's "process replay state was evicted", and an h2 ENHANCE_YOUR_CALM reset. Each is reported as cursor_out_of_window plus a jump to the tail.
  • There is also no cap on reopens that make no progress.
  • Fix: tag the guest's lag status with a reason, as process-output-cursor-ahead is tagged, and keep the bare match only for older guests. Cap reopens without progress, as the ECW stream does.
• The mixed-version exception.
  • An old replica that counts exactly k × 65,536 frames on from a new id produces an id that decodes as a position (sse.rs:343-345). Resuming from it duplicates output silently, or skips some if the count carries into the field.
  • A two-phase rollout would remove the case entirely: ship decoding first, then turn on the new ids once every replica decodes them.
• Older guests.
  • With an older guest, a positional cursor past the output waits instead of failing, so output is skipped without a resync. As the PR says, only a forged or foreign id gets there.
  • Separately, ecw_process/stream.rs:232-234 notes that older guests replay a retried Start from the output tail. On the native lost-binding path (streams.rs:23-50), positions would then be counted from 0 on a stream that begins at the tail, and a later positional resume would duplicate output silently.
  • Fix: run the old-guest release check from docs/versioning.md against a retained 0.1.x template before relying on exact positions there.
• Ids carry no session. load (sse.rs:415-423) accepts any decodable position, so a client that sends another session's id to a bound process resumes silently at that position. Either document that an id is only valid on the stream that issued it, or derive the guard bits from the session id.
• Operability.
  • cursor_out_of_window now covers eviction, cursor-ahead, a foreign id on the starting attach, and misclassified transport errors, and nothing tells them apart. The window-miss branch logs nothing.
  • The new 400 records no attach metric (sse.rs:98-107).
  • load now runs before the 409 check, so replay_miss counts frames that are never sent (native.rs:756-761).
• Backpressure. A reattach now connects at position 0, so it takes the guest's primary-reader slot there (process.rs:1056-1066). A slow reattached client can pause a process with a full buffer until the replay drains. Worth a line in the docs.
• Simulator and proto drift.
  • crates/libs/node/api/sim/src/processes.rs:10-13 says it follows the guest journal contract, but :250-253 clamps a cursor past the output to the tail.
  • proto/spaced/operations/v1/operations.proto:227-231 does not document the new OUT_OF_RANGE or the lag status.
• Smoke coverage. Neither smoke checks that output written while disconnected is replayed. M1 prints its marker before the disconnect (m1_conformance.py:774), and the REST smoke resumes after exit. A command that keeps printing across the disconnect would cover it.

Nits

• Cursor-ahead followed by an evicted start sends cursor_out_of_window twice: once at sse.rs:487-494, then again at :693-699.
• ProcessEventId::encode does not check its fields, and next from the last block wraps backwards (sse.rs:332-340, :390-393). Neither is reachable today. A debug_assert! and a const layout check would make the invariant local. The layout constants are pub(crate) only for tests, and PROCESS_EVENT_ID_MAX_RESUME_BLOCK could be derived from the field width.
• is_output_cursor_ahead's doc (sse.rs:822) talks about older guests. It should first say what the function matches.
• JournalFake (native/tests.rs:721-950) repeats most of BindingFake (:418-521).
• This changes documented native-API behavior, so it would help to say which release line it targets.

[NancyKwongPerplexity]

thanks for the careful trace. all addressed in b55f2d9..25561e4:

1. process_started: now tracked per response (started_sent), only later ones are dropped. covers the window-miss/cursor-ahead reopens too. test fails the first stream before Connected
2. public text: events_url doc, drain diagram (mmd/svg/png) and public_docs wording fixed as suggested. the double wording is moot now, see 4
3. the last+1 test is file-watch only, MAX_LAST_EVENT_ID doc fixed, tests probe the real layout edges, and the id doc names the k×65,536 case
4. yes: tag at bit 52, 16 guard / 8 frame / 28 position bits, so all ids are < 2^53. a JS Number round trip now resumes exactly on kind. ≥2^53 is a 400
5. yes, in this change: the guest tags the lag status process-output-lagged + space-process-output-first-sequence, and the data plane reopens there and keeps positional ids. a fresh attach past the window now gets the retained output

• frame budget: only stdout/stderr/resync_required carry ids now, so idle resumes/reconnects spend none (300 of each in tests and on kind). 2^27 output chunks is documented as an accepted limit
• route refresh on the window-miss reopen: done, it goes through the shared refresh path
• eviction by bare code: only a tagged status, or the old guest's message, counts. reopens are capped at 3 without output. proto + sim updated
• operability: gaps log their cause, the 400 is counted in attach, replay_miss only counts once the stream opens
• nits: single gap report, debug_asserts + const layout check, constants private, resume block derived, classifier doc, fakes merged
• two-phase rollout, old-guest release check, session-bound ids, backpressure doc and smoke coverage are listed under not covered. the docs now say an id is only valid for the session that sent it

[ac-pplx]

Thanks for the thorough round. I re-checked every round-1 item at 25561e4. Everything is either fixed or deferred in the PR body the way we discussed. The bit-52 layout, started_sent, gap_reported, the tagged lag status and the route-refresh reopen all hold up, and ids stay increasing and above Last-Event-ID on every path I traced.

One new problem comes from the reopen-at-oldest-retained path, and I'd like it fixed before merge. The rest are small.

Round-1 items

Item	Status
1. process_started lost when the backend fails before the first start	Fixed (native/sse.rs:559-566; test native/tests.rs:1444)
2. Wrong or stale public text	Fixed. One other bullet went stale this round; see below
3. Tests and comments describing dropped contracts	Fixed
4. Ids under 2^53	Fixed (bit 52, const check, 400 at or above 2^53; the example ids decode correctly)
5. Guest reports its oldest retained sequence	Fixed in the guest. The data plane's use of it has the issue below
Route refresh on window-miss reopen	Fixed (native/sse.rs:638-641)
Bare RESOURCE_EXHAUSTED match; reopen cap	Fixed (native/sse.rs:925-952, :604-607)
Ids carry no session	Fixed through the documented option
Operability	Fixed. The single wire reason is deferred
Simulator and proto	Fixed
Nits: double gap report, classifier doc, merged fakes, release line	Fixed
Nit: ProcessEventId invariants	Mostly fixed. See the last nit below
Two-phase rollout, older-guest check, backpressure docs, disconnect smoke coverage	Deferred in the PR body. Fine with me

Before merge

1. Reattaching to a busy process can fail on every attempt.

   • How it fails.
     • The guest reports first_sequence = F under the journal lock and ends the stream. The data plane reconnects at F one data-plane-to-guest round trip later (native/sse.rs:617-627).
     • A reader that lagged never takes the primary-reader slot, which requires cursor >= first_sequence (process.rs:1054-1059). With no primary, the journal evicts its oldest output on every append once it is full (process.rs:1143-1148, :1276-1281).
     • So a write during that round trip evicts F, and the reopen lags again. After three such reopens the stream ends with a retryable error (native/sse.rs:604-607). The retry from the gap id Position{F,0} repeats the same chase.
   • Scenario. A process has written more than the retention window and keeps writing faster than about once per round trip, for example a verbose build or a log follower. A fresh attach, or a resume after a long disconnect, gets process_started, cursor_out_of_window and then error. It gets no output until the process slows down or exits. On main, and at the previous head, this case went to the live tail, became the primary reader and streamed.
   • Why it isn't caught. window_miss_reopens_without_progress_are_capped (native/tests.rs:1234-1262) is exactly this guest behaviour, and it expects the error. The 8.6 MB Kind case reattached after the writer had finished, so it never hit the race.
   • Suggested fix.
     • Cheapest: when an eviction reopen at the reported first lags again before any output, or when the cap is reached on an eviction, continue at the live tail. continue_after_gap(None) is enough. A tail cursor becomes primary and cannot lag. Keep the error for a cursor-ahead loop.
     • More complete: this guest change already needs a template rollout, so the guest could start a lagged connect at its oldest retained output under the journal lock and report where it started. That leaves no round trip between reading first_sequence and claiming it.
     • Either way, the cap test should expect tail output for evictions. A fake whose first advances on every connect would pin it.

2. New tests assert on metrics. native/tests.rs:1688-1699 asserts the data_plane.attach result:invalid_last_event_id tag, and refused_attach_counts_no_replay_miss (:1702-1747) only asserts that a metric is absent. The repository's test rules (.agents/plugins/pplx-space/skills/space-code-quality/SKILL.md, "Tests") rule out unit tests that assert on metrics or metric tags. Keep the 400 status and no-backend-open checks, drop the metric assertions, and drop the metric-only 409 test. I asked for the metrics in round 1, not for tests of them; sorry for the confusion.

3. One published bullet went stale this round. public_docs.rs:68 lists "one sent after cursor_out_of_window" among ids that do not resume exactly. After an eviction gap, ids now keep their position, and native/tests.rs:1108-1121 resumes from the gap id with no resync. That also contradicts the bullet before it. Only a cursor-ahead or foreign-id restart, or the live tail on an older template, loses the position. The process_frame_has_event_id doc (native/sse.rs:886-891) makes the same claim, and the older-guest test at native/tests.rs:1155-1158 shows otherwise.

Worth deciding before merge

4. The frame counter barely gets used now.
   • Only output and resync_required carry ids, and every gap either moves the position or drops it. So a positional id's frame is only ever 0, or 1 for load's Position{0,1}.
   • The 8 frame bits still cost the field 7 bits. The published limit is 2^27 - 1 output chunks, about 3.7 hours at 10,000 writes a second.
   • The no-position capacity is 2^35 ids whatever the split. Bit 52 is already the highest tag below 2^53, so changing the split after release would need a new tag scheme.
   • Two frame bits would give 2^33 - 1 positions with no other change in behaviour. Your call; it's cheap now and expensive later.

Nits

• scripts/tests/smoke-spaced-rest.sh checks only that ids increase, not which frames carry them (:639-643). ids_continue now starts out true (:690), and the resumed stream of an exited process sends no ids, so that assertion cannot fail. The PR body says both smokes check which frames carry ids; only M1 does (m1_conformance.py:688-691). The no-resync and no-duplicate checks still catch regressions.
• The const check says a stream "cannot run out" (native/sse.rs:329), and next says "Always greater". Unknown{2^27-1, 255}.next() still wraps backwards in a release build. It is unreachable in practice, but the comments should state the finite headroom.
• PR body and reply corrections:
  • The reply says "300 of each on kind", but the Kind table only shows the 300-resume row.
  • "As before" for older guests means as at the previous head; on main, a mid-stream lag ended in an error.
  • "Every process stream test also checks that only output and resync_required carry ids" holds only for tests that read the stream to its close.

[ac-pplx]

The problem

A native process session streams a command's output over SSE. On main, a client that reconnects loses what the process wrote while it was away. Without Last-Event-ID the stream silently jumps to the live tail. With it, the stream sends resync_required and then jumps to the tail. A node-side restart mid-stream replays stale output. The guest already keeps a bounded journal of each process's recent output, but the data plane never resumed from it, and its SSE ids carried no position.

Example

A command prints 1,000 numbered lines over 10 s; the client drops after line 100 and reconnects 2 s later.

main:      no Last-Event-ID    -> 731 lines, 1..269 missing, no warning
           with Last-Event-ID  -> resync_required, then the tail; 101..269 missing
this PR:   Last-Event-ID = 4503599653584896 (output position 100)
           -> lines 101..1000 exactly, no resync, ids keep increasing

Verdict

Approving. Everything from round 2 is fixed or deferred as agreed.

The busy-process reattach is fixed the right way. The guest picks the oldest retained output under the same journal lock that claims the primary-reader slot (process.rs:1057-1078), and reports that start in ProcessConnected.output_cursor (:2280). The data plane treats a start past its position as the gap (native/sse.rs:608-616). A stream that lags again before any output falls back to the live tail (:640-650).

Each hop compiles one operations.proto, so an old hop drops the flag and the reply field together, and the guest never moves the start without the data plane learning it.

The remaining items below don't block. Please do the PR body before enqueueing.

Round-2 items

Item	Status
1. Busy-process reattach ends in error	Fixed
2. Tests asserting on metrics	Fixed
3. Stale OpenAPI bullet and process_frame_has_event_id doc	Fixed (one related omission below)
4. Frame-counter size	Fixed: 2 bits, positions for 2^33 - 1 outputs
REST smoke id checks	Fixed
Comments claiming ids cannot run out	Mostly fixed; first_after still says "a stream never runs out of blocks" (native/sse.rs:350-354)
PR body and reply corrections	Not done: the body is unchanged since round 2 (below)
Two-phase rollout, older-guest check, session-bound ids, backpressure docs, disconnect smoke coverage	Still deferred; fine

Before merge

1. Update the PR body. It still describes the round-2 design:

   • an 8-bit frame counter, a 28-bit field and the 2^27 - 1 limit;
   • evictions ending in error after 3 reopens;
   • the removed metric tests;
   • "No schema ... changes", though operations.proto gains ConnectProcessRequest.resume_from_oldest_retained and ProcessConnected.output_cursor.

   It doesn't describe the flag or its mixed-version behavior, and its validation is for 25561e4. Since the PR merges by squash, the body likely ends up as the commit message.

Follow-ups

2. The live-tail fallback can happen on current guests too.
   • public_docs.rs:68 says eviction is followed by the oldest retained output, "with live output on sandboxes from older templates".
   • A clamped reader is primary only if no other reader holds the slot and backpressure is enabled (process.rs:1067-1072). Example: an earlier backend stream for the same process hasn't closed yet. Without the slot the reader can lag again, and the stream then follows the live tail (native/sse.rs:648).
   • Suggest "with live output when output keeps being evicted before the stream can read it, including on older templates".
   • The proto comment (operations.proto:245-246), the guest cursor() doc and the matching test doc say output written meanwhile "cannot" evict the start. Qualify them with "while the stream holds the primary reader's place".
   • The same bullet could also say that ids sent after a cursor_replay_unavailable replay don't resume exactly.
3. Nothing tests the guest's reported start.
   • lagged_cursor_from_oldest_retained_starts_there_and_holds_it (process/tests.rs:1009) checks the journal clamp and backpressure, but nothing asserts connect_process's output_cursor (process.rs:2280).
   • If that reported the requested cursor instead, the data plane would skip output without cursor_out_of_window and number later ids wrong, and every test would still pass.
   • One assertion in spaced/guest/server/tests/it/process.rs (a connect from an evicted cursor reports the clamped start) would pin it.
4. The cap reset is untested on the live-tail path.
   • The eviction arm with a reported first no longer checks the cap. So window_misses_with_output_between_are_not_capped (native/tests.rs:1407-1436) still passes if the reset at native/sse.rs:893 is deleted.
   • The live-tail arm does check the cap. A variant in which the fake lags without reporting a first sequence, with output between misses, would pin that output resets it there.

[NancyKwongPerplexity]

thanks for the second pass. all addressed in 43971f6..329fcb8:

1. busy reattach: did the fuller fix. ConnectProcessRequest gets resume_from_oldest_retained: with it, the guest starts a lagged cursor at its oldest retained output under the same journal lock that makes it the primary reader, and reports that start in ProcessConnected.output_cursor. so there's no round trip to race eviction. the data plane treats a reported start past the client position as the gap (one cursor_out_of_window, then output, no reopen). the old lag-status reopen stays for older guests, and a stream already at the oldest output that lags again before output goes to the live tail. cap test is split, with a fake whose first moves past every connect. on kind, reattaching to a process that keeps writing errored 6/6 at 25561e4 and streams 6/6 with no reopen at 329fcb8. 43971f6
2. metric tests: dropped both assertions and the 409-only test. sorry for the noise there. 61dde36
3. stale bullet: fixed the openapi bullet and the process_frame_has_event_id doc. c2663b1
4. frame bits: went to 2, so positions now go to 2^33−1. docs, example id and an edge test updated. 7cd7889

nits: the rest smoke now checks which frames carry ids and the dead ids_continue is gone (329fcb8). comments state the 2^34 headroom and the wrap. pr body corrected (the 300 row is resumes only, "as before" means the previous head, the id-shape check applies only to tests that read to close)

[cla-assistant]

All committers have signed the CLA.

[ac-pplx]

The problem

A native process session streams a command's output over SSE. On main, a client that reconnects loses what the process wrote while it was away: without Last-Event-ID the stream silently jumps to the live tail, and with it the stream sends resync_required and then jumps to the tail. The guest already keeps a bounded journal of each process's recent output, but the data plane never resumed from it, and its SSE ids carried no position.

Example

A command prints 1,000 numbered lines over 10 s; the client drops after line 100 and reconnects 2 s later.

main:      no Last-Event-ID    -> 731 lines, 1..269 missing, no warning
           with Last-Event-ID  -> resync_required, then the tail; 101..269 missing
this PR:   Last-Event-ID = 4503599653584896 (output position 100)
           -> lines 101..1000 exactly, no resync, ids keep increasing

Verdict

Approving. The rebase onto 9b1985d leaves the PR's own patch unchanged; only diff alignment and context differ. It also combines cleanly with what main changed in the same files:

• #795: every process-stream error path, including the new ones, sends the fixed PROCESS_STREAM_FAILED message, and the revocation rule stays in session_is_revoked.
• #702: its untagged OUT_OF_RANGE size refusal can't be taken for the tagged cursor-ahead status.
• #801 and #811: they don't reach the events route.

The items below don't block. license/cla shows pending on the rebased head.

Round-3 items

Item	Status
Update the PR body	Done for the design; a few CONTRIBUTING.md rule issues remain (below)
Live-tail fallback on current guests in the docs	Not yet: public_docs.rs:74 and operations.proto:245-246 are unchanged
Test of the guest's reported start	Partly. My round-3 wording was too broad: spaced/guest/server/tests/it/process.rs:258-263 does assert the start, but only for an unclamped request at 0. A connect from an evicted cursor is still untested
Cap reset on the live-tail path	Not yet
first_after "never runs out of blocks" (native/sse.rs:350-354)	Not yet
Two-phase rollout, older-guest check, session-bound ids, backpressure docs, disconnect smoke coverage	Still deferred; fine

Before merge (cheap)

1. docs/api/index.md:31 still says "Streams are live, with no replay." docs/api/build.mjs renders that page as the overview of the published API reference, right above the new OpenAPI text, so the two now contradict each other. The sentence predates this PR; I missed it in earlier rounds. Suggested wording: "Save output as it arrives. The sandbox retains a bounded amount of each process's output; reconnect with Last-Event-ID to resume after the last event, and check for resync_required."
2. The PR body breaks the "Submit a pull request" rules in CONTRIBUTING.md (merged on main since the previous head):
   • It describes how the PR got here: "previous head", "this round", "It fails at the previous head". Say what merging does, and leave review history to comments.
   • It cites branch-only commits (25561e4, 329fcb8) that won't be on main after the squash, and its validation is headed "At this head (329fcb8)" while the head is 6e68697.
   • It names infra/space/kind/scripts/bootstrap.sh, which main removed in #735.
   • "No issue has been opened for these yet": open an issue for each follow-up and link it.
   • Smaller points:
     • "nothing written afterwards can evict it" (Summary) holds only while the stream holds the primary reader's place, as the next bullet says.
     • The Kind log row from: Some(0) applies only to the attempts without Last-Event-ID.
     • "On main the same reattach follows the live tail without a resync" holds only without Last-Event-ID.

Follow-ups (carried from round 3)

• Live-tail fallback wording.
  • public_docs.rs:74: say live output also follows when output keeps being evicted before the stream can read it, not only on older templates.
  • operations.proto:245-246, the guest cursor() doc and the lagged_cursor_from_oldest_retained_starts_there_and_holds_it doc: qualify "cannot evict" with "while the stream holds the primary reader's place".
• Tests.
  • A guest connect from an evicted cursor should report the clamped start, not the requested one; one case in spaced/guest/server/tests/it/process.rs would pin it.
  • A live-tail variant of window_misses_with_output_between_are_not_capped, where the fake lags without reporting a first sequence, would pin the reset at native/sse.rs:893.
• Cleanup. crates/libs/data/plane/server/Cargo.toml:67 keeps the pplx-space-metrics test-capture dev-dependency the removed metric tests needed. Nothing in the crate uses it now; main dropped the same line in #795.