feat(temporal): Made a session outlive the client that started it. - #8
Draft
moedash wants to merge 6 commits into
Draft
feat(temporal): Made a session outlive the client that started it.#8moedash wants to merge 6 commits into
moedash wants to merge 6 commits into
Conversation
A durable session outlives the process that started it, but nothing on the command line could start one that way, ask what is still running, or follow one from a machine that never had it. These are thin HTTP clients because any serve in the deployment can answer for any session.
Kills the serve that started a turn while a tool is still running, then asks a second serve that never saw the session to report it and replay it. The claim only shows up across processes, so it needs processes.
A fresh host has no tip note, so the check that protects somebody's working copy also refused to build a tree that was never there. The path being present is not the same as the project being present, and a mounted empty directory is the ordinary shape of a machine that has never seen this session.
Each worker is a container with its own filesystem, so a session that moves has to bring its worktree with it out of the shared store. That is the half a single host cannot exercise: there the tree is already on the disk the other process reads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes a coding session belong to the deployment instead of to whoever started it.
The durable executor already had every piece needed for this, which is the interesting part. A session is a workflow rather than a process, the running set comes from Temporal visibility, the store is shared, and a live tail re-reads so a subscriber sees work another process is doing. Put together, that means a session can outlive its client. Nothing said so, and nothing on the command line could use it.
So this is a client surface, not new machinery. It adds no dependency on Temporal: the commands are plain HTTP against any serve in the deployment, because the serves are interchangeable.
What is new
opencode session start "<prompt>"hands a prompt over and returns the session id. It holds no terminal, so a turn can be started by something that is not a person at a keyboard.opencode session runninglists what the deployment is executing right now. It reads the executor's own answer, so it survives a restart of whichever process happens to serve the call.opencode session watch <id>follows a session from a machine that has never seen it, and stops when the turn stops. It ends on the model's own finish reason. The running-session set cannot answer that question, because a session stays in it while its supervisor waits out the idle timeout with nothing left to do.All three take
--attach <url>(or$OPENCODE_SERVER). For an interactive terminal rather than a follower,opencode attach <url> --session <id>already existed.What is not here
A turn started from a schedule or a webhook still needs its own entry point;
session startis a command, so something has to run it. And the deployment is still a set of environment variables rather than a supported mode, so defaults, migration-on-deploy and credential distribution remain the operator's problem. Both are called out in the README.Issue for this PR
Issues are disabled on this fork, so there is no number to link. This continues the durable-executor work in #2 and #7.
Type of change
What does this PR do?
Adds
session start,session runningandsession watchunder the existingsessioncommand, so the detached lifecycle the durable executor already supports can be driven from a command line. Addspackages/temporal/scripts/detached-session-check.sh, which proves the claim against real processes. Documents both in the temporal README, along with the deployment shape and what is still missing.The commands are thin HTTP clients on purpose. In a durable deployment any serve reads the shared store and signals the same workflows, so a client needs an endpoint and a session id, never a particular host.
One fix fell out of writing the check: machine-readable output was going to stderr with everything else, so
--jsonwas not pipeable. It goes to stdout now.How did you verify your code works?
detached-session-check.shruns the whole story against real processes, with a Temporal dev server, one standalone worker, two serve processes and one shared store. Serve A starts a turn and is killed with a tool still running. The turn finishes on the worker. Serve B, which never saw the session, reports it running and replays the transcript including the work done while no client existed. Thensession startreturns without waiting,session runninglists it, andsession watchfollows it live and exits when the turn ends. All assertions pass from a clean run.The assertions were mutation-checked rather than trusted. Giving serve B its own
OPENCODE_DBmakes exactly the cross-process ones fail (activereturns{}, the replay is empty, the follower hangs) while the serve-A-and-worker ones still pass, which is what shows the shared store is load-bearing and that the check is testing it. Thewatchexit condition was found the same way: an earlier version asked the running-session set instead of reading the finish reason, and it hung, because a session stays running while its supervisor idles.Typecheck is clean for the changed files. Pre-existing errors in
packages/tui/src/component/dialog-move-session.tsxare untouched by this branch.Checklist
Across two machines (added after the first pass)
packages/temporal/scripts/cross-host-check.shruns the claim against containers: each worker has its own filesystem and hostname, and the store is a real libSQL server. A session writes a file on worker A, worker A's host is killed, and worker B, whose project volume is empty, continues the same session and reads that file back.That found a bug a single host cannot show, and it is the reason this PR now touches
packages/core.WorktreeMaterializer.ensuretreated any existing directory as somebody's working copy. A fresh host has no tip note, sobehindreturned false and the tree was never built. The tools then ran in an empty directory and the model was told a wrong answer, which is worse than a failure. Turn 1 wrote/project/note.txtand read it back; turn 2 of the same session on the other host gotcat: /project/note.txt: No such file or directory, with the packs sitting in the store the whole time.On one host the case never appears. Worker B either already has the project, or has no directory at all, and an absent directory materializes fine, which is exactly what the existing unit test covers. A mounted directory that exists and is empty is the ordinary shape of a machine that has never seen the session.
The fix is to treat an existing but empty directory as absent, which keeps the protection for a real checkout: an empty directory has no work to lose.
packages/core/test/worktree-materialize.test.tsgains that case so CI catches it rather than only the container check.Verified. With the fix, the container check passes end to end and the worker logs
materialized worktree from snapshot packs. Reverting the fix fails it with the original symptom. The unit test goes red on the revert and green with it.bun test test/worktree-materialize.test.tsis 4/4.Two things I got wrong first, both caught by mutation rather than by reading: the first version of the fix changed only the outer guard, and the re-check inside the lock still bailed on an empty directory, so nothing materialized. And the check's own completion poll used
history?limit=400, which the endpoint rejects at 100, so a rejected request looked exactly like a turn that never finished, and its transcript assertion matched turn 1's output for turn 2's result. Both are fixed; the poll now counts step endings rather than matching one, and the outcome is read off worker B's own disk rather than the shared transcript.Still open. A turn started from a schedule or a webhook needs an entry point of its own. The deployment is a set of environment variables rather than a supported mode. And this is one libSQL server, so it shows a shared store over a network rather than one that survives losing a node.