Spec: see, follow and Stop a machine's Commands from the CLI #711

Rewritten from the #711 grilling on 2026-10-18. The full record of decisions is .scratch/grill-with-docs/711-live-runs.md on branch web-ui. This Spec is the CLI half of #711, #368 and #574. The Console, a web page on each machine, is a separate Spec built on top of this one.

Problem Statement

A Command's state lives in its Command log, the Activity log and its Session logs, and you read them after the fact. Nothing on a machine says which Commands are running right now, which issue each is on, what stage each Run has reached, or where its logs are. Nothing stops one cleanly either.

#574 is the real case. On 2026-10-08, a cron Pickup run took chart35 #239 before it was ready. Finding it meant guessing the host, then reading process lists and logs by hand. Stopping it meant a hand-picked kill -TERM on the right parent process. The issue then had to be relabelled by hand, because otherwise the next minute's Pickup run would take it again.

Two machines now run thirdshift, thirdshift and stephens, and both pick up from some of the same repositories. Agents asked "what's running?" have no command to answer with either.

Solution

Each Command keeps a Command record on its machine. The record holds:

• its process;
• its issue, if it has one;
• each child Run, with the stages it has been through and when each began;
• once the Command ends, its outcome.

The record is kept beside the Command's Command log after it ends.

Four new commands read the records and the logs, and each has a --json that is the promised API:

• thirdshift status lists the machine's live Commands and, with --all, the whole history.
• thirdshift show <command> gives one Command in full.
• thirdshift logs <command> prints or follows its logs.
• thirdshift stop <command> is a Stop. It ends a live Command through the existing interruption path. The issue is left for the Day shift, labelled needs-triage and with ready-for-agent not put back, unless --requeue asks for it to go back in line. The Activity log records who stopped it.

User Stories

1. As the Day shift, I want one command that lists every live Command on this machine, so that I don't have to read process lists to know what the factory is doing.
2. As the Day shift, I want each live Command to show the issue it's working on, with its repository and title, so that I can tell at a glance whether it's one I meant to start.
3. As the Day shift, I want each Run to show its current stage and how long it has been in it, such as "CI watch · 52 min", so that a stalled Run stands out without opening a log.
4. As the Day shift, I want each Run to show its Harness, Model and Effort, so that I know which agent is working and at what effort.
5. As the Day shift, I want a Pickup run's Spec run and its Tickets' Runs nested under it, so that I can see the whole tree one Command started.
6. As the Day shift, I want a Spec run to show every Ticket in the Spec, whether running, merged into the Spec branch, failed, waiting on named blockers, or not started, so that I can see why a Spec run is slow.
7. As the Day shift, I want a Base fix shown under the Run that started it, so that the Run's wait is explained.
8. As the Day shift, I want a Command whose process has gone without an outcome shown as "died at ", so that a kill -9, a crash or a reboot is visible rather than looking like a live Command.
9. As the Day shift, I want a process id that's been reused by another process not to count as the Command still being alive, so that a dead Command is never shown as live.
10. As the Day shift, I want Commands that ended in the last day listed after the live ones, with their outcome, so that after a Run notification I can find the Run it was about.
11. As the Day shift, I want status --all to list every Command this machine has logs for, so that I can look back further than a day.
12. As the Day shift, I want Commands that ran before Command records existed listed from their Activity log and Command log lines, without stages, so that the history isn't cut off when this ships.
13. As the Day shift, I want Session logs from before the Activity logs began listed, grouped by issue and stamp, with no outcome, so that the oldest work on the machine still shows.
14. As the Day shift, I want each repository's idle Passes shown with their latest skip reason, such as "Ready issue make release.sh watch ci on default if running and start a release when it is done #765 goes first", so that I can see why Weeding or Fencing is waiting.
15. As the Day shift, I want to name a Command by its issue, as chart35#239, so that I don't need to look up a stamp or a process id.
16. As the Day shift, I want to name a Command by its full Issue URL, so that I can paste it from a Run notification or GitHub.
17. As the Day shift, I want the owner needed only when two repositories on the machine share a name, so that short names work in the usual case and a mismatch is never guessed.
18. As the Day shift, I want a Pass with no issue, such as an Architect run still in its Architecture review, named by its kind and Command log stamp, so that every Command has a name.
19. As the Day shift, I want an issue with several past Commands to name the live one, or else the latest, so that the short name does what I mean.
20. As the Day shift, I want show <command> to give one Command's whole state: its Runs, their stages with start times, its log paths and its latest event, so that I can see everything about it in one place.
21. As the Day shift, I want each live Run's latest event in one line, such as Bash: cargo test, so that I can tell what it's doing right now without reading its Session log.
22. As the Day shift, I want logs <command> to print its Command log, so that I can read what it printed without finding the file.
23. As the Day shift, I want logs <command> --follow to keep printing as the log grows, until the Command ends, so that I can watch a live Command over SSH.
24. As the Day shift, I want logs <command> --session <kind> to print its latest Session log of that kind, such as implement, spec-review or repair, so that I can read what the agent did.
25. As the Day shift, I want to name a Ticket's issue in logs and show, and get that Ticket's Run inside its Spec run, so that I can follow one Ticket.
26. As the Day shift, I want stop <command> to end a live Command cleanly, through the same cleanup as any other interruption, so that its work is pushed and nothing is left half-removed.
27. As the Day shift, I want a stopped issue labelled needs-triage, with ready-for-agent not put back, so that the next Pickup run doesn't take it again a minute later.
28. As the Day shift, I want stop --requeue to put the issue back in line instead, so that I can stop a Run I want restarted.
29. As the Day shift, I want stop to wait for the Command to end and then print its outcome, plus any Issue branch or pull request it left, so that I know it stopped and what remains.
30. As the Day shift, I want stop to say, while it waits, that the Command is still ending, so that a long cleanup doesn't look like a hang.
31. As the Day shift, I want stopping a Spec run to stop its Tickets' Runs with it, so that one Stop ends the whole tree.
32. As the Day shift, I want stopping a Pickup run to apply the Stop to the issue it took, so that make a way to easily see in progress thirdshift runs on machine and cut one off #574's case is one command.
33. As the Day shift, I want a Stop of an Architect run or a Security run with no issue to end it without touching any labels, so that Stop works on every Command.
34. As the Day shift, I want stop refused, with the reason, on a Command that has already ended or died, so that I never signal an unrelated process.
35. As the Day shift, I want stop on a Ticket's issue refused with the name of its Spec run, so that I know which Command to stop instead.
36. As the Day shift, I want the Activity log to record each Stop with who asked and from where, such as Pickup run #239 stopped from the CLI by jacob, so that the repository's history says why a Command ended.
37. As an agent with a shell on the machine, I want status --json and show --json in a documented, versioned shape, so that I can answer "what's running?" without scraping text.
38. As an agent, I want the JSON to use the glossary's terms (Command, Run, Spec run, Ticket, stage, outcome), so that what I report reads the same as the rest of thirdshift.
39. As an agent, I want stop to take no prompt, so that I can stop a Command when the Day shift tells me to.
40. As the Day shift, I want a Command record that can't be written to be only a warning, so that a full disk or a permissions problem never fails a Run.
41. As the Day shift, I want the Command record kept after a Command ends, with its outcome, so that history and later charts come from the same files.
42. As the Day shift, I want two child Runs of one Spec run never to overwrite each other's record, so that concurrent Tickets are all shown.
43. As the builder of the Console, I want every view and action it will offer to be one of these commands, so that the page can never do more than the CLI.
44. As the Day shift on two machines, I want each machine's status to name its host, so that I know which machine I'm looking at.
45. As the Day shift, I want status to cover every repository under the configured logs directory, so that one command answers for the whole machine.

Implementation Decisions

• The Command record is files, and each process writes only its own.

  • The Command's process writes the Command's record, beside its Command log.
  • Each child Run (a Ticket's Run, or a Base fix) writes its own child record beside it, named for its issue. No two processes ever write the same file.
  • Each write replaces the whole file atomically: write a temporary file, then rename it over the old one.
  • Records are kept after the Command ends.
  • SQLite is deliberately not used. An index built from the files can come later if a query gets slow, without moving any data.

• What a record holds:

  • a format version;
  • the Command's kind and repository;
  • its issue number, URL and title, when it has one;
  • the machine's hostname;
  • the process id and the process's start time;
  • the Harness, Model and Effort;
  • when it started;
  • the stages it has been through, each with when it began;
  • its current Session log;
  • for a Spec run, each Ticket with its state and its blockers;
  • for a Pass, what it dispatched;
  • once ended, its outcome, cause, and the Issue branch and pull request it left;
  • a Stop request, when there is one.

• Stages. A record begins a stage at each transition. The stages are:

  • for a Run: implement, Resume, Security review (when it has one), pull request (pushing it and marking it ready), CI watch, Repair, Self-merge and cleanup;
  • for a Spec run: Tickets, Spec review, and then the same Delivery stages for its Spec PR;
  • for Passes: Architecture review, Security audit, Security reproduction and Security fix publishing.

  A reader ignores stage names it doesn't know, so an older binary can read a newer record.

• Liveness. A record with no outcome is live only while its process id is running with the recorded start time. Otherwise it died at its last stage. Comparing the start time guards against a reused process id. The check has to work on both Linux and macOS, because thirdshift ships an aarch64-apple-darwin build.

• Writing never changes an outcome. A record that can't be written gives one warning per Command and carries on, as a Run notification failure does.

• The latest event comes from the tail of the current Session log, not the record, so a record isn't rewritten on every tool call.

• History when there's no record. status --all falls back in two steps:

  • Commands with Activity log lines get their start, end, outcome and Command log from those lines, without stages.
  • Before the Activity logs began (2026-10-03 on the thirdshift machine), Session logs in the current layout are grouped by issue and stamp, with no outcome.

  Logs in the flat layout from before ADR 0011 aren't read.

• Idle Passes. For each repository and kind of Pass whose latest Activity log line is a skip, status shows that skip's reason.

• Naming a Command.

  • <repo>#<n>, with <owner>/ required only when it's ambiguous on the machine, or the Issue URL.
  • A Pass with no issue is named <kind>-<stamp>, as its Command log is named.
  • If several Commands match, the live one is chosen, or else the latest.
  • A Ticket's issue resolves to that Ticket's Run inside its Spec run, for show and logs.

• The commands.

  • thirdshift status [--all] [--json]: live Commands, then those that ended in the last 24 hours. --all gives the whole history.
  • thirdshift show <command> [--json]
  • thirdshift logs <command> [--session <kind>] [--follow]: the Command log by default. --session gives that kind's latest Session log as its raw JSON lines. --follow keeps printing until the Command ends.
  • thirdshift stop <command> [--requeue]

• The JSON is the API. status --json and show --json carry a version field and are documented in the README. Field names use the glossary's terms. These commands, not any HTTP endpoint, are what agents and the coming Console rely on (ADR 0016).

• How a Stop works.

  1. stop refuses anything that isn't live, and anything that's a child Run, naming its Command.
  2. It writes a Stop request beside the record. The request says who asked: the Unix user from the CLI, and later an account email from the Console. It also says whether to requeue.
  3. It sends SIGTERM to the Command's process, the interruption make a way to easily see in progress thirdshift runs on machine and cut one off #574 already showed works. The existing forwarding passes it on to child Runs.
  4. The interrupted Command reads the request and writes the Activity log line <subject> stopped from the CLI by <who> before its ended: line.
  5. The Command then ends its Claim the Stop's way:
     • Without --requeue: needs-triage is added and ready-for-agent is not put back. in-progress follows the usual rules: it's removed when nothing reached origin, and kept while an Issue branch or pull request waits for the Day shift.
     • With --requeue: the Claim is released and ready-for-agent put back, even when something is on origin. The next Run then continues the Issue branch (ADR 0002).
  6. stop waits for the record's outcome, or for the process to be gone. It prints a "still ending" line each minute meanwhile. Then it prints the outcome and anything left on origin.

  There's no forced kill in this Spec.

• Where the code goes. The record grows out of the Command's existing log recording, which already knows the Command's kind, stamp, issue, child status and start and end. Claim ending gains a "stopped" path beside its interrupted one. The new commands join the existing command surface.

• The glossary already defines Command record and Stop, and its Activity log and Claim entries cover Stop. They were changed during the grilling.

Testing Decisions

• A good test checks behaviour you can see from outside: what the commands print, their JSON, the labels on the fake GitHub, and the Activity log and Command log lines. It doesn't check the internal layout of the record beyond the documented JSON.

• The main seam is the thirdshift binary, run end to end with the existing fakes. A test starts a real Command whose fake Harness waits, then runs status --json, show, logs (including --follow) and stop against it. Cases:

  • a live Run, its stage and the stage's age;
  • a Pickup run, then Spec run, then Tickets tree, including Tickets waiting on blockers and ones not started;
  • died at a stage, after kill -9;
  • Stop with and without --requeue, with and without anything on origin, checking labels and the Activity log line;
  • Stop on a Spec run ending its Tickets' Runs;
  • Stop refused on an ended Command and on a Ticket's Run;
  • naming by repo#n, by URL, by an ambiguous name, and by a Pass stamp;
  • history from Activity logs with no records, and from Session logs before the Activity logs began.

  Prior art: the Activity log tests, the Command log tests, the Pickup run tests, and the interruption cases in the Spec run tests.

• The existing recorder seam covers what's awkward from the binary: a record that can't be written warns once and the Run's outcome is unchanged; two child Runs writing at the same moment; and a reused process id. This is the Command log and Activity log recording module with its effects adapter for tests. Prior art: its existing recording tests.

• Claim ending's existing in-memory tests gain the Stop cases.

Out of Scope

• The Console: the web page, accounts and sign-in, Stop from a page, console install, Setup's Console questions, and the Console log. These are a separate Spec, which depends on this one.
• Readable Session log rendering. logs --session prints raw JSON lines. Rendering belongs to the Console Spec.
• Stopping a single Ticket's Run inside a Spec run. That's a separate issue, because a Spec run needs rules for a stopped Ticket.
• A forced kill for a Command that doesn't end after SIGTERM.
• Anything across machines: reading another machine's Commands, or a Claim saying which machine took the issue (a separate issue).
• Logs in the flat layout from before ADR 0011, and an SQLite index.
• Starting Commands from anything new.

Further Notes

• Gathers make a dashboard to view all present thirdshift activity on one server at a glance #368 and make a way to easily see in progress thirdshift runs on machine and cut one off #574. make a way to easily see in progress thirdshift runs on machine and cut one off #574's chart35 Base fix in the User config and Setup #239 incident is the acceptance case: thirdshift status shows the Pickup run and its stage, and thirdshift stop chart35#239 ends it. The issue is then labelled needs-triage and isn't taken again, and the Activity log says who stopped it.
• ADR 0009 still holds: nothing here schedules itself. ADR 0016 records why the coming Console may run all the time.
• The research on what to borrow is docs/research/console-ui-inspiration.md on branch web-ui. Symphony and Gas Town both build their status JSON and their CLI first, and the page on top, which is this Spec's order.