Repository navigation
refactor: one managed-service stop routine, one engine IPC client, one endpoint-key respawn guard and one log-tail routine for rocm and rocmd - #929
Conversation
Three modules read the last lines of a log their own way: the ComfyUI log view (whole file, sixteen-bit decode, ANSI strip, lossy text), the services text module (streamed, fails on invalid text) and the rocmd MCP service_logs tool (whole file into one string, fails on invalid text), while the seek-bounded routine in rocm-core serves only the engines. Add characterisation tests that pass on the unmodified code and document the differences before they are removed: a large file (the whole-file readers return a line far outside any tail window; the shared routine does not), a sixteen-bit little-endian log with ANSI colour codes, invalid eight-bit text, a missing file, a limit of zero, fewer lines than the limit, a final line without a trailing newline, and CRLF endings. Test-only; no behaviour change. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The ComfyUI log view, the services text module and the rocmd MCP service_logs tool each read a log tail their own way, while the seek-bounded rocm_core::tail_lines served only the engines. Converge all of them on it. rocm_core::log_tail gains tail_lines_decoded, taking a LogDecode: PLAIN (UTF-8, what the engines and the service logs use) or CONSOLE (UTF-16 little-endian detected by byte order mark or zero-byte pattern, ANSI sequences stripped; what the ComfyUI log view used). The UTF-16 path keeps the seek-boundary handling: the window starts on a character boundary and a partial first line is dropped. strip_ansi_sequences is exported for the assistant-prose note in tool_policy. A shared LOG_VIEW_MAX_TAIL_READ budget (4 MiB) is what the three log views pass. The private readers are deleted; each caller keeps a thin adapter that picks a LogDecode and its own error label. User-visible differences: - A log larger than 4 MiB is read from its last 4 MiB, not whole (all three). - Invalid UTF-8 no longer fails the view: `rocm services logs` and `rocm logs` showed one "<failed to read ...>" placeholder line and the MCP service_logs tool returned an error; both now show the readable text with U+FFFD for the bad bytes. - Service logs and the MCP tool still read UTF-8 and keep ANSI sequences; a UTF-16 service log is now shown as lossy text rather than as an error. - ComfyUI logs: ANSI stripping is per line, so an unterminated escape sequence ends at its line instead of swallowing the lines after it. - Error wording: a missing log is "failed to read <label> <path>" (cause: "failed to open <path>") in services text, was "failed to open <label> <path>"; ComfyUI and MCP name the open step in the cause. - A limit of 0 returns no lines without opening the file in all three (the MCP reader used to return the whole file; ComfyUI and MCP used to fail on a missing file first). Callers clamp the limit, so only direct calls see it. The characterisation tests from the previous commit are updated to the new behaviour, and new tests cover the bound (a counting reader shows a read touches no more than its budget), every budget from nothing to the whole file for UTF-8 and UTF-16, and each caller through the shared routine. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocm logs --search` reads the CLI's action logs through the shared log-tail routine. Add scenario logs-04: an action log with one byte that is not valid text still yields every matching line. Before the log views were converged on the shared routine, that byte made the whole file collapse into a single "failed to read" placeholder line and the search found nothing. The planting step gains an option for the stray byte; the existing steps and the match-count assertion are reused. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…e-the-shared-routine Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…e-the-shared-routine Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…its recorded PIDs rocm (services_text.rs) and rocmd (service.rs) each carry their own MANAGED_STOP_GRACE and terminate_recorded_service_pids, and the two have diverged. Pin what each copy does today, with no production change, so the change that merges them has to edit the tests that describe the difference: the grace value (10 s in both), the order the PIDs are visited (supervisor, then engine), the per-PID start-time pairing and the de-duplication of a PID recorded under both roles, zero PIDs, and the per-outcome signalled/forced/stopped classification. The two divergences are pinned as such: rocm drops the stopping process's own PID from the report while rocmd reports it as `self`, and the Windows `taskkill /T /F` step exists only in rocmd. The latter is a private cfg(windows) block with no injection point, so it cannot be observed from a test yet; only its portable half (an absent PID reaches terminate_verified unchanged) is pinned here. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
rocm and rocmd each carry an ensure_public_service_has_endpoint_key with the same two rules (a service that requires a key but has none; a public bind with no key) and different wording. Pin the shipped text of both rules in both copies, and prove the advice in them: the command rocm names is parsed by the real clap tree, run through validate_bind_host and resolve_endpoint_auth, and the key it stores satisfies the guard that refused. The same is shown for the key file rocmd's recovery reads. The rocm test records one known gap as it stands: the printed 'rocm serve --require-api-key' omits the required model, so it parses only as an incomplete reference. The follow-up that shares the guard fixes the text and flips that assertion. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…by rocm and rocmd `rocm services stop` and `rocmd`'s `stop_server` each carried their own MANAGED_STOP_GRACE (10 s, kept in step only by a comment) and their own terminate_recorded_service_pids, and the copies had diverged. Move one routine and one constant into rocm-core's proc_lifecycle.rs, beside terminate_verified, and delete both private copies. The termination of each PID stays injectable. rocmd's behaviour wins, because it is the more complete one: - The stopping process's own PID is reported as `self` (never signalled, counted as stopped). rocm used to drop it, so `pid_outcomes` listed one entry fewer than rocmd's for the same record. - On Windows a verified root is preceded by `taskkill /PID <pid> /T /F`, because terminate_verified cannot walk a process tree there and the engine runs one level below the recorded PID. rocm never did this, so on Windows `rocm services stop|restart`, the sandbox `stop_server`/`restart_server` tool and the attached `rocm serve` Ctrl-C stop now take the engine's subtree down as `rocmd` already did. A root the tree kill already took is reported `forced`, not `already_gone`. The Windows step sits behind terminate_recorded_pid_with, which takes the tree-kill, the identity read and the termination as arguments, so its tests run on every host. Each caller's tests drive its own stop through the shared routine, and each carries a source check that no private copy has come back. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
rocm and rocmd each carried an ensure_public_service_has_endpoint_key with the same two rules and different wording and remediation text. Move one guard into rocm-core (rocmd must not depend on rocm), together with is_public_bind_host, which it now applies itself. The caller passes only a ServiceStart (Launch, Restart or Respawn), which words what the action would do; both rules and the command the refusal names are the same for every caller. Delete both copies. The rocm wording wins: it says why the key cannot be recovered and how to choose one. Its remediation also gains the model, as rocm serve <model> ..., because rocm serve requires one and the command as printed was a fragment that did not run. The test that followed the command now follows it through the real parser, serve's key policy and the stored key file for every start, and a new scenario covers the restart refusal end to end, including that it leaves the running server running. Each caller is tested through the shared guard: its refusal must equal the guard's text for its own start, so a private copy or the wrong start fails. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…service-stop-routine Signed-off-by: Jussi Elo <jussi.elo@amd.com> # Conflicts: # apps/rocm/src/services_text.rs # docs/architecture.md
…service-stop-routine Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
Confirmed CLI stops retain stale process identities, allowing later Windows tree kills to terminate unrelated processes after PID reuse, while the tree-kill behavior lacks end-to-end Windows coverage.
2 open issues
What changed in this PR
Consolidates managed-service process termination in rocm-core so rocm and rocmd share one routine and the same 10-second stop grace period.
Changes:
- Replaces duplicated termination logic with shared PID outcomes and reporting helpers.
- Aligns CLI stops with daemon behavior, including Windows descendant termination and explicit
selfPID reporting. - Adds regression tests and documents the shared architecture and Windows coverage limitations.
| File | Description |
|---|---|
docs/testing.md |
Documents shared-stop tests and Windows coverage gaps. |
docs/architecture.md |
Updates module ownership and shared termination responsibilities. |
crates/rocm-core/src/proc_lifecycle.rs |
Adds shared termination routines, outcome types, grace constant, and tests. |
crates/rocm-core/src/lib.rs |
Exports the shared managed-stop API. |
apps/rocmd/src/service.rs |
Replaces private termination logic with shared calls and adds caller tests. |
apps/rocm/src/service_control.rs |
Routes stop/restart cleanup through the shared routine and updates tests. |
Review effort: Balanced
| /// one thing this adds is the e2e seam below. | ||
| fn stop_recorded_processes(record: &ManagedServiceRecord) -> Vec<rocm_core::RecordedPidStop> { | ||
| rocm_core::terminate_recorded_service_pids(record, |identity| { | ||
| scripted_stop_outcome().unwrap_or_else(|| rocm_core::terminate_recorded_pid(identity)) |
There was a problem hiding this comment.
Added in f8a44e2, with one caveat: it has not been run on Windows, because I have no Windows host here. service-stop-07 (@requires-os:windows, no GPU) records a PowerShell parent that has started a ping child, with the PID alone and no start-time token as Windows records it, runs rocm services stop --yes, and asserts that both processes are gone together with the stop report (Local server stopped, stopped processes: 1) and the persisted record (stopped, no process, no stop request, key file gone). Its Given checks both processes are running first, so the Then cannot pass for the wrong reason. Without the taskkill /T step the stop ends only the recorded process and the scenario fails on the surviving child. It compiles on the Windows target (clippy for the e2e harness's test target), the scenario ids are sequential on this branch and pass the feature naming checks, and the suite's host filter skips it on Linux (verified by a run that named all the stop scenarios by id: six ran, it was skipped). I also added service-stop-06, the Linux counterpart that runs rocm services stop --yes against a real recorded process and checks the persisted state, since the CLI's own stop of a real process had no scenario either. Note the number collision: the endpoint-key respawn guard PR adds its own service-stop-06, so whichever lands second renumbers.
…nd how they read its answer rocm and rocmd each carry their own client for the JSON-over-stdio request to an engine process, and the two have drifted: they build the child's command line differently, set different environment, treat a failing exit status differently, word their errors differently, and only rocm guards the request write against SIGPIPE. Nothing pinned any of it, so the first step toward one client is to pin what each does today. Both clients are driven against a stand-in engine child, a shell script that records its argv, environment and stdin and then answers (or crashes) as the test asks. The stand-in lives in rocm-engine-protocol behind a test-support feature so the two callers share one. Each caller's out-of-process half is split into a small function that takes the executable to run (engine_stdio_child_request in rocm, engine_request_via in rocmd), so a test can aim it at the stand-in; the bodies are unchanged and the public entry points call them with the same arguments as before. The tests assert the child's argv (rocm: stdio; rocmd: __engine-stdio <engine>), the single newline-terminated request line, the environment each caller sets (rocm: the app path variables, the engine environments root, the progress variable for installs; rocmd: the endpoint key file when it holds a usable key), what each does with a nonzero exit, an unparseable answer and an error envelope, and the SIGPIPE case under the default action in a re-executed child. Mutating rocm's write guard away turns its SIGPIPE test red. No behaviour change. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ls-use-the-shared-routine Signed-off-by: Jussi Elo <jussi.elo@amd.com> # Conflicts: # apps/rocm/src/services_text.rs
…ared by rocm and rocmd rocm and rocmd each carried their own client for the JSON-over-stdio request to an engine process, and the two had drifted. They now run the child through one implementation in rocm-engine-protocol (stdio_client.rs); each caller still chooses which program to run, and what it asks of the client is a small set of options: the app paths, the engine environments root, an endpoint key file, a progress request, and whether to forward the child's stderr as it arrives. rocm still answers built-in engines in its own process and resolves the engine binary; rocmd still re-executes its own executable as __engine-stdio. Where the two behaved differently, the client takes the behaviour that reports more or fails more safely: - A failing exit status is an error only when the child printed no answer; an answer on stdout is decoded even if the child then exited non-zero (rocm's rule; rocmd discarded the answer for the status). - A parse failure carries the child's stderr, and the decode errors use rocm's wording (rocmd's said "failed to parse engine response", "engine response did not include data" and "failed to deserialize engine response data"). - The write of the request is guarded against SIGPIPE (rocm's rule). rocmd had no guard, so the daemon hosted in rocm, which runs with SIGPIPE at its default action, was killed by a write to an engine that had already exited instead of reporting why. - The child runs on the caller's app paths in both (rocmd now sets the ROCM_CLI_*_DIR variables; it relied on inheriting them), and the spawn error names the program in both. - A request whose write fails kills and reaps the child, so the long-lived daemon does not collect a zombie per failure. - Forwarded stderr no longer panics if this process's stderr is closed. The endpoint key stays an option only rocmd uses, since none of rocm's requests is about a running service's endpoint; rocm still sets no key. Stderr is forwarded for rocm and only kept for rocmd. Neither client has, or gains, a timeout. Also moved to where both can reach them: with_sigpipe_ignored (and its re-exec test, same name) from rocm's main.rs to rocm-core; the three ROCM_CLI_*_DIR variables as AppPaths::env_vars, which rocm's app_path_env_vars and rocmd's capture wrapper now call instead of each listing them; and the byte-identical optional_arg, from both binaries into rocm-engine-protocol. The fake engine written for the tests now plants its script with rocm_core::fake_executable instead of retrying on ETXTBSY. Both callers' tests now run through the shared client and flip what the previous commit pinned (rocmd's app paths, nonzero-with-answer, SIGPIPE under the default action, error wording). Breaking either caller's options, the client's SIGPIPE guard or an environment variable turns a test red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… the standalone daemon rocmd asked an engine to do something by re-executing its own executable as `__engine-stdio <engine>`. That verb is served by rocm. Embedded as `rocm daemon` the daemon is rocm and it works, but the standalone rocmd binary shipped beside rocm answered every such request with "unrecognized subcommand '__engine-stdio'" and exit status 2. The request was made by the stop tool (`rocmd sandbox-tool stop_server`, `rocmd mcp-server`), by the supervisor's readiness wait and by the server-recover watcher's healthcheck, so from the standalone daemon the engine `Stop` always came back as an error and no healthcheck ever succeeded; for an engine whose stop decides confirmation, the stop could never be confirmed. When the running executable is exactly `rocmd`, the request now runs the `rocm` beside it (rocm_core::sibling_binary_path), and says so if there is none. Anything else, the embedded daemon, a test binary or a renamed copy, keeps using its own executable as before. The `service-stop-02` scenario drives the standalone rocmd and now also asserts that the engine `Stop` request was answered. Run against the previous commit's rocmd it fails with the engine_stop error above; with this change it passes, along with the rest of the service-stop scenarios. The choice of executable is a pure function with its own unit test that also runs on Windows. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Conflicts: the restart function and its tests moved from services_text.rs to service_control.rs, so the shared guard call (ServiceStart::Restart) and the restart-text assertions are re-applied there, and the stale mention of the deleted rocm copy is dropped from the notes in main.rs and service_control.rs. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Resolves the conflicts with the clap-tree move: the tests it relocated to cli.rs and the SIGPIPE test this branch moved to rocm-core are both gone from main.rs, the engine_ipc.rs module comment keeps this branch's wording, and the docs/architecture.md engine_ipc.rs entry keeps this branch's text plus the note that the watcher-mode conversion now lives in cli.rs. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocm` runs with SIGPIPE at its default action. When it forwards an engine child's progress to a stderr whose reader has gone (a closed pipe), the write delivers the signal and ends the CLI before `write_all` can return, so discarding the result did not protect the request. Both forwarding writes in `collect_stderr` (a line of progress, and the note recorded when reading the child's stderr fails) now go through one helper that runs the write inside `with_sigpipe_ignored`, which turns the signal into the error that is then discarded. `with_sigpipe_ignored` is now safe to overlap. It saved and restored the disposition per call, so two overlapping regions (the stderr forwarder of one request next to the stdin write of another in the daemon) could restore "ignored" over a default action, or the default under a region still open. It remembers the disposition in force before the first region and restores it when the last one ends, and restores on unwind. The re-executed test installs SIG_DFL in a fresh process and forwards into a real pipe whose reader is closed, once per write; the existing `Closed` mock returns the error itself and cannot raise the signal. A second re-executed test covers nested and interleaved regions. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`FakeEngine` built its script by interpolating the canned response and the scratch directory into single-quoted shell literals. A valid JSON response with an apostrophe (`can't stop`) ended the literal early, so the fake exited with a syntax error instead of printing the envelope, and a `TMPDIR` containing an apostrophe broke the `dir=` line the same way. Both values now go through one `shell_quote` (close the literal, emit an escaped quote, reopen it), and `FakeEngine::new_in` lets a test choose the scratch base without changing the process-wide TMPDIR. Tests cover the quoting helper, the response and the scratch directory against the script itself, and the apostrophe response through the real client. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… trailing escape fragment Two ways the shared tail read disagreed with what it advertises, both found in review. The UTF-8 branch seeked from the size taken by the earlier stat and then read to the current end of the file, so a log whose writer kept appending was consumed past the byte budget by however much it had grown. The sixteen-bit branch already capped its read with take(size - window_start); the UTF-8 branch now does the same. The new test hands the routine a source that grows on its first read and asserts the lines stop where the snapshot ended and the bytes touched stay within the window; it is red with the cap removed. A console log that ends in a colour reset after its last newline ended in an unterminated fragment made only of an escape sequence. The routine counted it as a line, so with a limit of one it evicted the real last line and returned an empty string, where the ComfyUI reader it replaced (strip ANSI from the whole text, then split) returned the line. A final unterminated fragment that strips to nothing is no longer counted when ANSI stripping is on; a terminated blank line, including one made only of an escape sequence, is still a line, and plain decoding strips nothing so it keeps the fragment. The tests cover UTF-8 and UTF-16 with and without a byte order mark, and compare the result with a copy of the old reader on inputs where the two agree. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…utput The log views read at most the last 4 MiB of a log and drop a partial first line. When the newest line is itself longer than that bound the window holds only a fragment of it, so the read returned no lines and the views printed "The saved log is empty." (rocm comfyui logs) and "Recent output: no output yet" (rocm services logs) for a file that is not empty. The byte bound stays. rocm-core gains tail_log_decoded, which returns the same lines in a LogTail that also records how many bytes at the start of the file the budget left unread, and LogTail::newest_line_exceeds_the_read_bound tells an empty tail of a log with content from an empty log (or a limit of zero). The two views use it: an empty file still says it is empty, and a log whose newest line does not fit says so and names the bound, derived from LOG_VIEW_MAX_TAIL_READ so the number cannot drift from it. The unit tests and four new scenarios (logs-05 to logs-08, one pair per view) assert each message together with the log on disk that it describes: the size and the absence of a newline for the long line, zero bytes for the empty file. The rocmd MCP service_logs tool and the chat and launch-failure readers print no claim about emptiness, so they are unchanged. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocmd`'s stop clears the supervisor and engine PIDs and their start-time tokens from the record once every recorded process is confirmed gone; `rocm services stop` did not. Its record kept naming the processes it had just stopped, so a later stop of the same record reached the termination again. A PID recorded without a start-time token (every one on Windows) cannot be refuted once the OS has recycled it, and the Windows tree kill that the shared routine added for the CLI would then take the unrelated process and everything under it. The clearing becomes one `ManagedServiceRecord::clear_recorded_processes` in `rocm-core`, called from the confirmed branch of both stops; `rocmd`'s four field assignments are replaced by the call, so the two leave the same record behind. An unconfirmed stop still keeps the PIDs, since the record is the only thing that names a survivor. `a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing` runs the real termination routine with the per-PID signal injected: after the first stop the persisted record names no process, and a second stop signals nothing and still reports stopped. Before the change the record kept PID 4242 and the second stop visited it again. The unconfirmed direction and the clearing method have their own tests. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The untested Windows subtree-termination path and documented recycled-PID risk from unavailable creation-time identity checks require live Windows validation and maintainer judgment.
1 resolved since last review
Review effort: Balanced
Rust 1.96 clippy::duration_suboptimal_units flags Duration::from_secs(60) in the service-stop step. The deadline stays 60 seconds; only the unit it is spelled in changes. The e2e harness target sets test = false, so the workspace clippy run skips it and only the dedicated e2e-cucumber --test e2e clippy step lints it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for the thorough consolidation work; the shared stop routine, single IPC client, respawn guard and log-tail routine all read well, CI is green, and the added scenario coverage (service-stop-06..08, logs-04..08) is solid.
Requesting changes mainly on scope, plus three smaller items inline:
- Scope: four independent changes in one PR. The stop routine, the engine IPC client, the endpoint-key respawn guard and the log-tail routine are described in the PR body as independent, and each tracking ticket asks for its key to be delivered in its own PR. Bundling them (with three merge commits) makes ~8.5k lines hard to review and to revert selectively. Please split into four PRs, or get agreement from the ticket owner to bundle.
- Child not reaped on stdout read failure (inline,
stdio_client.rs). - MCP
service_logsstill returns an ambiguous empty tail (inline,mcp.rs): same class as the CLI log views fixed here. - Windows
taskkill /T /Fblast radius (inline,proc_lifecycle.rs): needs an explicit decision on recycled-PID protection.
Also worth confirming each of these user-visible changes is intended: rocmd now decodes the answer on a non-zero engine exit, the rocmd respawn wording is now rocm's, and a confirmed rocm services stop clears the recorded pids.
| .stdout | ||
| .take() | ||
| .context("engine stdio child did not expose stdout")? | ||
| .read_to_end(&mut stdout) |
There was a problem hiding this comment.
If reading stdout fails, ? returns before child.wait() and leaves the stderr thread detached, so the child is neither reaped nor killed (the write-failure branch above does kill+wait). Consider killing and waiting here too, since this client now runs in the long-lived daemon.
There was a problem hiding this comment.
You are right: the ? on the stdout read returned before child.wait(), so a failed read left the child running and never reaped, in a client that now lives in the daemon. Fixed in 5d9a570: the read moved into a small read_response step that kills and waits on the child before returning the error, like the write-failure branch above it. I did not join the stderr collector on that path, because a grandchild holding the pipe could block the join; the thread ends by itself when the pipe closes. The new test reads through a failing reader on a real child and checks that its /proc entry is gone, and I confirmed it fails with the kill removed and with the wait removed.
| lines.push_back(line.to_owned()); | ||
| } | ||
| Ok(lines.into_iter().collect::<Vec<_>>().join("\n")) | ||
| let lines = rocm_core::tail_lines(path, limit, rocm_core::LOG_VIEW_MAX_TAIL_READ) |
There was a problem hiding this comment.
tail_lines returns no lines both for an empty log and for a log whose newest line exceeds LOG_VIEW_MAX_TAIL_READ; the caller then reports "Read the last N line(s)" with an empty tail. The CLI views got tail_log_decoded + newest_line_exceeds_the_read_bound for exactly this; the MCP tool is the same class. Use the same signal here or note the gap in the PR.
There was a problem hiding this comment.
Agreed, same class as the CLI log views. Fixed in 4398998 (plus a lint-only follow-up, 8ff2ff9): service_logs now reads with tail_log_decoded and answers through one shared sentence, rocm_core::service_log_tool_summary, which says "has no log output yet" for an empty log, names the 4 MiB bound when the newest line exceeds it, and otherwise reports how many lines were read. The structured reply also carries newest_line_exceeds_read_bound. The CLI's own copy of the tool (rocm mcp-call service_logs) had the same problem and uses the same sentence, so scenarios logs-13 and logs-14 cover it against the file on disk, and a rocmd unit test covers the daemon's copy because the harness only drives the rocm binary. A missing log is still an error rather than an empty tail.
| /// has been recycled by an unrelated process still passes it, and `taskkill /T | ||
| /// /F` then takes that process and its whole subtree. Closing that needs a | ||
| /// Windows process creation time in [`process_start_ticks`]. | ||
| pub fn terminate_recorded_pid(identity: &ProcessIdentity) -> TerminationOutcome { |
There was a problem hiding this comment.
This puts taskkill /T /F behind every rocm stop path on Windows while process_start_ticks is still None there, so a recycled PID passes the Matches gate (documented above). Clearing the record on a confirmed stop removes the repeat-stop route but not the "service died by itself, PID recycled, record still names it" route. Worth an explicit decision: land a Windows creation-time identity first, or accept and track.
There was a problem hiding this comment.
You are right that a recycled PID passed the gate. I closed it instead of tracking it, in 5f9baad: on Windows process_start_ticks is now the process creation time from GetProcessTimes, so a record carrying a token refutes a recycled PID the same way as on Linux, and the stop reports identity_mismatch without running taskkill on it. A record written before this has no token and stays best-effort, and a token that can no longer be read is reported as unverified rather than signalled; that outcome is now reachable on Windows, so the check command it prints is tasklist /FI "PID eq <pid>" there instead of ps. Scenario service-stop-09 stops a server whose recorded PID belongs to a running bystander and checks the bystander is still running (it passed on Linux and fails when a recycled PID is let through), and there are unit tests for the FILETIME join and for the refusal. I could not run Windows code here, so the Windows-only tests and the Windows half of service-stop-09 are for the Windows jobs; the Windows clippy check of the new FFI is clean.
…cannot be read run_engine_stdio killed and reaped the child when writing the request failed, but a failed read of its stdout returned through `?` first, leaving the child running or as a zombie. The client now runs in the long-lived daemon, so each such failure leaked a process. The read moves into read_response, which kills and waits on the child before returning the error; the stderr collector is not joined on that path because a grandchild holding the pipe could block the join, and the thread ends once the pipe closes. The test reads through a failing reader on a real child and checks that its /proc entry is gone, which a kill without a wait would not achieve. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…s the read bound in the service_logs tool The tool read a log's tail without saying what the byte budget left unread, so an empty log and a log whose newest line is longer than the 4 MiB a log view reads both came back as "Read the last 80 line(s)" with an empty tail. The CLI log views already separate the two; the tool is the same class. It now answers through one shared sentence, rocm_core::service_log_tool_summary (no output yet, newest line too long to show, or how many lines were read), and its structured reply carries newest_line_exceeds_read_bound. Both copies of the tool use it: the daemon's MCP server and the CLI's own tool bridge (rocm mcp-call). The notice clause moves from the CLI into rocm-core so the daemon, which cannot depend on the CLI, words it the same way, derived from the read budget. The line count in the summary is now the number of lines returned, not the number asked for. A missing log file is still an error. Scenarios logs-09 and logs-10 run rocm mcp-call service_logs against an over-long line and an empty log and check the reply against the file on disk; the daemon's copy is covered by a rocmd unit test, since the harness drives only the rocm binary. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…hort clippy's too_long_first_doc_paragraph rejects the two doc comments added with the service_logs summary; split each into a one-sentence summary and the rest. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ndows, so a recycled PID is refused before the tree kill process_start_ticks read /proc and was None on Windows, so no recorded identity could be refuted there and the identity gate in front of `taskkill /T /F` only turned away a PID that was not running at all. A service that died by itself, whose PID an unrelated process then took, still passed the gate, and the stop took that process and its whole subtree. Clearing the record on a confirmed stop closed the repeat-stop route but not this one. On Windows the token is now the process creation time from GetProcessTimes, a FILETIME that only has to be equal to the value read when the PID was recorded. A record carrying one refutes a recycled PID exactly as on Linux: terminate_recorded_pid reads it back as Recycled, skips the tree kill and reports identity_mismatch. A record written before this has no token and stays best-effort. A recorded token that can no longer be read (the process is more privileged) reads as Indeterminate, and the stop reports it unverified instead of signalling it. That outcome is now reachable on Windows, and its advice named `ps -p`, which Windows does not have; the check it names is now `tasklist /FI "PID eq <pid>"` there. The FILETIME join is a pure function tested on every host. The refusal runs against the test process's own PID with a wrong token on Linux and Windows, and against a bystander process on Windows, where a stop that went ahead would run taskkill. Scenario service-stop-09 stops a server whose recorded PID belongs to a running bystander under a token no live process has and checks the bystander is still running; it runs on every OS. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Conflicts: docs/architecture.md (main's --follow addition to the services_text.rs entry, with read_tail_lines now read_log_tail), tests/e2e-cucumber/features/logs.feature (main's follow scenarios keep logs-04 to logs-07; this branch's log scenarios move to logs-08 to logs-14), and apps/rocm/src/services_text.rs (main's io imports; main's private follow LogTail now sits beside rocm_core::LogTail, which is named by path). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Conflicts: apps/rocm/src/service_control.rs and service_launch.rs (main's port-lease restart and launch tests meet this branch's shared stop routine and respawn guard), apps/rocmd/src/engine_ipc.rs (main edited the private respawn guard this branch deleted) and docs/architecture.md (main's serve_port.rs and engine-protocol lib.rs entries). Main now requires an explicit --port with any host but the default loopback, so the public-bind remediation the shared guard prints names `rocm serve <model> --host <host> --port <port> --allow-public-bind`; the text, its tests, and the check that the named command clears the refusal (which now also applies the port rule) follow. Main's stop stand-in RecordedStopTermination::confirmed_stopped is an empty Vec of RecordedPidStop here, rocm_engine_protocol::is_public_bind_host is rocm_core::is_public_bind_host, and the restart test that expected a kept PID after a confirmed stop expects the cleared one. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Conflicts: apps/rocm/src/engine_runtime_select.rs (a module doc paragraph: main's note that the single-ready-runtime rule lives in runtime_adopt, without the optional_arg mention this branch moved to the engine-protocol crate) and docs/architecture.md (main's edits to the engine_ipc, engine_runtime_select, hardware_root and log_tail entries, merged word by word with this branch's rewrites of the same entries). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
|
On scope: I bundled the four changes deliberately. They share one routine per concern in On the three inline items: the unreaped child on a stdout read failure is fixed in 5d9a570, the ambiguous MCP On the user-visible changes you asked about: each is intended and listed in the body. I also merged |
| let process = spawn_parent_with_descendant(&pid_file); | ||
| let pid = process.id(); | ||
| plant_server_record(world, "0.0.0.0", true, process, None); |
There was a problem hiding this comment.
You are right: a stop that killed PowerShell but left ping would have left it running for its five minutes, because teardown only killed the recorded parent. Fixed in 806c907: the Given step keeps the child's identity (PID plus creation time where the host reads one) on the world, and teardown, which runs when a step fails too, ends it when it is still that process. Whether to end it is a pure function, stray_process::leftover_action, with a table test on every host (ended only while it is still the recorded process, never once gone, reused or unconfirmed), and the Windows clippy check compiles the teardown; the scenario itself has not run on Windows here.
Conflicts: apps/rocm/src/chat.rs (main moved the tool bridge into chat/mcp_bridge.rs, so this branch's service_logs change is re-applied there and the old block stays deleted), apps/rocm/src/engine_runtime_select.rs (main's doc paragraph now names chat/mcp_bridge.rs), apps/rocm/src/service_control.rs (module doc: service_launch.rs no longer imports this module, and the respawn guard is rocm_core's) and docs/architecture.md (main's edits to the service_control, services_text and setup_cmd entries merged word by word with this branch's; the log_tail entry now points at chat/mcp_bridge.rs). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The regression test lacks failure-safe cleanup for the descendant process, so a failed assertion can leave ping running on the Windows runner for the remainder of its five-minute lifetime.
3 open issues
Review effort: Balanced
… covers for every recorded and live token A Windows lane run showed service-stop-07, which records a PowerShell parent with no start-time token, ending with the recorded process still running after a stop that reported success. Reading the stop path, a record with no token already gets the whole stop (identity_state reads it as a match, so the tree kill and the verified termination run); I could not run Windows code here and did not find a path in the creation-time change that skips it. What the change did leave was that this decision was spread over identity_state and terminate_verified with the system reads mixed in. The decision now lives in two pure functions, classify_live_identity (a process is refused only when a token was recorded and the live one is provably different; no recorded token is a match; a recorded token that cannot be read back is indeterminate) and refusal_before_signalling, which identity_state and terminate_verified use. A table test drives terminate_recorded_pid_with through them for no token (the shape service-stop-07 plants), a matching token, a differing token and an unreadable one, asserting whether the tree kill runs and the verdict label together; each row was shown red by a targeted mutation of the classification. A Windows-only test stops a real process recorded without a token. The step that checks the recorded process has exited now allows its handle a few seconds to signal, since the stop confirms an exit from the OS exit code; a survivor still fails it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The interleaved half of overlapping_guards_restore_the_original_disposition_once only made sure both guards were open before the first left, not that the first thread to be spawned entered first. If the other closure entered first, a guard that saved and restored the disposition per call passed both assertions. The second thread now waits on a barrier that the first reaches from inside its guard, so the first region always opens, and closes, before the second opens its own; with a per-call save and restore in place of the shared counter the test fails on every run (five of five). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…, even when the scenario fails service-stop-07 records a PowerShell parent that has started a ping child. If a stop kills the parent but leaves the child, which is the regression the scenario exists to catch, the world's teardown only killed the recorded parent and a later stop cannot recover a child from a dead parent, so ping ran on for its five minutes. The Given step now keeps the child's identity (its PID with its creation time where the host reads one) and teardown, which runs when a step fails too, ends it when it is still that process. Whether to end it is a pure function, stray_process::leftover_action, with a table test that runs on every host: only a process that is still the recorded one is ended, never one that is gone, whose PID has been reused, or whose identity cannot be confirmed; each row was shown red by a targeted mutation. The Windows clippy check compiles the teardown; the scenario itself has not run on Windows here. Signed-off-by: Jussi Elo <jussi.elo@amd.com>


Closes ROCMAI-874, ROCMAI-875, ROCMAI-876 and ROCMAI-879.
Summary
rocmandrocmdare two binaries that act on the same managed-service records, and four pieces of logic they need in common had each been written twice, or three times, and had drifted apart. This change replaces each pair with one implementation in a crate both binaries may depend on (rocm-core, orrocm-engine-protocolfor the engine client;rocmdmust not depend onrocm, so the shared code cannot live in either app), deletes the private copies, and makes the callers thin. In each case the copy that reports more or fails more safely is the one that survives, and every resulting difference a user or an MCP client can see is listed below. Along the way the work found and fixed three real defects that the duplication had hidden: the standalonerocmdcould not reach any engine, a printed remediation command was not runnable, and a log view reported a log that had content as empty.The four consolidations are independent of one another and are delivered as four sections below, each with its own commits, in the order they appear in the history: the managed-service stop routine, the endpoint-key respawn guard, the engine stdio client and the log tails. Within each section the first commit only pins what the copies did before (tests that pass on the unmodified code), so the commit that follows reads as a diff of behaviour. Signed merge commits bring the four lines of work together (and later merges bring in
main); the conflicts were indocs/architecture.md,docs/testing.md, theservice_stopfeature file andapps/rocmd/src/engine_ipc.rs, and the one scenario-number collision is described under User-visible behaviour.Managed-service stop routine (ROCMAI-874)
rocm services stopandrocmd'sstop_serverstop the same service records, but each carried its own copy ofMANAGED_STOP_GRACE(10 s, kept in step only by the doc comment "Matchesrocm services stop") and its ownterminate_recorded_service_pids. The copies had diverged.rocmdreported the stopping process's own PID asselfand, on Windows, rantaskkill /PID <pid> /T /Fon a verified root beforeterminate_verified, because the engine runs one level below the recorded PID andterminate_verifiedcannot walk a Windows process tree (itsprocess_tree_pidsreturns only the root there). Therocmcopy dropped its own PID from the report and had no tree-kill step.Whether
rocm services stopon Windows actually leaves the engine child running is not confirmed: I did not reproduce it and have no Windows host here. What can be verified from the code is only that therocmcopy had no step that could take a child below the recorded PID on Windows, and thatrocmdadded one for exactly that reason. This change makes both binaries run the same routine, so the question stops depending on which binary stopped the service.rocmd's behaviour wins, because it is a superset: it reports every recorded PID including the stopping process's own, and it carries the Windows descendant kill. The one routine lives inrocm-core'sproc_lifecycle.rsbesideterminate_verified, with the per-PID termination still injectable asrocmdalready had it.MANAGED_STOP_GRACEapps/rocm/src/service_control.rsand inapps/rocmd/src/service.rsrocm_core::MANAGED_STOP_GRACEterminate_recorded_service_pidsRecordedStopTermination(rocm) andVec<RecordedPidStop>(rocmd)rocm_core::terminate_recorded_service_pids(record, terminate) -> Vec<RecordedPidStop>terminate_recorded_pid(Windowstaskkill /T /Fbefore the verified termination)cfg(windows)block inrocmdonlyrocm_core::terminate_recorded_pid, built onterminate_recorded_pid_withwhose tree-kill, identity read and termination are argumentsRecordedPidStop,RecordedPidOutcomerocmdprivate struct (string outcome) androcmprivate structrocm_core::RecordedPidStop/RecordedPidOutcomewithsignaled(),forced(),stopped()andreport_entry()rocm'sRecordedStopTermination,RecordedPidOutcomestop_recorded_processesinservice_control.rsis a thin wrapper that adds only theROCM_E2E_STOP_OUTCOMEtest seamCommits, in order: the first pins what each copy did (grace value, visiting order, start-time pairing, aliasing, zero PIDs, classification) and the two divergences, with no production change; the Windows tree kill could not be pinned there because it was a private
cfg(windows)block with no injection point, only its portable half is. The second adds the routine and constant torocm-core, deletes both private copies, rewrites the two callers, replaces the stop stand-ins inservices_text.rs's tests, and updates the docs. Two commits from review: a confirmedrocmstop now clears the recorded processes through the new sharedManagedServiceRecord::clear_recorded_processes(whichrocmdcalls too), and the last commit adds the end-to-end scenarios for the Linux and Windows stops. Two merges ofmainsit inside this line of work because the stop path was moved out ofservices_text.rsintoservice_control.rsby an earlier merged change; the consolidation was re-applied atservice_control.rs.Differences a user can see, with the surface each appears on:
rocmnow takes a stopped service's descendants down withtaskkill /PID <pid> /T /F. This applies to every path that goes throughstop_internal_managed_service_confirmed_with:rocm services stop,rocm services restart(and the restart that moves a service onto a new runtime),rocm sandbox-run stop_server|restart_server, and the stop in the attachedrocm serveCtrl-C handler. The step runs only for a root whose identity gate matches, before the bounded wait. A root the tree kill already took would read back asalready_gone, so it is reportedforcedinstead (asrocmdalready did); that means such a PID now appears insignaled_pidsand in thestopped processes: Ncount. On Linux nothing is added:terminate_recorded_pidhas no tree-kill step there, exactly as before.pid_outcomesinrocm's stop report now includes the stopping process's own PID as{"pid": <pid>, "role": ..., "outcome": "self"}, asrocmd's already did. It was previously omitted. It is not signalled, counts as stopped, stays out ofsignaled_pids, and does not change the unconfirmed-stop text, which only acts ontimed_outandunverifiedoutcomes. In practice it appears only when the record's PID is the calling process (an attachedrocm serve, or a test); a separaterocm services stopprocess is never the recorded one.rocmd: no change. Same visiting order (supervisor, then engine, each with its own start-time, an aliased PID once undersupervisor_and_engine), sameselfhandling, same classification intosignaled_pids,force_signaled_pidsandskipped_pids, same 10 s grace. The grace is 10 s in both before and after.rocm services stopnow clears the recorded processes from the record, asrocmd's stop already did: after it, the persisted record hassupervisor_pid0,engine_pidnull and both start-time tokens null (statusstopped, no stop request). Before,rocmleft the stopped PIDs and tokens in the manifest. It is one call,ManagedServiceRecord::clear_recorded_processes, made from the confirmed branch of both stops (the four assignments inrocmd's stop are replaced by it). An unconfirmed stop keeps the PIDs, because the record is the only thing that names a survivor. This is what makes the next point safe: a later stop of the same record finds nothing to signal.Windows identity (review fix). The tree kill is behind the same identity check as the rest of the stop. On Windows
process_start_ticksis now the process creation time (GetProcessTimes, aFILETIME); it returnedNonethere, so no recorded identity could be refuted and a recycled PID passed the gate in front oftaskkill /T /F. A record written by this version refutes a PID that an unrelated process has taken: the stop reads it back as recycled, runs notaskkillon it and reportsidentity_mismatch. A record written before this has no token and stays best-effort, and a confirmed stop clears the recorded PIDs (difference 4), so that exposure is limited to records that predate the token and have not been stopped since. A recorded token that can no longer be read (a process owned by a more privileged account) reads as unverified and is not signalled. That outcome used to be Linux-only, so the check it advises is nowtasklist /FI "PID eq <pid>"on Windows (it wasps -p <pid> -o pid,args, which Windows does not have). I could not run any Windows code here: the Windows-only tests and the Windows half of service-stop-09 are for the Windows jobs to exercise, and the Windows clippy check compiles the new FFI.rocm-corenow exportsMANAGED_STOP_GRACE,RecordedPidOutcome,RecordedPidStop,terminate_recorded_pidandterminate_recorded_service_pids;terminate_recorded_pid_withstays private.taskkillis run exactly asrocmdran it (blockingstd::process::Command::status, no timeout, output discarded). I kept that so this stays a consolidation; a bounded wait would be a separate change.Endpoint-key respawn guard (ROCMAI-876)
The guard that refuses to bring a managed service back up when it needs an endpoint API key and has none existed twice: in
rocm(service_launch.rs) and inrocmd(engine_ipc.rs, which said it "mirrors the guard of the same name inrocm"). Both applied the same two rules (a service launched with--require-api-keythat has no key; a service bound to a public host with no key) but worded them differently and named slightly different remediation commands, so a user saw one text whenrocm services restartrefused and another when the daemon's recovery refused the same service. Both are replaced by one guard inrocm-corethat takes the action wording as a parameter, and both copies are deleted.is_public_bind_host, which the guard has to apply itself, moves with it fromrocm-engine-protocoltorocm-core(no engine used it, androcm-engine-protocoldepends onrocm-core, not the other way round).While proving that the remediation command the refusal names actually clears the condition, the new test found that the printed command was not runnable:
rocm serve --require-api-keyleaves out the required model, so clap rejects it as printed. The shared text now namesrocm serve <model> --require-api-key(androcm serve <model> --host {host} --port <port> --allow-public-bind;mainhas since started requiring an explicit--portwith any host but the default loopback, and the text names it).ensure_public_service_has_endpoint_keyapps/rocm/src/service_launch.rsandapps/rocmd/src/engine_ipc.rs(two copies, deleted)crates/rocm-core/src/endpoint_key.rs, taking aServiceStartServiceStart(Launch,Restart,Respawn)crates/rocm-core/src/endpoint_key.rsis_public_bind_hostcrates/rocm-engine-protocol/src/lib.rscrates/rocm-core/src/endpoint_key.rs(its test moved with it, same leaf name)Callers:
service_launch.rs(spawn,Launch),service_control.rs(rocm services restart,Restart),supervise.rs(two sites) andwatchers/server_recover.rs(Respawn).mcp_argv.rsandis_loopback_hostinservice_launch.rsnow reachis_public_bind_hostthroughrocm_core. Commits, in order: the first pins both copies' shipped text and the check that the command each names clears the refusal (forrocm, through the real clap tree,validate_bind_host,resolve_endpoint_authand the stored key file; forrocmd, the key filerocm servewrites), and records the missing-model gap as it stood; the second moves the guard, flips that assertion, rewrites the call-site tests to go through the shared guard, adds the restart scenario and updates the docs. A merge ofmainfollows.Which wording wins:
rocm's. It says why the key cannot be recovered ("the key is dropped when a service stops") and how to choose one (--api-key,ROCM_SERVE_API_KEY), and it states the rule that actually fires (--require-api-key) whererocmd's said "launched with an endpoint API key", which is also true of every public bind. The action wording is the only thing the caller supplies. Every difference, per binary:rocm services restart, both rules: the text is byte-identical except the named command now carries the model,rocm serve <model> --require-api-keyandrocm serve <model> --host {host} --port <port> --allow-public-bind(wasrocm serve --require-api-keyandrocm serve --host {host} --port <port> --allow-public-bind).rocm servespawn (reached only when a key file is present but unusable, sinceserve()stores the key first): "so restarting it would reopen it without authentication" becomes "so launching it would open it without authentication", because nothing is being restarted there; plus the same<model>change.rocmdrespawn (the supervisor's refusal on stderr, and theserver-recoverwatcher'srestart_managed_service_refusedevent, which wraps it in "cannot recover managed service ... : ..."): rule 1 was "refusing to recover a service that was launched with an endpoint API key but no longer has one: it would come back up without authentication, and something outside this machine may still be publishing its port. Relaunch it withrocm serve --require-api-keyto issue a new key." and is nowrocm's rule 1 with "respawning it would reopen it without authentication", the dropped-key sentence, the<model>command and the "add--api-key <key>, or set ROCM_SERVE_API_KEY" hint. Rule 2 was "refusing to respawn a service bound to the public host{host}without an endpoint API key: it would come back up without authentication. Relaunch it withrocm serve --host {host} --port <port> --allow-public-bindto issue a new key." and is now the shared rule 2 with the same changes. The "refusing to ..." prefix is gone.rocm_engine_protocol::is_public_bind_hostno longer exists; nothing in the workspace outside the two binaries used it. The<model>placeholder is the oneadvised_commands.rsalready substitutes, so its scan covers the shared text too (it scanscrates/).Engine stdio client (ROCMAI-875)
rocmandrocmdeach carried their own client for the JSON-over-stdio request to an engine process, and the two had drifted apart. They now run the child through one implementation,stdio_client.rsinrocm-engine-protocol, and each caller keeps only what is genuinely its own: which program to start, and a small set of options for what it asks of the client. The wire format does not change: the request and response envelopes are untouched, so the engines need no change.Writing the characterisation tests first also settled the open question about the standalone
rocmdbinary, and it turned out to be a live bug rather than a corner case.rocmdreached an engine by re-executing its own executable as__engine-stdio <engine>, a verb that onlyrocmserves. Embedded asrocm daemonthat works. The standalonerocmdshipped besiderocm(it is in the release packages and is what the e2e suite drives) answers every such request witherror: unrecognized subcommand '__engine-stdio'and exit status 2. So from the standalone daemon the engineStopofrocmd sandbox-tool stop_serveralways came back as an error, and every healthcheck failed; for an engine whose stop decides confirmation (aim) the stop could never be confirmed. One commit fixes that on its own.rocm'sengine_stdio_child_requestbody: spawn, write the request, collect stderr, read stdout, wait, parse (apps/rocm/src/engine_ipc.rs)run_engine_stdioincrates/rocm-engine-protocol/src/stdio_client.rs;rocmkeeps a thinengine_stdio_child_requestthat picks the optionsrocmd'sengine_requestbody (apps/rocmd/src/engine_ipc.rs)run_engine_stdio;rocmdkeepsengine_request/engine_request_viaand picks its optionsdecode_engine_response(rocm) and the inline decode (rocmd)decode_engine_responseinstdio_client.rsengine_request_envelopeinstdio_client.rswith_sigpipe_ignoredand its re-exec test inapps/rocm/src/main.rsrocm_core::with_sigpipe_ignoredin the newcrates/rocm-core/src/sigpipe.rs, test with the same nameROCM_CLI_*_DIRpairs listed inrocm'sapp_path_env_varsand inrocmd'srun_rocm_capture_for_pathsAppPaths::env_varsinrocm-core; both call it. TheROCM_CLI_ENGINE_ENVS_ROOTname isrocm_core::ENGINE_ENVS_ROOT_ENVoptional_arginapps/rocmd/src/common.rsandapps/rocm/src/engine_runtime_select.rsrocm_engine_protocol::optional_arg, besidegpu_indices_to_csvFakeEngineincrates/rocm-engine-protocol/src/fake_engine.rsbehind atest-supportfeature, planted withrocm_core::fake_executableThe shared client lives in
rocm-engine-protocolrather thanrocm-corebecause it needs the envelope types androcm-engine-protocolalready depends onrocm-core;cargo xtask check-crate-edgesis unchanged and green. The crate gains a dependency onanyhow(already in the workspace) and thetest-supportfeature.Commits, in order: the first only pins behaviour: it adds
FakeEngine, splits each caller's out-of-process half into a function that takes the executable (engine_stdio_child_request,engine_request_via) so a test can aim it at the fake, and adds tests that pass on the unmodified code, including the SIGPIPE case in a re-executed child and the divergences below. The second is the shared client; read its test changes as the diff of what the first commit pinned. The third makes the standalonerocmdrun therocmbeside it and is independent of the second. Two review fixes follow: the forwarding writes of the child's stderr are now guarded against SIGPIPE too (andwith_sigpipe_ignoredis safe to overlap, restoring the disposition in force before the first region when the last one ends, so two regions in different threads cannot restore "ignored" over a default action), andFakeEnginequotes the response and scratch directory it puts in shell literals, so a response containing an apostrophe or aTMPDIRcontaining one no longer breaks the fake.Where the two copies differed, the client takes the behaviour that reports more or fails more safely. Every user-visible difference:
rocmalready did this;rocmdthrew the answer away for the status. In practicerocm __engine-stdioalways exits 0, so this only matters for a real crash.failed to parse engine response envelope; stderr: ...), and the decode errors userocm's wording.rocmdusedfailed to parse engine response,engine response did not include dataandfailed to deserialize engine response data; those strings appear only in the daemon's own errors, and nothing in the repository matched on them.rocmhad the guard on the request write only;rocmdhad none. Rust starts a process with SIGPIPE ignored, so the standalonerocmdwas never exposed, butrocm daemonruns the daemon insiderocm, which resets SIGPIPE to its default action inmain. There, a write to an engine child that had already exited killed the whole daemon before it could say why, and forwarding progress to a closed stderr ended the CLI. Both are now ordinary errors.rocmdnow setsROCM_CLI_CONFIG_DIR,ROCM_CLI_DATA_DIRandROCM_CLI_CACHE_DIRon the engine child from its ownAppPaths, asrocmdid. Before, it relied on inheriting them. No change in production, where the two agree; it makes the child act on the roots the daemon was given.failed to spawn engine stdio process <path>);rocm's used to omit the path.read_responsestep that kills and waits on the child before it returns the error. The stderr collector is not joined on that path, because a grandchild holding the pipe could block the join; it ends with the pipe.a_failed_read_of_the_answer_kills_and_reaps_the_child(Linux) reads through a failing reader on a real child and checks that its/procentry is gone.rocmusedeprint!; the client ignores a failed forward and still collects the text).rocmd: when the running executable is exactlyrocmd, engine requests run therocmbeside it (rocm_core::sibling_binary_path), and the error says so if there is none. Embedded inrocm, a test binary or a renamed copy still use their own executable.What stays an option per caller, deliberately:
rocmforwards the child's stderr to the terminal (that is how an install's progress shows) and setsROCM_ENGINE_PROGRESS_STDERR=1for an install;rocmdonly keeps stderr for the error and asks for no progress.rocmdthreads the service's endpoint key file onto the child when it holds a usable key (the healthcheck's HTTP probe needs it);rocmpasses none, because none of its requests (detect, install, resolve-model, stop) is about a running service's endpoint, sorocmdoes not gain the key. The engine environments root isrocm-only. Neither client has, or gains, a timeout; an engine install legitimately runs for minutes, so adding one is a separate decision. The in-process answer for the built-in engines stays inrocm;rocmdcannot link the engines.Two things found and not changed. In the embedded daemon,
restart_managed_servicere-launchescurrent_exe() supervise ..., butrocmhas nosuperviseverb (rocm superviseanswersunrecognized subcommand), so recovery restarts from the embedded daemon cannot start a supervisor; this is the mirror of the bug fixed here, and outside this change. Androcm'sresolve_engine_binary_pathlooks plugins up throughAppPaths::discover()rather than thepathsit is given.Log tails (ROCMAI-879)
Three places read the tail of a log their own way, and the seek-bounded routine in
rocm-corethat the engine adapters use served none of them: the ComfyUI log view read the whole file, decoded sixteen-bit text and stripped ANSI; the services text module streamed every line and failed on any byte that is not valid UTF-8; and therocmdMCPservice_logstool read the whole file into one string and failed the same way. The visible symptoms were that one stray byte turnedrocm services logs/rocm logsinto a single<failed to read ...>placeholder line (and the MCP tool call into an error), and that a multi-gigabyte log was read in full just to show its last few lines.All three now go through
rocm_core::tail_lines_decoded, which takes aLogDecodeoption:PLAIN(UTF-8, what the engines and the service logs use) orCONSOLE(UTF-16 little-endian detected by a byte order mark or by the zero-byte pattern the old code used, plus ANSI stripping, what the ComfyUI view used). The read is bounded by a byte budget, the window starts on a character boundary (for UTF-16 too), a partial first line is dropped, and invalid text is decoded lossily. The log views share oneLOG_VIEW_MAX_TAIL_READbudget of 4 MiB. The routine lives inrocm-corebecauserocmdmust not depend onrocm. The private readers (decode_log_bytes,looks_like_utf16_le, the ComfyUIstrip_ansi_sequences, and the stream/whole-file bodies in the other two) are deleted; each of the three callers keeps a thin adapter that picks aLogDecode, passes the shared budget and adds its own error label.comfyui.rsread_tail_linesbody,decode_log_bytes,looks_like_utf16_le,strip_ansi_sequencesrocm_core::tail_lines_decodedwithLogDecode::CONSOLE;strip_ansi_sequencesis exported fromrocm-core(the doc note intool_policy.rspoints there now)services_text.rsread_tail_linesbody (streams withBufRead::lines)rocm_core::tail_lines(LogDecode::PLAIN); the function stays as a labelled adapter becausechat.rs,service_launch.rsandserve_summary.rscall itrocmdmcp.rsread_tail_linesbody (read_to_string)rocm_core::tail_lines(LogDecode::PLAIN), joined with newlines as beforerocm_core::tail_linestail_lines_decoded(.., LogDecode::PLAIN); engine callers are untouchedCommits, in order: the first only adds tests that pass on the unmodified code and pin each reader's behaviour (large file, UTF-16 with ANSI, invalid text, missing file, limit 0, fewer lines than the limit, no trailing newline, CRLF); the two readers that read whole files return a line far outside any window there, which is the divergence. The second converges the three callers on the routine, flips the pinned divergences to the new behaviour (the diff of those tests is the clearest list of what changed), adds the routine's own tests and updates
docs/architecture.md. The third adds the scenario for a log with an invalid byte. Two review fixes follow. The UTF-8 branch seeked from the size taken by the earlier stat and then read to the current end of the file, so a log whose writer kept appending was consumed past the byte budget; it now caps the read at that snapshot as the sixteen-bit branch already did. A console log ending in a colour reset after its last newline left an unterminated fragment made only of an escape sequence that was counted as a line, so with a limit of one it evicted the real last line; a final unterminated fragment that strips to nothing is no longer counted when ANSI stripping is on, while a terminated blank line still is. The last fix covers a log whose newest line is itself longer than the 4 MiB bound: the window then holds only a fragment, the read returned no lines, and the views said "The saved log is empty." (rocm comfyui logs) or "Recent output: no output yet" (rocm services logs) for a file that is not empty.rocm-coregainstail_log_decoded, which returns the lines in aLogTailthat also records how many bytes at the start of the file the budget left unread, andLogTail::newest_line_exceeds_the_read_boundtells an empty tail of a log with content from an empty log; the byte bound stays, and the two views name it, derived fromLOG_VIEW_MAX_TAIL_READso the number cannot drift.Where the three copies disagreed, the shared routine's choices win, because it is the one that is bounded and the one that never fails a log view. Every difference a user or an MCP client can see:
rocm services logs <id>,rocm logs, the chat tool summaries and the managed-start failure detail showed one<failed to read ...>placeholder line, and the MCPservice_logstool returned an error; they now show the readable lines with U+FFFD in place of the bad bytes. ComfyUI logs were already lossy and are unchanged.failed to read <label> <path>with the causefailed to open <path>(it wasfailed to open <label> <path>); ComfyUI and MCP keep their outerfailed to read <path>and now carryfailed to open <path>as the cause instead of the bare I/O message.service_logstool (review fix), in the daemon's MCP server and in the CLI's own tool bridge (rocm mcp-call service_logs), now says so too, through one shared sentence (rocm_core::service_log_tool_summary): "has no log output yet" for an empty log, "has log output, but none is shown: the newest line is longer than the 4 MiB a log view reads" for an over-long newest line, and "Read the last N line(s)" otherwise, where N is the number of lines returned (it was the number asked for, so an empty log read "Read the last 80 line(s)"). Their structured reply gainsnewest_line_exceeds_read_bound, a missing log is still an error, and the notice clause moved from the CLI intorocm-coreso the daemon words it the same way. The launch-failure reader prints no claim about emptiness and is unchanged.Two judgement calls worth a look: the UTF-16 window is rounded up to an even offset and a byte-order mark is not mistaken for the end of a previous line (both pinned by the every-budget test), and
strip_ansi_sequencesis now public inrocm-corebecause a doc note intool_policy.rsalready named it as the tool for assistant prose.Verification
On the branch after the review fixes and the last merge of
main(tip806c907cd), I ran, in a private target directory:cargo fmt --all --check: clean.cargo clippy --locked -p rocm-engine-protocol -p rocm-core -p rocmd -p rocm --all-targets -- -D warnings(one invocation per crate) andcargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings: clean.x86_64-pc-windows-msvcover those four crates ande2e-cucumber(--lib --bins --tests), with a stub C toolchain (the build scripts ofringandaws-lc-sysneed a compile and archive step that produces files, and check and clippy never link them): clean. It compiles the newGetProcessTimescode, thecfg(windows)tests and the Windows scenario's step definitions; it does not execute them.rocm-engine-protocolstdio_client(18 passed),rocm-coreproc_lifecycle::andlog_tail::(63),rocmdmcp::andservice::(35),rocmservices_text::,service_control::,attached_ctrl_c_andchat::(185), and a wider set over the respawn guard, launch, restart, supervisor and recovery tests (rocm-core84,rocmd138,rocm293): all passed.--listof the unit tests of the four crates before the review fixes and after the last merge: no name disappeared except five thatmainmoved or renamed (fourmanaged_service_status::teststests now inservice_launch::tests, andsave_runtime_manifest_replaces_devel_rather_than_merging, nowwrite_runtime_manifest_replaces_devel_rather_than_merging).cargo xtask manifest --check,cargo xtask check-crate-edges(rocmdstill does not depend onrocm),cargo xtask check-architecture-docandcargo xtask lychee: clean;feature_naming(4 passed).e2e-test-hooks(ROCM_CLI_BINARY,ROCM_CLI_ROCMD_BINARY) and the scenario ids named: 19 scenarios, 78 steps, all passed. They are therocm logs --followscenarios ofmain(logs-04 to logs-07), logs-08 to logs-14, and service-stop-01 to -06, -08 and -09 (service-stop-07 is Windows-only).internalused in an identifier (run_internal_mcp_call,stop_internal_managed_service); no ticket tracker, wiki,.corpor.internalhost. No conflict markers. Every commit I added carries a signature and aSigned-off-bytrailer.Not run on the combined branch: the full workspace gate (
cargo test --workspace --all-targets, workspace clippy),cargo xtask coverage,scripts/smoke_local.py, the fullcargo xtask e2e, the GPU lanes, and the tests of the three engine crates (their sources are untouched by the combined diff apart from being callers of the shared tail routine, which has its own tests); CI carries those. The Windows scenario was not run anywhere, see below. The per-change mutation checks and test-count comparisons below are from each change's own branch and were not repeated on the combination.New tests were each shown to fail when the code they cover is broken (targeted edit, red, restored):
MANAGED_STOP_GRACE10 s to 5 smanaged_stop_grace_is_ten_secondstree_kill(identity.pid)callthe_tree_is_killed_before_the_root_is_terminatedAlreadyGonetoForcedafter a tree kill removeda_root_the_tree_kill_already_took_is_reported_forced_not_already_goneonly_windows_has_a_separate_tree_kill_stepselfrocm-core,a_stop_reports_its_own_pid_as_selfanda_stop_of_a_record_naming_only_its_own_pid_confirmsinrocm,a_stop_reports_its_own_pid_as_self_and_skips_itinrocmdMANAGED_STOP_GRACEin either callerthis_module_carries_no_private_copy_of_the_shared_stop_routine, inrocmand inrocmdrocm's confirmed stopa_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing, and scenario service-stop-06rocmd's confirmed stopstop_managed_service_clears_recorded_pids_once_the_processes_are_goneengine_start_ticksout of the clearingrocm-coretest and therocmtest aboverocm-coreguard tests, threerocmcaller tests, the twoservice_controlrestart tests, threerocmdtests, and scenario service-stop-08 at its refusal step--require-api-keyrulea_managed_spawn_refuses_an_invalid_key_file_on_a_service_that_requires_oneandthe_key_file_rocm_serve_stores_clears_the_recovery_refusalrocmdsupervisor; the wrongServiceStartat therocmspawn, therocmrestart or theserver-recoversiterocm's androcmd's, each a re-executed process killed by SIGPIPE)rocmd's options, fliprocm'sforward_stderr, drop the environments-root variablerocmdwith a private copy of its old clientrocmdtestsrocm_executable_fromignore the standalone caseengine_stopstep added to service-stop-02 (it fails against the shared client without the standalone fix with theunrecognized subcommand '__engine-stdio'error)PLAINa_failed_read_of_the_answer_kills_and_reaps_the_child(without the kill it also takes the 60 s of the child's ownsleepto fail)service_log_tool_summaryrocm-coresummary test, therocmdservice_logstest and scenario logs-13FILETIMEa_filetime_is_its_high_word_above_its_low_wordno_tree_kill_is_run_for_a_pid_that_is_gone_recycled_or_unverifiablethe_process_check_command_is_the_one_the_host_hasmain)--port <port>from the public-bind textthe_command_a_respawn_refusal_names_clears_the_refusal, which now also appliesrocm serve's port rulewhat_a_stop_does_for_each_recorded_and_live_tokenanda_recorded_token_refutes_a_live_one_only_when_both_are_known_and_differ, one row eachoverlapping_guards_restore_the_original_disposition_once, five of five runs, now that the interleaving is fixedonly_a_process_that_is_still_the_recorded_one_is_endedThe stop routine is tested in
rocm-core(proc_lifecycle::tests): visiting order and per-PID start-time pairing, an aliased PID, zero PIDs, the stopping process's own PID, and the signalled/forced/stopped classification of everyTerminationOutcome. The Windows step runs throughterminate_recorded_pid_with: the tree kill comes before the termination, runs only for an identity that matches (not for a gone, recycled or indeterminate PID), mapsAlreadyGonetoForced, and passes every other verdict through;only_windows_has_a_separate_tree_kill_stepties that injected path to what production uses. Each caller's own tests drive its stop through the shared routine and assert the report. The source checks are the "a test fails if a copy re-diverges" guard for the constant and the routine; they read their own source file and assemble their needles so they do not match themselves. The log-tail bound itself is shown by a counting reader: a read of an 8 MiB source under a 64 KiB budget touches at most the budget plus one probe byte (plus the 512-byte sniff for UTF-16).User-visible behaviour
Scenarios that cover this change, all with no GPU and no network:
@id:service-stop-yes-stops-a-running-service(service-stop-01),@id:service-stop-confirms-a-stop-and-cleans-up(service-stop-02, Linux, drives the standalonerocmd sandbox-tool stop_serverand now also asserts that the engineStoprequest was answered rather than turned into theengine_stoperror), and service-stop-03 to -05 (Linux, theROCM_E2E_STOP_OUTCOMEseam) existed already; they pass and are unaffected apart from what is said here. They run on the Linux mock lane (blocking, GitHub-hosted).@id:service-stop-cli-confirms-a-stop-and-clears-the-record(service-stop-06, Linux, mock lane) runsrocm services stop --yesagainst a recorded process the scenario starts and checks the persisted state with the report: the process exited, the record reads stopped and names no process, the key file is gone. With the clearing call removed from the CLI's stop it fails on the record step.@id:service-stop-takes-down-the-whole-process-tree-on-windows(service-stop-07,@requires-os:windows) records a PowerShell parent that has started apingchild, with the PID and no start-time token as Windows records it, runsrocm services stop --yes, and asserts that both processes are gone together with the report (Local server stopped,stopped processes: 1) and the record (stopped, no process, key file gone). It fails if thetaskkill /Tstep is removed, because the stop would then end only the recorded process and the child would survive. The Windows scenario was written on Linux and has never been run on Windows. It compiles (the Windows-target clippy over the e2e harness covers its step definitions), the suite's host filter skips it on Linux, and the lane that will run it is the full-suite self-hosted Windows lanee2e-gpu-strix-windows(non-blocking). The tree-kill step is also covered by unit tests that drive it through injected functions on every host, which the requiredwindows-debuglane runs, and by the Windows clippy gate compiling thecfg(windows)arm. service-stop-01 has no@requires-ostag and runs on that Windows lane too, but it plants PID 0 and never reaches the tree kill. I added no scenario for theselfrow in difference 2 of the stop section: it is observable only when the recorded PID is the calling process, which no CLI invocation a user types produces, so it is pinned at the caller level instead.@id:service-stop-restart-refuses-a-public-server-without-its-key(service-stop-08, Linux, mock lane) plants a publicly bound server with no key file and a live recorded process, runsrocm services restart --yes, and asserts the exit status, the refusal text, that the namedrocm serve <model> ...command is accepted by the real parser (model filled in,--helplast so the parse ends in help only if every earlier word was accepted, with a bogus flag as the negative control), and that the recorded process is still running with no stop recorded and no key created. With the public-bind rule dropped it fails at its refusal step. Nothing in the feature files printed these messages before;serve-24andserve-30exercise the separate reuse refusal, which is unchanged, andserve-24is@requires-gpu @requires-engine:lemonadeso it does not run on the mock lane. The daemon's respawn has no scenario because it needs the daemon's recovery tick over a failed service; each of its call sites is covered by a unit test that drives the real call site.@id:logs-service-logs-tool-longer-than-the-read-bound-is-not-called-empty(logs-13) and@id:logs-service-logs-tool-empty-says-no-output-yet(logs-14), mock lane:rocm mcp-call service_logsagainst a server log holding one line longer than the bound and against an empty log. Each checks the reply (text, empty tail,newest_line_exceeds_read_bound) against the file on disk that it describes. With the empty-log check placed before the over-long one, logs-13 fails with "has no log output yet".@id:service-stop-leaves-a-process-that-reused-the-recorded-pid-alone(service-stop-09, every OS, mock lane): records a running bystander process under a start-time token no live process has (the real one plus one on Linux,1on Windows), runsrocm services stop --yes, and asserts the stop is reported, the bystander is still running, the record names no process and the key file is gone. With a recycled PID treated as matching, it fails at the bystander step. The Windows half has not run.@id:logs-invalid-byte-keeps-matching-lines(logs-08, mock lane): an action log with one invalid byte still yields all nine matching lines; it fails with the old services text reader and passes now. logs-09 to logs-12 (one pair per view, mock lane) assert each message together with the log on disk that it describes: for a line longer than the bound, the size and the absence of a newline; for an empty file, zero bytes. logs-01 (logs-search-reports-match-count) and logs-02 (logs-search-absent-term-no-matches) read the CLI action logs through the changed path and pass unchanged; logs-03 does not reach the reader. There is no scenario forrocm comfyui logsbeyond logs-11 and logs-12 (its output is otherwise unchanged apart from the 4 MiB window and the per-line ANSI edge, covered by unit tests through the adapter) Theservice_logstool is covered by logs-13 and logs-14 below for the CLI's copy and by arocmdunit test for the daemon's copy (the harness drives only therocmbinary).engine_stopassertion in service-stop-02, which is its user-visible effect. Install, serve and engines scenarios go through the samerocmcode and are unchanged apart from the spawn error now naming the program; the ones that start an engine child run on the GPU or lifecycle lanes.Both service-stop-06 and the endpoint-key restart scenario were added with the number 06 on their own lines of work; on this branch they are numbered 06 (stop) and 08 (restart), the Windows stop keeps 07, the review fix adds 09, and
feature_namingpasses.main'srocm logs --followscenarios hold logs-04 to logs-07, so this branch's log scenarios are logs-08 to logs-14. The documented command to run the Linux stop scenarios on a Linux host is-n "service-stop-0[1-689]"; a name filter bypasses the suite's host filter, so-n "service-stop"would also run the Windows-only scenario and fail for want of PowerShell.Notes for reviewers
Read each section's commits in order as described above; the history is the four lines of work, the merge commits that join them, and then the review fixes, so the diff against
mainis large but each consolidation can be reviewed alone. The review fixes (after the first review round) are four commits plus the merges ofmainthey needed:5d9a5704e(the engine child is killed and reaped when its answer cannot be read),439899851and its lint follow-up8ff2ff9bf(theservice_logstool tells an empty log from an over-long newest line),5f9baadd8(a Windows start-time token, so a recycled PID is refused before the tree kill, and the check command that outcome names), then the merges852acf359,106af9224,2f907097cand9d166667c, and after a Windows lane runef877b78b(the stop's identity verdict decided in one place, with a table over no token, a matching token, a differing token and an unreadable one),2d1960939(the overlapping SIGPIPE guard test now enters its regions in a fixed order) and806c907cd(scenario teardown ends the child a stop scenario recorded).106af9224adapts this branch tomain's port-lease work:mainnow requires an explicit--portwith any host but the default loopback, so the public-bind remediation names--port <port>, and the test that runs the named command throughrocm serve's own policy applies that port rule too (it fails with the port left out).docs/architecture.mdanddocs/testing.mdare the files most likely to conflict with other open work, andcrates/rocm-core/src/lib.rsis touched by all four. The merged result has no conflict withmainas of the last fetch (git merge-treeexits 0). After the last merge I ran the fullcargo test -p rocm --bin rocm(1714 passed, 9 ignored),cargo test -p rocmd(220 passed, 2 ignored) andcargo test -p rocm-engine-protocol(44 passed); the Windows clippy and the e2e scenarios listed under Verification were last run on the previous merge tip2f907097c.One note on files touched by more than one section:
apps/rocmd/src/engine_ipc.rslost its own copy of the respawn guard in the guard section and gained the shared-client call in the engine-client section, and the module doc entry indocs/architecture.mddescribes both. Therocmdmodule's key-file helperusable_endpoint_key_filestays there; only the guard moved.