[mbeczynski]

Problem

In the SMB layout every health check asked the machine whether its units are active (systemctl is-active), whether rclone's rc answers, and where tmutil points — all through limactl shell, i.e. ssh over Lima's own user network. Nothing read through the path Time Machine actually uses (Mac →vzNAT→ smbd → tm-bands-bind → punchfs FUSE → rclone FUSE → Drive), and nothing talked SMB from the Mac.

So these states showed "Ready" / green in the panel and a clean vm-status for up to 3 h, until the age of the last backup crossed the threshold:

• punchfs or the rclone mount hung (the processes are alive, so active; rc still answers);
• punchfs holding the dead mount of an rclone that systemd restarted under it (Restart=always restarts rclone alone);
• vzNAT between the Mac and the machine cut (Lima's ssh goes another way);
• smbd active but not answering;
• tm-bands-bind inactive after vm-prepare-bundle — servicesHealthy passed it as fine, and the bands then land on the machine's local disk instead of Drive.

The Mac does not keep the share mounted (Time Machine mounts it per backup), so there is no "remount" on the Mac side; this PR is detection only.

What changes (mac-app only — nothing in vm/)

1. ShareReadProbe — part of the existing status script (limactl shell + sudo bash; only head, sed, ls, sh, all in the Ubuntu 24.04 image, nothing installed). Reads 1 byte, read-only (head -c1) of /srv/tm-local/<image>/bands/0 — through the bind, punchfs and rclone once the image is prepared. Band 0 is read on every mount, so it is in rclone's cache: a local read, not a download.
   • The read runs detached; the script waits at most 10 s and reports hung <age>. A lock dir in /run keeps it to one reader (a stuck reader is reported at once by every later run, not multiplied per panel refresh).
   • The reader ignores SIGHUP/INT/TERM (Ubuntu's sudo use_pty hangs up the script's process group when bash exits or limactl is killed) and removes its own lock; a lock whose reader is gone (pid not alive, or no pid and older than 60 s) is taken over.
   • Verdicts: readable / no image yet / dead (EIO, ENOTCONN, ENXIO, ENODEV, ESTALE, ENOENT on a prepared image) / failed (other errors) / no answer (hung, or no verdict line at all).
   • Before vm-prepare-bundle an image directory without band 0 yet (Time Machine creating it) is "no image yet", not dead.
2. SMBProbe — an SMB2 NEGOTIATE from the Mac to <machine address>:445 (a TCP connect alone would be answered by the kernel even with smbd hung). Non-blocking socket + poll with a 5 s deadline, off the main actor, EINTR retried. Never logs in. Verdicts: answers / refused / not SMB / no answer / unreachable (EHOSTUNREACH/ENETUNREACH) / no address. The panel reuses an answer for 60 s (smbd forks a child per connection); a failure is asked again at every refresh.
3. "Unknown" and "dead" are never green or "Ready" — BackupHealth (backup-health alarm), vm-status (two new lines, exit code), the menu-bar headline/badge/tone and the machine card in the window ("Services" row was "All running" in green; also fixes "Not running: " with an empty list).
4. servicesHealthy(_:bundlePrepared:) — an inactive tm-bands-bind is fine only before vm-prepare-bundle; prepared → failure; not known → unknown. Also used by vm-update's final check.
5. VMStatus.reportTimeout 60 → 90 s: probe 11 s + two curl -m 20 + limactl start must fit, or a hung rclone turned the whole report into "no answer".
6. NSLocalNetworkUsageDescription in Info.plist (see failure modes).
7. New strings translated into all 21 languages.

Probe state → behaviour

Probe	backup-health	vm-status	panel
read: readable / no image yet	—	Share read: readable	can be green
read: dead (EIO, ENXIO, ENOTCONN, ESTALE, ENODEV, ENOENT)	alarm share-unreadable (from the first second)	unhealthy, reason shown	red, "The machine's share cannot be read"
read: other error (EACCES…)	same alarm	unhealthy	red
read: hung / no verdict	alarm share-read-no-answer (never deferred)	unhealthy, age shown	amber, "Unknown whether the machine's share can be read"
SMB: answers	—	Samba from Mac: answers	can be green
SMB: refused / not SMB	alarm smb-no-connection (deferred in the 20 min startup grace)	unhealthy	red
SMB: no answer within 5 s / no address	alarm smb-no-answer / smb-no-address	unhealthy	amber
SMB: unreachable	alarm smb-unreachable, names Local Network privacy	unhealthy	amber
bind inactive after prepare	machine-service-down	unhealthy	red, "Not running: tm-bands-bind"

The probe alarms are raised only while every service is active; with a service down the existing harder message speaks alone. Nothing is restarted or unmounted by this PR.

Failure modes and limits

• Local Network privacy (macOS 15+): the app and the launchd agent may be refused a connection to 192.168.64.x while the same probe from Terminal answers. That shows as amber "unknown" (unreachable), not red, and the message says where to allow it. Could not be verified on a real Mac in this PR (no running of anything on the owner's Mac); see the checks below.
• A hung read keeps one head process in D state inside the machine (one, not one per run) — it also holds the FUSE mount busy, so a later umount/restart of punchfs may get EBUSY until the read returns or the machine reboots.
• Each panel refresh with a failing SMB verdict opens one SMB connection (every 10 s); answering verdicts are reused for 60 s. smbd's log level = 1 should not log these.
• Two status runs taking over a dead lock at the same instant can start two readers once; both write one result, the verdict stays right.

Tests

• 379 tests, swift-format lint --strict clean.
• Canaries on the real script in bash: real file, missing file, empty target, FIFO (blocks like a hung mount — returns within the limit, one reader), SIGHUP to the script's own process group (reader survives and cleans its lock), dead reader's lock, old lock without pid, three parallel runs share one reader; locate logic on scratch bundle.env/image dirs.
• SMB canaries on 127.0.0.1: answering server, silent server (accepts, never answers → no answer within the limit), nothing listening.
• Mutation canaries (each turned tests red): missing PROBE line read as readable, no single-flight lock, SMB timeout read as an answer, green without the probes, inactive bind fine again, unknown prepare state fine, hung read without an alarm, reader dying of the hangup, dead reader's lock kept, new unprepared image read as dead, no route read as refusal, cache reusing failures, machine card green without the probes, script limit back to 60 s.
• Not covered by tests: the live wiring in BackupHealth.currentReport() and the panel controller (they call limactl/tmutil), EINTR in poll.

Release

mac-app/Sources/** and mac-app/Resources/** are in release.yml's push paths: merging this releases a new patch version (cask in the tap). Nothing in vm/, so the machine template is unchanged and vm-update is not needed.

For the owner after merging

1. brew upgrade --cask renacode/tap/cloudmachine (and restart the app).
2. cloudmachine-agent vm-status — expect Share read: readable and Samba from Mac: <address>:445 - answers.
3. If the panel shows amber "Unknown whether Samba … answers" with EHOSTUNREACH while vm-status from Terminal says "answers": allow CloudMachine in System Settings › Privacy & Security › Local Network (the app should ask once; the launchd agent may not be able to).

Proposed next step (separate PR, needs the owner's decision): self-repair

Not in this PR. A draft was written and tested, then taken out on purpose — it restarts services on the only copy of the backup, possibly during a Time Machine run.

• Where/what: vm-guard (every 60 s) runs the same probe; on a confirmed dead read it queues systemctl restart --no-block punchfs, which through Requires= also restarts tm-bands-bind and smbd and leaves rclone (the upload and its 120 GB cache) alone.
• Conditions: only .dead, never "no answer" (a hung read may be a write still going; a restart under it loses it); only with every service active (a down service is systemd's, and right after boot the bind is not there yet); two dead reads in a row, a minute apart.
• Limits: at most 3 attempts per failure, 60 min apart; a failed restart counts as an attempt; after that it stops, logs "GIVING UP" once and leaves it to backup-health. A readable share resets the count.
• A backup in progress: if the path is dead, that run is already failing; restarting smbd drops Time Machine's session, Time Machine retries at its next run. A run that is merely hung (no answer) is never touched.
• Known limit: a read stuck in D state keeps the FUSE mount busy — the restart's fusermount -uz is lazy, so punchfs comes back, but the stuck reader stays until the machine reboots.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AU1Mhxpy65J75YYTkmt7NF