daemon retains an exclusive serial handle after deploy; next open gets EBUSY and is reported as "serial driver may be wedged"

[zackees]

Summary

fbuild-daemon keeps an exclusive handle on a device's serial port after the operation that opened it. Every subsequent open() on that port fails with EBUSY, and the failure is surfaced as:

attach failed: open_port(/dev/ttyACM2) exceeded 3s; serial driver may be wedged

The driver is not wedged and the device is not wedged. The port is simply held by the daemon.

Proof

Measured on a Linux bench with an RP2350W and an ESP32-C6 attached, during FastLED's two-device --net-peer --ota autoresearch flow.

1. The holder is identified. Scanning /proc/*/fd for the port, while no run is active:

pid=1427887 fbuild-daemon holds: ttyACM2

Note it holds ttyACM2 (the companion) and not ttyACM0 (the primary).

2. The failure is EBUSY, not a driver fault. A plain os.open() bypassing all tooling:

plain open(): OSError: [Errno 16] Device or resource busy: '/dev/ttyACM2'

3. The device is healthy throughout. Still enumerated, same node:

C6 303a:1001 devnum=20 tty=ttyACM2

4. Removing the holder fixes it immediately. Killing the process that holds the fd:

holders: none
plain open(): SUCCESS

That is a complete causal chain: holder present → EBUSY → attach fails; holder gone → open succeeds.

5. A single-device flow on the same port passes. fbuild-backed autoresearch against the C6 alone, which also flashes it and attaches immediately afterwards:

RESULT: RPC smoke PASS (discovery, help, ping, payload, status, drivers, testNoSerial, error handling)

So neither the board, its USB-CDC, nor the port is at fault. The failure needs a second serial interface on the same port.

Why it is always the second device

In the caller, the primary reuses the interface built during its deploy, while the companion constructs a fresh one:

primary = RpcClient(upload_port, serial_interface=serial_iface)              # reused
peer    = RpcClient(peer_upload_port,
                    serial_interface=create_serial_interface(peer_upload_port))  # fresh

A fresh interface means a fresh monitor, which contends with the handle the daemon still holds from that device's deploy. The primary never contends because it never opens a second one. Across every captured occurrence on this bench the failure is on the companion, never the primary.

Retrying makes it worse rather than better, because closing the client nulls its interface and the next attempt constructs another one:

ESP32-C6 did not answer on /dev/ttyACM2 (attempt 1/3: No response with ID 1 within 10.0s); reconnecting
ESP32-C6 did not answer on /dev/ttyACM2 (attempt 2/3: No response with ID 2 within 10.0s); reconnecting
attach failed: open_port(/dev/ttyACM2) exceeded 3s; serial driver may be wedged

Attempts 1–2 reach a session that holds the port but answers nothing; by attempt 3 the port cannot be opened at all. Backing off further (2 s → 4 s → 6 s) does not recover it.

Requests

1. Release the port when an operation completes. A deploy or monitor that has finished should not retain an exclusive fd. Release was observed within ~5 s in one instance and not at all in others, so it is not reliable today.
2. Report EBUSY as contention, naming the holder. serial driver may be wedged sent this investigation to the device and the USB layer for a long time before /proc/*/fd showed a plain lock. Something like port held by fbuild-daemon (pid N) would have made it immediate.
3. Make re-attaching to a port the daemon already owns non-destructive — either hand back the existing session or release and re-acquire cleanly, so a caller opening a second interface does not deadlock against the first.

Context

• fbuild 2.5.23, Linux, two USB-CDC devices on one host.
• Related: deploy/port: accept a USB serial where --port takes a node path (unattended benches renumber) #1428 (this bench also needs serial-number port selection; node paths renumber).
• FastLED-side issues where this surfaced: follow-up: harden RP2350W peer OTA flow from #3863 FastLED#3956 (peer OTA never reaches an OTA step), test(rp2350): complete RP2350(W) loopback driver and subsystem HIL FastLED#3899.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KkufoNxfnNRU9psT3R9F51

[zackees]

Correction and generalisation: the daemon retains whichever port it last used

My original report framed this as the daemon holding the companion's port. That framing was too narrow, and the narrow version would invite too narrow a fix.

A single-device run, where only one port is ever touched:

bash autoresearch rp2350w --rpc-smoke --upload-port /dev/ttyACM0
RESULT: RPC smoke PASS (...)

pid=1620617 fbuild-daemon holds: ttyACM0

The daemon retains ttyACM0 — the only port in play, and the run passed. So the retention is not specific to a companion, a second device, or an ESP32-C6. fbuild-daemon keeps an exclusive fd on the last port it used, always.

Why only the second device ever fails

That single rule explains the whole asymmetry, with no device-specific reasoning:

1. The daemon retains the last port it used.
2. In a two-device flow the companion is deployed last (primary deploy, then companion deploy).
3. The companion's serial interface is then created while the daemon still holds that port → EBUSY → attach failed: open_port(...) exceeded 3s; serial driver may be wedged.
4. The primary's port was released when the daemon moved on to the companion, so the primary attaches cleanly every time.

Across every occurrence on this bench the failure was the companion and never the primary, and I spent a long time looking for something specific to that board — its USB-JTAG CDC, its post-flash reset, its re-enumeration behaviour. None of it was relevant. The board is simply the one that happens to be second.

Falsifiable prediction: reverse the deploy order and the failure follows the order, not the hardware — the RP2350W would fail to attach and the ESP32-C6 would attach cleanly.

Why the single-device case looks fine

It has the same retention; nothing simply tries to reopen that port afterwards, so the leak is invisible. Any workflow that deploys, finishes, and then re-attaches to the same port hits it — which is exactly what a second device makes routine.

Bearing on the request

Request 1 in the original report should be read in this stronger form: release the fd whenever an operation completes, whichever port it was, not merely "release the companion's port". Retention was observed to expire after roughly 5 s once and to persist indefinitely in other runs, so it is not reliably bounded today either way.

Requests 2 and 3 stand as written. Request 2 in particular — report EBUSY as contention and name the holder — would have collapsed this entire investigation to one step; serial driver may be wedged sent it into the device and the USB stack instead.

[zackees]

Correction: fd retention alone does not block opens — my central claim was wrong

I reported that fbuild-daemon retaining a handle is what makes the next open() fail with EBUSY. That is not correct, and I would rather say so than leave a wrong mechanism in an issue someone may act on.

A short single-device run, immediately afterwards:

pid=1679686 fbuild-daemon holds: ttyACM2      <- /proc says it holds the fd
port FREE after 0.0s                          <- plain os.open() succeeds instantly

So the daemon can hold an fd on a port while another process opens it without difficulty. Retention is real, and by itself it is harmless.

What is actually established

Observation	Confirmed
Daemon retains an fd on the last port it used	yes — both single- and two-device runs
Retention alone blocks a subsequent open()	no — measured above, opens in 0.0 s
open() returns EBUSY after a failed two-device attach	yes — reproduced, [Errno 16] from plain os.open()
Killing the holder clears that EBUSY	yes
Single-device run on the same port passes	yes

The EBUSY is real and reproducible, but it correlates with the two-device flow, not with mere fd retention. Something in that path makes the port exclusively locked where the single-device path does not — plausibly two monitors open on the same port at once (the deploy's and the freshly-constructed companion interface's), or an exclusive-mode flag set on one of them. I have not isolated which, and I am not going to guess a third time.

What still stands

• The failure is genuinely EBUSY, and reporting it as serial driver may be wedged is misleading — the device is healthy and enumerated throughout. Request 2 of the original report is unaffected and remains the highest-value fix: name the holder and call it contention.
• It is always the second device, because the companion is deployed last and is the only one whose interface is constructed while fbuild has recently been on that port.
• Killing the holder resolves it immediately.

What I withdraw

Request 1 as I phrased it — "release the fd when an operation completes" — was premised on retention being the cause. Retention may still be worth tidying, but it is not what breaks the attach, and fixing it alone would likely not fix this. The real question is why the two-device path ends up with an exclusive lock on a port that the single-device path leaves freely openable.

Apologies for the churn; the earlier framing was measured too narrowly and I generalised it before testing the single-device case.

[zackees]

Mechanism isolated: the EBUSY is TIOCEXCL, reproduced from first principles

Following the correction above, here is the actual mechanism, derived without guessing.

The tty permits multiple simultaneous openers. Exclusive mode is what produces the failure:

first open:                         OK
second open (no TIOCEXCL):          OK                       <- multiple openers are fine
TIOCEXCL set on first handle
second open (after TIOCEXCL):       Device or resource busy  <- the observed failure, exactly

So attach failed: open_port(...) exceeded 3s; serial driver may be wedged is a second open() against a port on which some component has set TIOCEXCL and not cleared it. That reconciles every observation that previously looked contradictory:

Observation	Explanation
Daemon holds an fd, yet open() succeeds in 0.0 s	fd held without exclusive mode — harmless
open() returns EBUSY after a failed two-device attach	exclusive mode set and still in force
Killing the holder clears it instantly	the exclusive handle dies with the process
Single-device run passes	nothing re-opens the port afterwards, so the lock is never contested
Always the companion, never the primary	the companion is deployed last, so its exclusive handle is the one still live when its interface is constructed

TIOCEXCL is advisory-until-set and process-scoped, which is why it survives as long as the holder lives and vanishes the moment it exits — matching the "sometimes released in ~5 s, sometimes not" behaviour I reported earlier. That was not a timing race; it was whether the holding handle happened to be closed.

Who sets it — stated as a hypothesis, not a finding

I have not isolated the component. Ruling out the obvious candidate: pyserial 3.5 implements its exclusive parameter with flock(LOCK_EX), not TIOCEXCL, and does not lock by default — so pyserial-based tooling is not the likely source.

A plausible candidate is the Rust serial layer: serialport-rs sets TIOCEXCL on Unix when opening. If a monitor or deploy handle is opened through it and kept alive, every later open() on that port gets EBUSY. That is a hypothesis from library behaviour, not something I measured, and you are far better placed to confirm it.

Suggested fix, revised

• Clear exclusive mode (TIOCNXCL) or close the handle when an operation completes. This supersedes my withdrawn Request 1: the problem is not that an fd lingers, it is that a exclusive fd lingers.
• Report EBUSY as contention and name the holder (unchanged, and still the highest-value change). serial driver may be wedged is what sent this investigation through the device, the USB stack, CDC re-enumeration, and two wrong root causes before a five-line TIOCEXCL test settled it.

Reproducer, if useful — no hardware-specific setup, any tty will do:

import os, fcntl, termios
a = os.open(port, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
os.close(os.open(port, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK))  # fine
fcntl.ioctl(a, termios.TIOCEXCL)
os.open(port, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)            # OSError: [Errno 16]

[zackees]

fbuild daemon locks reports the defect directly: a port held for a dead client

I had been reconstructing this from /proc/*/fd and a hand-rolled TIOCEXCL test. fbuild already reports it, and its own output is decisive. Immediately after a successful single-device run (rc=0, RPC smoke PASS):

Port Locks:
  /dev/ttyACM2 [HELD] open=true baud=115200 owner=d6c2eb24-ad47-4cbc-8094-f0459331de75
                     writer=none readers=0 age=5s last_activity=0s rx=18867 tx=481
    client d6c2eb24-ad47-4cbc-8094-f0459331de75 pid=2076451 (dead)
      exe:  .../cpython-3.11.16-linux-x86_64-gnu/bin/python3.11
      cwd:  .../scratchpad/wtretry
      argv: .../python3 -m ci.autoresearch esp32c6 --rpc-smoke --skip-lint \
            --upload-port /dev/ttyACM2 --timeout 4m

The daemon knows the owning client is dead — it prints (dead) — and the port is still [HELD] with open=true, readers=0, writer=none. Concurrently, a plain os.open() on that port returned Device or resource busy.

So the lock outlives the client that took it, and the daemon has the information needed to reclaim it at the moment it renders that line.

It is time-limited, not permanent

Re-checking the same port a couple of minutes later, os.open() succeeded. That resolves the inconsistency in my earlier reports — I described release as "about 5 s once and not at all in other runs", and treated the difference as significant. It was not: the lock is reclaimed eventually, and I was sampling at different points in that window. Apologies for the noise.

The practical consequence stands, because the window overlaps exactly the moment a caller needs the port: a two-device flow deploys the companion, then immediately constructs an interface for it and hits the held lock.

fbuild daemon restart clears it immediately

$ fbuild daemon restart
daemon restarted
$ fbuild daemon locks
Port Locks: (none)
Project Locks: (none)

Useful as a workaround, and it confirms the lock is daemon-held state rather than anything at the device or driver level.

What this changes about the earlier reports

• Withdraw the TIOCEXCL-attribution question. I showed TIOCEXCL reproduces the identical EBUSY and speculated about which library sets it. That is beside the point: the daemon is holding the port open for a dead client, and locks says so. Whether it additionally sets exclusive mode is an implementation detail you can see directly.
• Request 1 is reinstated, precisely. Not "release the fd on completion" (my first phrasing, which I withdrew when retention alone proved harmless), but: reclaim a port lock when its owning client is gone. The (dead) annotation shows the condition is already detected.
• Request 2 is unchanged and still the highest-value fix. attach failed: open_port(...) exceeded 3s; serial driver may be wedged describes a device fault for what locks correctly reports as a stale lock. Pointing that message at fbuild daemon locks — or simply saying "port held by a dead client, run fbuild daemon restart" — would have ended this investigation on day one instead of sending it through the device, the USB stack, and CDC re-enumeration.