Design: general USB translation layer (Linux usbdevfs over IOKit)

[jotpalch]

Moved from jotpalch/elfuse#8 at @jserv's request.

Motivation

In #310, discussing the serial ioctl work, the maintainer asked for something wider (comment):

I wonder if there is a way to avoid the VM entirely by translating the Linux USB interface into its macOS counterpart. In other words, rather than limiting the approach to USB serial devices, could we provide a more general USB compatibility/translation layer that exposes arbitrary Linux USB device interfaces to macOS applications?

The named scenario is embedded development: let probe-rs flash and debug real hardware from inside elfuse, skipping the VM plus VZUSBPassthroughDevice route entirely. Demand outside that niche is old and unmet: docker/for-mac#900 (serial/USB device access from Linux containers on macOS) has been open since November 2016 and carries 239 upvotes, because a VMM needs hypervisor-level USB forwarding that macOS never offered. elfuse is positioned differently: guest syscalls already execute in a host process that can call IOKit directly, so there is no VMM device model and no guest driver stack to feed.

This issue proposes the layer, shows what a working prototype does on real hardware, and asks for a decision on scope and PR slicing before anything is submitted.

What the hardware allows

Probed on an ESP32-S3 (VID 0x303a, PID 0x1001, a CDC + vendor-JTAG composite) from an unentitled, ad-hoc-signed, non-root process on macOS 15.6:

Interface	Bound macOS driver	USBInterfaceOpen result
0, CDC control (class 0x02)	AppleUSBACMControl	kIOReturnExclusiveAccess, always
1, CDC data (class 0x0a)	AppleUSBACMData + IOSerialBSDClient	succeeds while nobody holds /dev/cu.usbmodem*; ExclusiveAccess once someone does
2, vendor JTAG (class 0xff)	none	succeeds; both bulk pipes usable (0x02 OUT, 0x83 IN, mps 64)

USBDeviceOpen, GetConfigurationDescriptorPtr (no open needed) and ep0 GET_DESCRIPTOR control requests all succeed non-root. Driver detach via USBDeviceReEnumerate(kUSBReEnumerateCaptureDeviceMask) as non-root returns success and does nothing: the registry ID stays the same and the driver stays bound. libusb reaches the same verdict by refusing outright without root or the com.apple.vm.device-access entitlement (darwin_usb.c, darwin_detach_kernel_driver), and usbipd-darwin gates its capture on geteuid() == 0.

Conclusion: vendor/bulk devices are fully usable unprivileged. That covers the mainstream probe-rs transports (CMSIS-DAP v2, ST-Link, J-Link, WCH-Link, pure DFU devices, the ESP32 USB-JTAG interface). Class devices (CDC, HID, Mass Storage) sit behind macOS drivers; the escape hatches are root-mode whole-device capture, or the class interfaces macOS already exposes: CDC through the ttyACM/ttyUSB mapping (piece 3 of #310), HID through an IOHIDManager-backed /dev/hidraw* view. The layer and those side doors complement each other.

Minimum viable surface

Derived from reading what libusb (current git), nusb 0.2.7 (probe-rs' transport), hidapi and dfu-util actually call on Linux:

#	Surface	libusb	nusb	hidapi	dfu-util
1	/dev/bus/usb/BBB/DDD char node: open O_RDWR, read() returns device + all config descriptors, lseek	yes	yes		yes
2	ioctls: SUBMITURB (control/bulk/interrupt), REAPURBNDELAY, DISCARDURB, CLAIM/RELEASEINTERFACE, SETINTERFACE, SETCONFIGURATION, CLEAR_HALT, RESET, DISCONNECT_CLAIM, IOCTL(DISCONNECT/CONNECT), GET_SPEED	yes	yes		yes
3	poll on the fd: POLLOUT when a completed URB is reapable, POLLERR/HUP on disconnect	yes	yes (EPOLLOUT)		yes
4	/sys/bus/usb/devices/<bus>-<port> tree: busnum, devnum, idVendor, idProduct, bcdDevice, speed, string attrs, binary descriptors, interface subdirs; entries must survive realpath; statfs("/sys") returns SYSFS_MAGIC	yes, with usbfs fallback	required; one missing attr hides the device	parent nodes	via libusb
5	GET_CAPABILITIES reporting ZERO_PACKET, BULK_CONTINUATION, NO_PACKET_SIZE_LIM, BULK_SCATTER_GATHER (never MMAP)	yes			yes
6	GETDRIVER: ENODATA when unbound, "usbfs" after claim	yes			yes
7	NETLINK_KOBJECT_UEVENT socket + SO_PASSCRED at init time	hard requirement	watch_devices only		via libusb
8	hidraw: /dev/hidrawN read/write/poll, HIDIOC* ioctls, /sys/class/hidraw tree			yes

Note

Row 7 is the sharpest edge: libusb_init() fails outright when it cannot create that netlink socket (linux_usbfs.c:416-430). elfuse today answers -EAFNOSUPPORT for anything except NETLINK_ROUTE (src/syscall/netlink.c:443-449 at ae46dbf), so every libusb program (dfu-util, pyusb, ...) currently dies inside elfuse before touching any device. Fixing this alone is a self-contained bug fix.

Deliberately deferred: usbfs mmap, streams, isochronous URBs, CONNECTINFO, udev-format netlink group 2, writable bConfigurationValue.

Design sketch

A new src/syscall/usbdev.c implements the usbdevfs ioctl set (semantics from drivers/usb/core/devio.c and include/uapi/linux/usbdevice_fs.h at v7.2) over IOUSBLib, following the mapping libusb's darwin backend already validates in production:

Linux operation	IOKit call	Gap to bridge
open(/dev/bus/usb/B/D)	plug-in for kIOUSBDeviceUserClientTypeID, then USBDeviceOpen	open can fail with ExclusiveAccess; libusb/nusb tolerate that and continue
read() descriptors	GetConfigurationDescriptorPtr (raw bytes, no open)	no raw device-descriptor getter; rebuild from registry properties
control URB / USBDEVFS_CONTROL	DeviceRequestTO / DeviceRequestAsyncTO	usable without open in practice
CLAIMINTERFACE	interface iterator, then USBInterfaceOpen	ExclusiveAccess when a macOS driver holds the interface; map to EBUSY, the errno Linux gives for a driver-bound interface
SETINTERFACE	SetAlternateInterface	pipe table must be rebuilt
SETCONFIGURATION	SetConfiguration	needs open; triggers driver matching
bulk URB	ReadPipeAsyncTO / WritePipeAsyncTO	endpoint address to pipeRef mapping per claimed interface
interrupt URB	ReadPipeAsync / WritePipeAsync	the TO variants reject interrupt pipes, so no host-side timeout
CLEAR_HALT	ClearPipeStallBothEnds	none
DISCARDURB	AbortPipe	aborts the whole pipe, no per-URB cancel; queue one URB per endpoint, report collateral kills as -ECONNRESET
RESET	USBDeviceReEnumerate(0)	tears down the user client; keep busnum/devnum stable and wait for re-attach (libusb allows 10 s)
DISCONNECT_CLAIM	USBDeviceReEnumerate(capture mask)	root only; non-root gets EACCES, and Mass Storage never detaches

Around that core: the /sys/bus/usb/devices and /dev/bus/usb trees are synthesized into the existing scratch-dir mechanism from IOKit registry properties, without opening any device, and statfs("/sys") reports SYSFS_MAGIC. URB data rides bounce buffers: SUBMITURB copies in, IOKit completes into host memory on a single CFRunLoop thread, REAPURB copies out on the vCPU thread, so no host thread ever touches guest memory. ppoll/pselect/epoll report a reapable URB as POLLOUT for this fd type, and disconnect as unmaskable POLLERR|POLLHUP. USB fds are process-local like FD_NETLINK today: after fork the child sees EBADF, the parent keeps working. Errno mapping from kIOReturn codes follows the darwin_usb.c table (kIOUSBPipeStalled to -EPIPE, kIOReturnAborted to -ENOENT after DISCARDURB, and so on).

The hardest semantic gaps and their treatment:

1. Cancel granularity. Linux kills one URB (usb_kill_urb); AbortPipe kills every URB on the pipe. Queueing one URB per endpoint inside elfuse trades throughput for correct DISCARDURB semantics, and libusb/nusb resubmit anything reported -ECONNRESET.
2. RESET. USBDeviceReEnumerate behaves like a replug. The synthetic tree keeps BUSNUM/DEVNUM stable, hides the registry ID change, and mimics libusb's wait for re-attach. A device whose descriptors changed (DFU after a firmware swap) gets -ENODEV, which matches the Linux kernel's usb_reset_device behavior.
3. Timeouts. usbfs URBs carry none (the guest issues DISCARDURB), so only the synchronous control/bulk paths use the TO variants and map to -ETIMEDOUT.
4. ZLP and URB splitting. Advertising NO_PACKET_SIZE_LIM plus SCATTER_GATHER keeps libusb from splitting into 16 KiB URBs, so BULK_CONTINUATION never triggers; ZERO_PACKET is emulated with a trailing zero-length write, as libusb's darwin backend does.

x86_64 guests under Rosetta share the arm64 dispatch: the usbdevfs structs are identical on both LP64 ABIs.

The two projects linked in #310 confirm the boundary from the other side. usb-macos-vm documents that VZUSBPassthroughDevice needs a profile-backed paid entitlement and captures whole devices into a VZ XHCI controller. usbipd-darwin implements root-mode capture with the same ReEnumerate mask the follow-up stage would use, and its capture code is a ready reference for it.

Working prototype

Stages 0 through 3 exist on the fork branch usb-layer, staged as reviewable pieces in jotpalch#4 through jotpalch#7; the upstream PRs will be cut from those. Verified against the ESP32-S3, output condensed:

$ build/elfuse --timeout 0 --sysroot sysroot-debian /usr/bin/lsusb
Bus 002 Device 001: ID 303a:1001 Espressif USB JTAG/serial debug unit
$ build/elfuse --timeout 0 ./usbfs-async 2 1
STEP CLAIMINTERFACE_2 = 0 (0 ok)
  reaped urb=0x7ffec00 (submitted 0x7ffec00) status=0 actual_length=18
STEP DISCARDURB = 0 (0 ok)
  reaped urb=0x7ffec38 (submitted 0x7ffec38) status=-2 (-ENOENT=-2) actual_length=0
second: CLAIMINTERFACE iface2 = -1 (errno 16 Device or resource busy)
Process 15833: 0 leaks for 0 total leaked bytes.

The first command is a stock Debian lsusb binary run unmodified through libusb; lsusb -v prints the full descriptor tree, and a raw-descriptor test confirms the synthetic sysfs descriptors file is byte-identical to what read() on the device node returns, an invariant the Linux kernel guarantees and the tools rely on. The async test claims the vendor interface, reaps an 18-byte control URB, and gets the Linux-correct -ENOENT status for a discarded bulk URB. The EBUSY line is a second elfuse process losing the claim race; the leaks line is leaks --atExit over an open/claim/URB/close loop. A fork test confirms the child gets EBADF while the parent's fd keeps working. make check, tests/driver.sh (77/77) and the elfuse-aarch64 lane of tests/test-matrix.sh (256 passed, 0 failed) are green on the branch.

Proposed PR series

PR	Content	Size	Acceptance
A	accept NETLINK_KOBJECT_UEVENT as a silent socket, SOL_SOCKET options on netlink fds	~350 lines	libusb_init() returns 0 (today -99)
B	synthetic /sys/bus/usb + /dev/bus/usb from the IOKit registry, SYSFS_MAGIC, links IOKit/CoreFoundation	~1100 lines	stock lsusb lists attached devices
C	FD_USBDEV fd type: claim/release, SETINTERFACE, SETCONFIGURATION, CLEAR_HALT, GETDRIVER, GET_*, sync control/bulk	~1300 lines	lsusb -v full tree; concurrent claim gets EBUSY
D	async URBs: SUBMITURB/REAPURB/DISCARDURB, CFRunLoop completion thread, per-endpoint queues, poll/epoll semantics	~1300 lines	nusb-style async path works on hardware
E+	follow-ups, each its own issue: hidraw over IOHID (CMSIS-DAP v1), root-mode capture, hotplug uevents, RESET re-enumeration, dup		per issue

Sizes are measured from the prototype. Each PR stands alone and is testable without the later ones, and the series will be submitted strictly serially: one PR at a time, the next opened only after the previous one merges, never as simultaneous stacked PRs. CI has no USB hardware and IOKit has no loopback, so tree shape, ioctl validation and errno semantics run in the normal test matrix, while transfer tests are gated on an env var naming the expected VID:PID and skip with a reason, following the silent-skip policy from #38.

Non-goals

• Mass Storage raw access: macOS never detaches MSC interfaces, capture mask or not.
• Isochronous URBs: a different frame-scheduling model, and no consumer in scope needs them.
• USB/IP: the protocol terminates in a guest kernel's vhci_hcd, which elfuse does not have.
• VZUSBPassthroughDevice (macOS 27): needs a paid, profile-backed entitlement, assignment is manual per device, and it feeds a Virtualization.framework XHCI controller, so an in-process translator cannot consume it.

Open questions

1. Non-root behavior for class interfaces: should CLAIMINTERFACE return EBUSY (matching Linux when a driver holds the interface), or should the synthetic tree omit interfaces macOS drivers fully own (clearer tool error messages)? I lean EBUSY.
2. Is root-mode capture in scope? It mutates host state (the device leaves macOS until release or replug), which sits oddly with elfuse's process-scoped, no-daemon design. If accepted, elfuse should auto-release on exit.
3. USB fds after fork: EBADF like FD_NETLINK/FD_INOTIFY today (the prototype does this), or a broker handoff? dfu-util and esptool wrappers fork before open, and probe-rs' gdb server does not share handles, so EBADF looks sufficient until a real tool disagrees.
4. Hotplug: start with no events plus a tree rescan on each libusb_get_device_list (libusb semantics allow it), or wire IOKit notifications into synthetic uevents from day one? I lean rescan first.
5. PR B adds -framework IOKit -framework CoreFoundation to the link line, which today carries only Hypervisor. Flagging early in case that needs discussion.

References

• Linux usbdevfs semantics: drivers/usb/core/devio.c, include/uapi/linux/usbdevice_fs.h (v7.2)
• Consumer call sites: libusb linux_usbfs.c, libusb darwin_usb.c (the production-tested usbfs-model-to-IOKit mapping), nusb linux_usbfs
• Prior art on the capture boundary: usb-macos-vm, usbipd-darwin
• External demand: docker/for-mac#900
• Prototype: jotpalch/elfuse usb-layer, staged PRs Accept uevent netlink sockets as silent jotpalch/elfuse#4-Harden signalfd read semantics #7

[jotpalch]

A boundary worth recording, found while exercising the prototype: lsusb -t (tree mode) prints an empty tree and warns about five missing sysfs attributes, while plain lsusb and lsusb -v work.

The cause is structural, and deliberate. Tree mode does not enumerate through libusb; it walks /sys/bus/usb/devices itself and roots the tree at the usbN root-hub entries (usbutils lsusb-t.c). The prototype synthesizes no usbN entries, because none of the consumers this layer targets read them: nusb keeps only [0-9.-] names and drops root hubs (enumeration.rs), and libusb derives topology from the <bus>-<port> names alone, tolerating a missing parent. The skip and its rationale live in a comment in the prototype's usb-sysfs.c.

Observed output:

/sys/bus/usb/devices/2-1/bmAttributes: No such file or directory
/sys/bus/usb/devices/2-1/maxchild: No such file or directory
(three more per device; then no tree, exit 0)

Two different costs hide in there:

• The five attribute files (bmAttributes, configuration, maxchild, rx_lanes, tx_lanes) are static and cheap; PR B should carry them just to keep stderr quiet.
• A working tree is a coherent chunk: usbN entries occupy Device 001 on each bus on Linux, so real devices would renumber to 002+ (touching /dev/bus/usb paths), and the per-interface driver symlinks would have to agree with what GETDRIVER reports. No tool in scope needs any of it today, so I am recording it as a known boundary rather than planned work. If a real workflow turns out to need tree mode, it can become its own issue then.