ipc: half-open outbound connections are never detected — no keepalive/user-timeout, and .ipc.send has no deadline

[vbmithr]

An outbound connection whose peer becomes unreachable without closing the socket — a suspended laptop, a NAT or firewall that silently drops the flow, a peer host that loses power — is never detected. The client blocks in .ipc.send forever, or (for a receive-only subscriber) waits forever for pushes that will never arrive. .ipc.on.close never fires, because nothing ever tells the socket the peer is gone.

This is the one failure mode left after #503. That issue gave outbound connections a close event for every observable teardown: an orderly close, a reset, a socket error, a local .ipc.close. A half-open connection produces none of those, so there is no event to deliver, and no way to synthesize one from Rayfall.

Why it cannot be worked around in Rayfall

Two things combine, and each one on its own would be survivable.

After the handshake, all socket timeouts are explicitly cleared. ray_sock_connect installs the connect budget as SO_RCVTIMEO/SO_SNDTIMEO (src/core/ipc.c:1400), and once the handshake completes src/core/ipc.c:1501-1506 zeroes both back to "block indefinitely" before handing the fd to the poll. Nothing replaces them: SO_KEEPALIVE, TCP_KEEPIDLE and TCP_USER_TIMEOUT appear nowhere in src/ or include/. The connection inherits the OS default, which for a socket with no unacked data in flight is never.

The sync wait has no deadline. ray_ipc_send's loop blocks on ray_sock_wait_readable_intr((ray_sock_t)sel->fd, -1) (src/core/ipc.c:1629). The -1 is infinite. Only sync_ready, a deregistered selector, or a signal breaks it out.

So a client polling a peer with .ipc.send wedges its whole thread inside that wait — and because it is wedged, its own .time.timer.set watchdog cannot run. A Rayfall-level staleness timer is exactly the right design for a push-driven subscriber, and we use one; it is structurally unavailable to anything that polls synchronously. There is no timeout argument on .ipc.send, and no socket option reachable from Rayfall.

Repro

;; terminal 1
./rayforce -p 5000

;; terminal 2
(set h (.ipc.open "127.0.0.1:5000" 1000))
(set .ipc.on.close (fn [h] (println "closed " h)))
(.ipc.send h 42)          ;; => 42, fine

Now make the peer unreachable without closing the socket — suspend the peer host, or drop the flow:

sudo iptables -I INPUT -p tcp --sport 5000 -j DROP

(.ipc.send h 42)          ;; blocks forever; no error, no on.close, no timer runs

Killing the server instead of dropping the flow gives the correct #503 behaviour, which is the useful contrast: the RST arrives, .ipc.send returns io, and on.close fires.

What would fix it

Either would do, and they are complementary rather than alternatives:

1. TCP_USER_TIMEOUT (plus SO_KEEPALIVE) on established connections. TCP_USER_TIMEOUT is the precise knob on Linux: it bounds how long transmitted data may go unacknowledged before the connection is torn down, which covers the .ipc.send case directly, and keepalives cover an idle receive-only subscriber. The teardown surfaces through the existing path — .ipc.send gets its io error and .ipc.on.close fires — so no new semantics and no caller changes. Sensible as a default with an opt-out, since a wedged connection is never what anyone wants.

2. An optional recv deadline on .ipc.send, e.g. (.ipc.send h msg 5000), returning the same io error shape on expiry. This gives the caller per-call control, which matters when one round-trip is a cheap poll and another is a long query the peer is legitimately still computing — a single socket-level timeout cannot tell those apart.

We would take (1) as the default and (2) where a caller knows its own latency budget.

Context

Ours is a Rayfall service reading a depth feed. The push-driven subscriber we can defend with a last_message_ms watchdog timer; the polling replica we cannot defend at all, and a suspended laptop hangs it indefinitely. Happy to test a patch against both.