[gareth-morgan-nv]

Summary

Stacked on #1340 (base gmorgan/sensing-mock-backend). CaptureServer can now serve cameras from the SENSING plugin as a new SENSING_COLOR camera type, through the plugin's existing command line: the server launches camera_plugin_sensing, tails each selected sensor's H.264 output file, decodes it on the CPU and sends JPEGs on the existing image path. The same code serves real SENSING hardware and the plugin's mock backend. The server process uses no CUDA.

Requires a build of the SENSING plugin

None of this runs without a camera_plugin_sensing executable:

• No hardware (desktop, NVIDIA GPU): the mock backend from sensing: add a mock backend so the plugin runs without a Jetson #1340, built with -DSENSING_BACKEND=MOCK. Its encoder must write real H.264; the mock rig is src/plugins/sensing/mock_assets/mock_rig.json (a stereo head camera and two wrist cameras).
• Real hardware: the SENSING plugin build for the Jetson SIPL cameras. Not run on hardware yet.

Station configuration points --sensing-config at that executable; the file format is documented on load_sensing_station in capture_server/acquisition/sensing.py.

What changed

• New camera type SENSING_COLOR = 2 (OAK_D_COLOR and the wire format are unchanged); a request for it is checked against the station entry and is never built as a mock OAK.
• Station config file (--sensing-config): one rig (executable, platform config, link masks, optional encoder settings) plus named cameras selected by module name, with sensor_index to pick one sensor of a multi-sensor module.
• Discovery uses the plugin's existing --list-sensors, parsed strictly. expected_backend is a station assertion checked against --mock.
• One owned producer process per session (private runtime directory, bounded log tail, terminate/kill/reap, parent-death signal) and one adapter per stream; degraded status, required/optional policy, stall detection, and no restart within a session.
• Reader: one persistent parser and decoder per sensor; every access unit is decoded in order and the newest decoded frame is JPEG-encoded at the station quality (2 MiB wire ceiling enforced, never resized). Consumed file bytes are reclaimed with a checked fallocate; the undecoded backlog is bounded.
• The camera cap is now four (protocol, backend, frontend, docs) so both eyes of the stereo head camera show as separate streams.
• av (PyAV) is now a declared dependency, imported only when a SENSING stream attaches.

Known limitations

• The two head eyes have no shared capture timestamp in plain H.264 files, so no synchronized-pair claim is made.
• Every frame the producer writes is decoded (not only delivered ones), so CPU use scales with the producer's rate and bitstream. tests/capture_server/bench_sensing_decode.py measures per-frame cost. No Jetson measurement.
• Timestamps are host monotonic time at frame selection; the plugin's capture timestamps are not carried by the H.264 files.

Validation

• SKIP=check-copyright-year pre-commit run --all-files passes.
• tests/capture_server, tests/recording/unit and tests/backend via uv run --extra dev pytest: 1298 passed, 3 skipped (opt-in real-episode and NVENC tests). The reader/adapter tests use libx264 fixtures, so they establish the decoder path, not native integration.
• End to end against the sensing: add a mock backend so the plugin runs without a Jetson #1340 mock build — not run.
• Real SENSING hardware (Jetson) — not run.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/IsaacCapture
• Branch: gmorgan/sensing-mock-backend
• File: .coderabbit.yaml
• Commit: 2548e57
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 05f30f24-f9a5-4e33-b103-6f80032a7a65

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[NVdejung]

🤖 Automated first-pass review (Claude, for Dejun) — I'll follow up personally where needed.

Summary — Adds a SENSING_COLOR camera type. CaptureServer launches camera_plugin_sensing, tails each sensor's H.264 file, decodes it and sends JPEGs. Decoding is PyAV, or since the second commit Jetson NVDEC through a GStreamer subprocess. The camera cap goes from three to four. Producer ownership, shutdown budgeting and the tests are thorough. Two things need fixing: the reader's overload bound will likely kill real 60 fps streams at the producer's default settings, and the protocol reference was not updated.

Must fix

1. acquisition/sensing_h264.py:250 — The overload bound is shorter than the producer's default GOP. _MAX_PENDING_BYTES is 8 MiB, about 1.7 s at the default 40 Mbps (sensing_types.hpp:49). The producer's IDR period defaults to fps×5 = 5 s (jetson_encoder.cpp:146). So when pending passes 8 MiB, it usually holds no IDR+SPS+PPS unit, _queue raises decoder cannot keep up, and the stream is dead. required defaults to true, so the session ends. Two likely triggers:

   • Software decode of the 2560x1984 @ 60 fps sensors (shw5g.json:202). decoder: auto falls back to software, with only a log warning, on a Jetson without /dev/v4l2-nvdec or the GStreamer plugins (e.g. in a container). README.md:94 says the picture "falls further behind", and the comment at sensing_h264.py:58 says it "jumps to a restart point". Neither matches what the code does.
   • Read gaps at startup, with either decoder. Adapters enter one at a time (acquisition.py:71), and polling starts only after all of them have (server.py:867). _drain_file also pulls up to 32 MiB into pending before decoding anything (sensing_h264.py:223). A stream left unread for about 1.7 s, or one that attaches to that much backlog, fails on its next read. That is plausible with four streams that each start a GStreamer pipeline; please verify on the Orin.

   The 64x32 test fixtures never reach the bound. Suggested fix:

   • On overflow with no usable restart point, drop pending and skip input until the next restart point arrives, instead of raising.
   • Have the rig pass a short --gop (about 1 s) unless the station sets one, so a skip stays inside the 5 s stall rule.
   • Correct the README sentence and the comment.

2. Docs contradict the code.

   • capture_server/README.md:163 — the protocol reference still says:

     • at most three cameras (lines 163, 753, 776)
     • stream IDs 1–3 (lines 439, 517, 550, 1030, 1202) and counters 0–3 (line 572)
     • CameraType is only OAK_D_COLOR=1 (lines 331, 442, 600, 794, 1205)

     The description says the cap change covers the docs.

   • sensing.py:13, sensing_h264.py:5 and :17 still describe CPU-only PyAV decoding. Since the NVDEC commit, the default auto decodes through gst-launch-1.0.

   • The PR description is also stale: it still says CPU-only and "no Jetson measurement", and doesn't mention the GStreamer requirement or the decoder key.

Suggestions

• sensing_jpeg.py:43 — an oversize JPEG fails the stream for good (sensing.py:737). The GMSL path's own help text (server.py:1727) says a full-resolution head camera can exceed 2 MiB, and that path offers a configured downscale. Consider a configured output size here, or dropping and counting oversize frames instead of failing. At least give jpeg_quality guidance for 2560x1984.
• server.py:1943 — --sensing-config is silently ignored with --replay or --gmsl-ipc. Every other source-specific flag gets a parser.error; add one here.
• launch.py:334 — the GMSL-feed path serves these same cameras as OAK_D_COLOR, while this path uses SENSING_COLOR. The station YAML type then depends on the transport. Align them or document it.
• sensing_nvdec.py:306 — proc.wait() after kill() has no timeout, although the module promises a close within a second. _fail() also calls it mid-session. Bound it the way SensingRig._close_sync does.
• server.py:1255 — the inspection is cached by mutating the shared ServerConfig, and the factory (launch.py:97) only works if it reads that same object. Consider passing the resolved sensors explicitly.
• backend/egocap_web/app.py:73 — the recording ingress budget assumes two or three cameras at a few hundred KB per frame. Four 5 MP streams at up to 2 MiB per frame would fill 64 MiB in well under a second. Worth checking.
• Nits:
  • bench_sensing_decode.py:26 has a stray # noqa: E402 and writes to a fixed /tmp path.
  • os.strerror(0) returns "Success", so the fallback at sensing_h264.py:527 never fires.
  • station_config.py:40 and server.py:56 import private constants; make them public.

Conflicts

• With gmorgan/sensing-mock-backend: 0 commits behind and no textual conflicts, so no rebase is needed. This PR is stacked on #1340; retarget it once #1340 lands.

• With open PRs (same author, different bases, so conflicts were not computed):

  • #1426 shares protocol.py, server.py, client.py, captureProtocol.ts, pyproject.toml and uv.lock. Expect lock-file conflicts, and check it doesn't add its own CameraType value 2 or assume three cameras.
  • #1437 changes camera timing in protocol.py and captureProtocol.ts; reconcile it with this adapter's host-time-at-selection timestamps.
  • #1408 and #1409 overlap in server.py, launch.py, client.py and README.md.

  Whichever PR lands second rebases.

Change-by-change (27 files)

File / area	What changed	Makes sense?
backend/egocap_web/capture_preview.py	Cap 3→4 in docstring and constant (the constant is unused).	✅
backend/egocap_web/station_config.py	Cap reuses the protocol constant; YAML type sensing_camera → SENSING_COLOR.	✅ (private-import nit)
capture_server/README.md	New SENSING section.	⚠️ protocol tables unchanged; software-decode sentence is wrong
acquisition/__init__.py	Exports SensingAdapter and SensingRig.	✅
acquisition/acquisition.py	Owns the optional rig and starts it before cameras. Close runs independent closes plus an adapters→rig chain inside one 5 s budget, with every task observed.	✅ chain is sized to the whole budget (about 0.5 s slack)
acquisition/sensing.py	Strict YAML loader; --list-sensors parser (matches main.cpp); sensor resolution; producer ownership (private tmpfs dir, PDEATHSIG, log tail, terminate/kill); stream adapter (startup, stall, dead-stream reclaim).	✅ / ⚠️ stale module docstring
acquisition/sensing_h264.py	File tailer, persistent parser and decoder, overload policy, hole punching, finish, hardware/software choice.	⚠️ overload bound (must fix 1)
acquisition/sensing_jpeg.py	OpenCV JPEG encoder with a 2 MiB ceiling.	⚠️ oversize frame is terminal
acquisition/sensing_nvdec.py	nvv4l2decoder through gst-launch-1.0: non-blocking input, three-buffer newest-frame output.	✅ / ⚠️ unbounded wait after kill; assumes unpadded I420 (true for these widths) — verify on the Orin
client.py, protocol.py	SENSING_COLOR = 2; stream IDs 1–4; cap 4.	✅ (no version bump; old peers fail to parse, as the tests state)
launch.py	SENSING composition in the mock and real factories; depthai imported only for OAK cameras.	✅
server.py	SENSING station entries; _prepare_sensing (shared cached inspection, gate re-checked after the await); backend/mock check; --sensing-config.	✅ / ⚠️ flag not rejected with --replay or --gmsl-ipc
frontend/README.md, captureProtocol.ts, captureProtocol.test.ts	Camera stream_id ≤ 4; camera type documented.	✅
pyproject.toml, uv.lock	Adds av>=14 (locks 19.0.2).	✅
test_station_config.py, test_protocol.py, test_acquisition_adapters.py, tests/support/capture_server.py	Cap and type updates; close-chain cancellation tests; the factory shares the server's config.	✅
test_sensing_adapter.py, test_sensing_h264.py, test_sensing_nvdec.py, h264_fixtures.py	Config, inspection, rig, adapter, reader and NVDEC wrapper tests, using libx264 fixtures and stand-in decoder processes.	✅ (read more lightly; 64x32 fixtures never reach the 8 MiB bound)
bench_sensing_decode.py	Benchmark of per-frame CPU cost.	✅ (nits)

Checked against the plugin at this head:

• --list-sensors output format and --add-stream/--link-masks syntax match.
• BT.601 with --full-range matches.
• No B-frames, and SPS/PPS precede every IDR.
• The mock encoder writes real H.264.

Internal-name scan: no hits, and nothing internal was found in the added docs or fixtures.

_{Reviewed at 35f18ee · 27 files · 2026-11-05 14:41 PST}

[gareth-morgan-nv]

Superseded by #1426