[gareth-morgan-nv]

Description

Draft. Stacked on #1428 (base is gmorgan/RearchBackend), which is stacked on #1426. This branch holds the Argus capture backend and the encoder's DMA-BUF input. There is no Argus SDK, Argus camera or Jetson available to the author, so none of the Argus code has been built or run, and the encoder and output changes are unbuilt too. It is written against the Jetson Multimedia API and the repository's existing Argus example as the reference.

• Argus capture (core/argus_capture.*, built only with -DSENSING_WITH_ARGUS=ON, off by default): one capture session per camera, an Argus buffer stream into a pool of NvBufSurface NV12 DMA-BUFs, a repeating request pinned to the sensor mode's frame rate, and a non-blocking acquire that keeps the newest frame and returns older ones to the camera. Frames carry a lease that returns the buffer to the camera, and keeps the Argus objects alive until it does. The sensor timestamp and capture id are passed through unscaled.
• Encoder DMA-BUF input (JetsonEncoder::submit_dmabuf): the output plane switches to V4L2 DMA-BUF memory for an encoder created for it (one input mode per encoder). The capture buffer's lease is held until the plane returns that input buffer, and released at once for a dropped frame. No CPU or CUDA copy.
• Output: a leased-frame path next to the CUDA snapshot path; DMA-BUF input cannot be combined with --livekit-scale-div above 1.
• Startup: --camera-backend=auto|sipl|argus probes both backends; SENSING_WITH_SIPL=OFF builds an Argus-only rig that needs no SIPL headers or configuration; --argus-mode=K picks the sensor mode.
• Contract: the DMA-BUF is an NvBufSurface allocation whose layout (block-linear) only the hardware encoder reads; the header says so instead of describing pitches.
• Plan: src/plugins/sensing/ARGUS_PLAN.md, with its status and open precondition.

Not done

• Any build or run of the Argus path, and any measurement (latency, resource use).
• A supported per-frame correlation between encoder input and output; units are still paired by submission order.
• Dynamically loaded backend modules.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Testing

• Not built or run on a Jetson. The new and edited files are clang-format-14 clean; the capture factory, output, sensing camera and main.cpp pass a g++ syntax check (with and without SENSING_HAS_SIPL). argus_capture.cpp and jetson_encoder.cpp could not be syntax-checked here (they need the Argus and Multimedia API headers).
• Backend-selection and frame-lease unit tests (from sensing: capture-backend rearchitecture (interface, selection, SIPL adapter) ahead of Argus (draft) #1428) are re-run standalone; the CMake test target itself was not configured.

Checklist

• I have read and understood the contribution guidelines
• I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
• I have made corresponding changes to the documentation
• I have added tests that prove my fix/feature works (or explained why not): the Argus path cannot be tested without Argus; selection and lease tests exist from sensing: capture-backend rearchitecture (interface, selection, SIPL adapter) ahead of Argus (draft) #1428
• I have signed off all my commits (git commit -s) per the DCO

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/IsaacCapture
• Branch: gmorgan/RearchBackend
• File: .coderabbit.yaml
• Commit: 69d7c23
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 48b661fb-013c-404f-8f87-ca3ef5f22ec9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[NVshawnhwang]

Built and run on an AGX Orin — Argus backend works with 3 GMSL cameras

Tested the Argus path on real hardware, since the PR says it had never been built or run:

• Orin: AGX Orin Developer Kit, JetPack 6.2 / L4T R36.4.3 (Ubuntu 22.04), SENSING SHW5G GMSL cameras on three ports (Argus devices 0, 4, 8), nvargus-daemon with enableCamInfiniteTimeout=1.
• Build: -DBUILD_PLUGIN_SENSING=ON -DSENSING_WITH_SIPL=OFF -DSENSING_WITH_ARGUS=ON -DBUILD_VIZ=OFF -DBUILD_EXAMPLES=OFF -DBUILD_TESTING=OFF -DBUILD_PYTHON_BINDINGS=OFF -DCMAKE_CUDA_ARCHITECTURES=87.
• Run: camera_plugin_sensing --camera-backend=argus publishing three tracks to a LiveKit dev server on the PC; CaptureServer --livekit-streams + the egocap backend/frontend from this branch showed all three tiles.

Result: all three cameras capture at the sensor's 60 fps and publish 2560×1984 H.264 at the 30 fps cap (~24–30 fps observed), DMA-BUF straight into NVENC. --list-sensors enumerates the devices and modes correctly.

Fixes needed to get there

Branch nvshawnhwang/argus-build-fixes (one commit on top of this PR's head, DCO signed) — happy to open it as a PR against this branch or for you to cherry-pick:

1. livekit_video_output.cpp does not compile: FrameMeta has no stamp_us member (line 415). This comes from sensing: capture-backend rearchitecture (interface, selection, SIPL adapter) ahead of Argus (draft) #1428 and the PR description's g++ syntax check must not have covered this file. Fix: carry monotonic_us through track_in_flight and use it as the transport timestamp.
2. CMake ordering: find_path(ARGUS_INCLUDE_DIR … HINTS "${JETSON_MMAPI_DIR}/argus/include") runs before JETSON_MMAPI_DIR is discovered, so SENSING_WITH_ARGUS=ON always fails on a fresh configure unless -DJETSON_MMAPI_DIR is passed. Moved the MMAPI discovery ahead of the Argus block.
3. Link errors: argus_capture.cpp uses NvBufSurf::NvAllocate from samples/common/classes/NvBufSurface.cpp, which is not in JETSON_MMAPI_SOURCES; that file in turn needs libnvbufsurftransform. Added both.
4. LiveKit SDK prebuilt: the hard-coded ubuntu-24.04 tarball needs GLIBC_2.38/GLIBCXX_3.4.32 and does not link on JetPack 6 (Ubuntu 22.04). The release ships a ubuntu-22.04-arm64 build; the CMake now picks the tarball by /etc/os-release (LIVEKIT_SDK_UBUNTU overrides), with both hashes verified.
5. README: build prerequisites the top-level CMake imposes on a stock Jetson (CMake ≥ 3.24 — JetPack ships 3.22 — plus uv and patchelf), and the JetPack 6.2 Argus-only configuration.

Observations not fixed in that branch

• Shutdown hangs if a camera never delivers a frame: with a stream requested on an empty GMSL port the process ignored SIGTERM for 7+ minutes (futex_wait, presumably stopRepeat/waitForIdle on that session) while holding every camera session; needed kill -9 and a daemon restart. A bound on the wait, or skipping waitForIdle for a session that never produced a frame, would help. Note Argus enumerates all 10 GMSL ports whether or not a camera is attached, so --list-sensors cannot tell an empty port from a camera.
• Daemon log flood: nvargus-daemon logs Error Timeout … BufferOutputStreamImpl::acquireBuffer() roughly 300 times a second with three healthy cameras — the non-blocking acquireBuffer(0) poll in acquire_latest is logged as a timeout on every empty call. A short non-zero timeout, or polling less often than the frame period, would stop the journal spam.
• Stray stdout: the stats ticks are interleaved with bare integers 875967048 / 842091854 (the H264 and NV12 V4L2 fourccs), printed by something in the encoder path on every tick.
• Bitrate: the controller reported 1867 kbps target against the 8 Mbps default once three encoders ran (12 000 kbps with one camera), so the picture is soft at 2560×1984. Worth checking what the congestion controller is reacting to on a LAN.
• Recording: EncodedVideoFrame::user_timestamp_us is the publisher's processing time, so the PC-side streams file must declare processing_unix_us, which the egocap backend previews but refuses to record. A sensor-capture-time basis would be needed for recording from this publisher.

Full build/run log excerpts available on request.

[NVdejung]

🤖 Automated first-pass review (Claude, for Dejun) — I'll follow up personally where needed.

Summary — This PR adds an Argus backend that captures into NV12 block-linear NvBufSurfaces and passes their fds straight to the Jetson encoder. It also adds a capture factory, independent SIPL/Argus build switches, and a leased DMA-BUF path through the LiveKit output. The code follows the repository's argus_sender closely and the lease ownership reads correctly. However, the Argus build cannot link as written, and a fresh configure cannot find the Argus headers. I reviewed it as a draft, though GitHub doesn't mark it as one. Paths are under src/plugins/sensing/ unless given in full.

Must fix

1. core/argus_capture.cpp:219 — NvBufSurf::NvAllocate is defined in the Multimedia API sample NvBufSurface.cpp. JETSON_MMAPI_SOURCES (CMakeLists.txt:144) does not compile that file, so camera_plugin_sensing will not link with -DSENSING_WITH_ARGUS=ON. argus_sender builds the same seven MMAPI classes and carries examples/camera_streamer/argus_sender/src/NvBufSurfaceLite.cpp:31 for exactly this reason; the sample file would also pull in libnvbufsurftransform. Fix: call NvBufSurfaceAllocate() directly (libnvbufsurface is already linked) and take the fd from surfaceList[0].bufferDesc.
2. CMakeLists.txt:109 — the Argus find_path uses ${JETSON_MMAPI_DIR}/argus/include as its hint, but JETSON_MMAPI_DIR is only searched for later, at CMakeLists.txt:123. The README documents -DJETSON_MMAPI_DIR as optional. On a fresh configure without it, the hint becomes /argus/include and configure stops with "needs the Argus headers". Fix: move the Argus block below the Multimedia API section.

Suggestions

• core/argus_capture.cpp:359 — Argus stamps monotonic_ns when the poll loop acquires the buffer. SIPL stamps it on its own frame thread when the frame is ready (core/sipl_camera.cpp:923). The output paces on that stamp with 10% slack (core/livekit_video_output.cpp:169, "so poll jitter does not drop a frame that is due"), and the loop polls every half frame. When capture and publication rates are equal (30/30 by default), this will likely pace out a frame each time the poll phase wraps. Check the paced out count on the target. Stamping on arrival keeps the pacing's assumption true, e.g. a per-camera thread on a blocking acquireBuffer.
• core/livekit_video_output.cpp:66 — the check that refuses DMA-BUF input with scale_div > 1 runs after LiveKitSession::connect and publish_video (core/sensing_camera.cpp:77). So Argus with --livekit-scale-div=2 joins the room before it fails. Also check it in the no-network loop at core/sensing_camera.cpp:55.
• core/argus_capture.hpp:28 — nothing ties buffer_count = 10 to kOutputBuffers = 6 (core/jetson_encoder.cpp:28). A lease is released only when its index is reused (core/jetson_encoder.cpp:325), so the encoder pins 6 buffers. Adding pending_dma, the worker's frame and a fresh acquire, the peak is 9 of 10. The comment counts only the encoder's 6 and the fresh acquire. Derive one constant from the other, or static_assert the margin.
• core/argus_capture.cpp:299 — query() throws if any device lacks mode K (core/argus_capture.cpp:152). One camera with fewer modes therefore fails the probe and --list-sensors for every camera; skip or flag such devices instead. The ISensorMode casts here and at line 119 are dereferenced without a null check.
• Comments that are still SIPL-only: core/sensing_types.hpp:21, app/main.cpp:82, app/main.cpp:130, core/sensing_camera.hpp:26 and app/main.cpp:431. The last says capture is "unaffected" by the poll loop, but with Argus the poll loop is what returns buffers to the camera. The class comment at core/livekit_video_output.hpp:51 still describes only the snapshot path.
• app/main.cpp:414 — DISPLAY is unset after the probes, but its comment says it must be gone before the first EGL call. The Argus probe (app/main.cpp:336) now loads libargus before that point. This is likely harmless with the daemon client. For SENSING_ARGUS_SINGLE_PROCESS=ON, moving the unsetenv above the probes removes the doubt.
• Nits:
  • No tests were added. The Argus requested-id matching at app/main.cpp:336 could sit beside select_camera_backend and be covered in test_camera_backend.cpp.
  • The description still lists ARGUS_PLAN.md, which 1b2f214 removed.
  • The long line at app/main.cpp:134 appears to have made clang-format re-indent all of print_usage.
  • If Argus returns no metadata, sequence stays 0, and core/sensing_camera.cpp:119 treats every later frame as a re-read and drops it. A counter fallback avoids this.

Conflicts

• With gmorgan/RearchBackend: the branch is up to date (0 commits behind) and a local 3-way merge finds no conflicting files, so no rebase is needed. It lands after #1428 and #1426.
• #1437 (gareth-morgan-nv, stacked on this branch) — 10 shared files including core/argus_capture.cpp, with no textual conflict. The semantic risk: #1437 aligns camera time. With Argus, monotonic_ns is the poll time, and the capture_tsc user data carries Argus nanoseconds instead of SIPL's counter, with nothing on the wire saying which. #1437 needs a rebase after any stamping change here.
• #1443 (gareth-morgan-nv, stacked above #1437) — 9 shared files, with no textual conflict; it needs a rebase after changes here. #1440 (gareth-morgan-nv) touches only the README and is low risk.
• #998 and #1078 (jiwenc-nv, other bases, conflicts not computed) — both share CMakeLists.txt; #998 also shares core/CMakeLists.txt, app/main.cpp, core/sensing_camera.* and core/jetson_encoder.*. This stack re-indents the SIPL CMake block, makes the SIPL sources conditional and changes the SensingCamera constructor, so textual conflicts are likely. Whichever lands second should rebase.

Change-by-change (16 files)

I read every hunk. Nothing was built or run for this review.

File / area	What changed	Makes sense?
CMakeLists.txt	SIPL lookup behind SENSING_WITH_SIPL (default ON); optional Argus lookup that links the daemon client by default	⚠️ the Argus lookup runs before JETSON_MMAPI_DIR is known (must fix 2)
core/CMakeLists.txt	SIPL sources and yuv_to_rgba.cu made conditional; capture_factory.cpp and argus_capture.cpp added; SENSING_HAS_SIPL / SENSING_HAS_ARGUS defines	⚠️ nothing defines NvBufSurf::NvAllocate (must fix 1)
core/argus_capture.hpp, .cpp	New backend: one session per camera; buffer stream into 10 NV12 block-linear surfaces; repeating request pinned to the mode's minimum frame duration; acquire keeps the newest frame; the lease keeps the backend alive	✅ mirrors argus_sender; ⚠️ poll-time stamp, probe fails on one device
core/camera_capture.hpp	PixelKind and CameraInfo::pixels added; the DMA-BUF contract no longer describes pitches and offsets	✅ as described
core/capture_factory.hpp, .cpp	make_capture() added; SIPL config mapping moved here from sensing_camera.cpp	✅
core/jetson_encoder.hpp, .cpp	EncoderConfig::input; DMA-BUF output plane with a wrapper and a lease per index; shared take_input(); submit_dmabuf(); drain releases a lease and sends an fd -1 EOS	✅ matches argus_sender; ⚠️ leases held until the index is reused
core/livekit_video_output.hpp, .cpp	input added to the config; submit() takes a mutable frame; one-frame pending_dma mailbox, with the replaced frame released outside the lock; encode_dma(); track_in_flight() factored out; no snapshot slots for DMA	✅; ⚠️ scale check runs after the network join; class comment stale
core/sensing_camera.hpp, .cpp	Takes an opened CameraCapture; passes pixels to the output; error message reworded	✅; ⚠️ comments still SIPL-only
core/sensing_types.hpp	argus_sensor_mode added	✅; sensor_id comment stale
app/main.cpp	--argus-mode added; probes behind build flags, and an explicit backend skips the other backend's probe; --list-sensors lists the chosen backend's cameras; uses make_capture(); usage text reflowed	✅; ⚠️ stale usage text, DISPLAY unset too late
README.md	Documents backend selection, the build switches and the Argus caveats	✅

_{Reviewed at e28d3bc · 16 files · 2026-11-06 18:47 PST}