Skip to content

sensing: capture-backend rearchitecture (interface, selection, SIPL adapter) ahead of Argus (draft) - #1428

Open
gareth-morgan-nv wants to merge 8 commits into
gmorgan/FastLiveKitfrom
gmorgan/RearchBackend
Open

gareth-morgan-nv wants to merge 8 commits into
gmorgan/FastLiveKitfrom
gmorgan/RearchBackend

Conversation

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor

Description

Draft. Stacked on #1426 (base is gmorgan/FastLiveKit, so this diff is only the capture-backend rearchitecture). The Argus backend itself will be a further branch (gmorgan/Argus) on top of this one. There is no Argus SDK, Argus camera or Jetson available to the author, so none of the Argus path exists yet or has been tested. This PR does the part that does not need that hardware.

  • Capture boundary (core/camera_capture.hpp): a small vendor-neutral CameraCapture interface, a move-only FrameLease, and a tagged pixel type (borrowed CUDA RGBA for SIPL, leased NV12 DMA-BUF reserved for Argus). The release and DMA-BUF contracts are written down in the header.
  • SIPL behind the interface (SiplCapture): a thin adapter with no extra copy; the output still copies the borrowed frame out after its pacing check, as before. SensingCamera and LiveKitVideoOutput take the interface's frame type; other pixel types are rejected until an encoder input exists. Behaviour is intended to be unchanged.
  • Backend selection (--camera-backend=auto|sipl|argus): chosen once at startup from each backend's probe result. An explicit request must match or fail with that backend's own reason (no silent fallback); auto takes the only match and otherwise names the override. Only SIPL is built in, so argus fails with "this build does not include that backend".
  • Encoder: a stopped capture thread is now reported by poll() instead of looking like a healthy encoder; removed an unused per-frame stamp.
  • Unit tests for the selection rules and the frame lease (Catch2, no CUDA/SIPL needed).

Not done (needs an Orin or an Argus camera)

  • The Argus capture module and the encoder's NV12 DMA-BUF input.
  • A supported per-frame correlation between encoder input and output on the Jetson. Units are paired by submission order today, so an encoder-internal drop can shift frame ids; this needs target-header verification before it is fixed.
  • Distinguishing "no input buffer yet" from a terminal driver error on the encoder's output dequeue (depends on the driver's errno contract).
  • Optional dynamically loaded backend modules.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Testing

  • Selection and frame-lease tests: 12 cases pass when built standalone with g++ against a Catch2 checkout. The CMake test target itself was not configured.
  • New and edited C++ files pass clang-format-14; the SIPL adapter, selection and main.cpp pass a g++ syntax check. jetson_encoder.cpp could not be syntax-checked here (needs the Multimedia API headers).
  • Nothing has been built or run on a Jetson since these changes.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix/feature works (or explained why not)
  • I have signed off all my commits (git commit -s) per the DCO

🤖 Generated with Claude Code

@gareth-morgan-nv gareth-morgan-nv self-assigned this Nov 5, 2026
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@gareth-morgan-nv
gareth-morgan-nv marked this pull request as ready for review November 5, 2026 17:23
@ivany-nv ivany-nv added the NVIDIA label Nov 5, 2026
@gareth-morgan-nv
gareth-morgan-nv force-pushed the gmorgan/RearchBackend branch 2 times, most recently from 0288b95 to 5341f7a Compare November 6, 2026 05:42
gareth-morgan-nv and others added 8 commits November 6, 2026 14:37
…rough

A vendor-neutral CameraCapture interface, a move-only frame lease and a
tagged pixel representation (CUDA RGBA for SIPL, NV12 DMA-BUF for Argus).
Header only; nothing uses it yet, so behaviour is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
SiplCapture adapts SiplCamera without copying: frames stay the borrowed
CUDA RGBA views, which the output still copies out before it returns, after
the pacing check as before. SensingCamera and LiveKitVideoOutput now take
the interface's frame type; other pixel types are rejected until an encoder
input exists. Behaviour is unchanged. Syntax-checked here; the Jetson build
and run are not.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…llback

select_camera_backend takes each backend's probe result: an explicit request
must match or fail with its own reason; auto takes the only match and
otherwise names the --camera-backend override. main.cpp parses the flag and
probes SIPL, the only backend built in. Syntax-checked here; Jetson build
not run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…d encoder stamp

Release callbacks must not throw and may run on any thread; a DMA-BUF frame is
producer-complete pitch-linear NV12 on return. FrameMeta no longer carries a
stamp the FIFO pairing does not use.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…rom poll, document the flag

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
… as a dropped frame

Only EAGAIN and EINTR are the ordinary no-buffer result; any other errno is
recorded for poll() to surface. Relies on the Multimedia API plane leaving
errno set by the failed ioctl, unverified here (no Jetson); the first Orin
run shows whether a healthy encoder trips it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…able for a match

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>

This branch was successfully deployed

1 active deployment
dev — 69d7c230 Deployed Nov 7, 2026 by gareth-morgan-nv via publish-wheel #6595
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants