feat: add integration with vision camera v5 by NorbertKlockiewicz · Pull Request #810 · software-mansion/react-native-executorch

NorbertKlockiewicz · 2026-02-16T09:41:46Z

Description

This PR adds real-time camera frame processing capabilities to React Native ExecuTorch by integrating with VisionCamera v5. It introduces a new VisionModule base class and extends ObjectDetectionModule to support three input methods: image URLs/paths, raw pixel data (PixelData), and live camera frames.

This PR includes:

a dedicated screen for testing object detection + vision camera
new tests for added methods

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Run the computer vision example app and test object detection and object detection (live)

Screenshots

Related issues

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

packages/react-native-executorch/common/rnexecutorch/host_objects/JsiConversions.h

packages/react-native-executorch/common/rnexecutorch/host_objects/ModelHostObject.h

packages/react-native-executorch/common/rnexecutorch/metaprogramming/TypeConcepts.h

packages/react-native-executorch/common/rnexecutorch/tests/integration/ObjectDetectionTest.cpp

msluszniak

LGTM, but I would wait for a review from someone that knows TS more than me ;)

chmjkb · 2026-02-26T12:04:28Z

packages/react-native-executorch/src/errors/ErrorCodes.ts

+  /**
+   * Thrown when a feature or platform is not supported in the current environment.
+   */
+  PlatformNotSupported = 119,


Make sure to add this to error handling docs

chmjkb · 2026-02-26T12:07:27Z

packages/react-native-executorch/src/hooks/computer_vision/useObjectDetection.ts

    model,
    preventLoad: preventLoad,
-  });
+  }) as ObjectDetectionType;


do we need the type assertion here?

chmjkb · 2026-02-26T12:15:45Z

packages/react-native-executorch/src/hooks/useModule.ts

  const [isGenerating, setIsGenerating] = useState(false);
  const [downloadProgress, setDownloadProgress] = useState(0);
  const [moduleInstance] = useState(() => new module());
+  const [runOnFrame, setRunOnFrame] = useState<any>(null);


does this have to be any?

chmjkb · 2026-02-26T12:16:58Z

packages/react-native-executorch/src/hooks/useModule.ts

+        // Use "state trick" to make the worklet serializable for VisionCamera
+        if ('runOnFrame' in moduleInstance) {
+          const worklet = moduleInstance.runOnFrame;
+          if (worklet) {
+            setRunOnFrame(() => worklet);
+          }
+        }


Maybe it would be explaining what the state trick is and why we need it

chmjkb · 2026-02-26T12:46:02Z

packages/react-native-executorch/src/types/common.ts

+export interface PixelData extends Omit<TensorPtr, 'dataPtr' | 'scalarType'> {
+  /**
+   * RGB pixel data as Uint8Array.
+   * Expected format: RGB (3 channels), not RGBA or BGRA.
+   * Size must equal: width * height * 3
+   */
+  dataPtr: Uint8Array;


I think this should be Pick instead

msluszniak · 2026-02-26T13:42:00Z

I think you need to rebase because your changes to auto-generated api is as for the older code (before my commit with packages bump).

packages/react-native-executorch/common/rnexecutorch/models/VisionModel.h

…sionModel.h

NorbertKlockiewicz · 2026-02-27T17:39:20Z

The memory leak we've catched is caused by vision camera and hopefully will be fixed soon. The PR should be now ready to merge, just need an approve from @chmjkb

NorbertKlockiewicz changed the title ~~@nk/vision camera~~ feat: add integration with vision camera v5 Feb 18, 2026

NorbertKlockiewicz force-pushed the @nk/vision-camera branch 2 times, most recently from 96f2c14 to f3e17e2 Compare February 19, 2026 21:03

This comment was marked as resolved.

Sign in to view

NorbertKlockiewicz changed the title ~~feat: add integration with vision camera v5~~ feat!: add integration with vision camera v5 Feb 23, 2026

NorbertKlockiewicz changed the title ~~feat!: add integration with vision camera v5~~ feat: add integration with vision camera v5 Feb 23, 2026

NorbertKlockiewicz marked this pull request as ready for review February 23, 2026 18:41

NorbertKlockiewicz self-assigned this Feb 23, 2026

NorbertKlockiewicz linked an issue Feb 23, 2026 that may be closed by this pull request

Vision Camera integration - initial API (Object Detection) #828

Open

msluszniak reviewed Feb 23, 2026

View reviewed changes

NorbertKlockiewicz requested review from chmjkb, mkopcins and msluszniak February 24, 2026 08:06

This comment was marked as resolved.

Sign in to view

msluszniak added the feature PRs that implement a new feature label Feb 24, 2026

This comment was marked as resolved.

Sign in to view

NorbertKlockiewicz force-pushed the @nk/vision-camera branch 3 times, most recently from c1ff490 to 983242e Compare February 25, 2026 13:09

msluszniak approved these changes Feb 25, 2026

View reviewed changes

chmjkb reviewed Feb 26, 2026

View reviewed changes

chmjkb requested changes Feb 26, 2026

View reviewed changes

NorbertKlockiewicz force-pushed the @nk/vision-camera branch from 74be0c6 to 28a7bad Compare February 26, 2026 13:29

NorbertKlockiewicz requested a review from chmjkb February 26, 2026 13:36

NorbertKlockiewicz added 4 commits February 27, 2026 14:04

fix: correct frame data extraction

5d94538

feat: frame extractor for zero-copy approach

303f2fb

chore: num minSdkVersion to 26

ebcddbb

feat: unify frame extraction and preprocessing

491df80

NorbertKlockiewicz and others added 21 commits February 27, 2026 14:04

feat: remove unused bindJSIMethods

9145370

feat: initial version of vision model API

e9c166c

refactor: errors, logs, unnecessary comments, use existing TensorPtr

9e0a824

fix: change Frame import in BaseModule

333847f

feat: use TensorPtrish type for Pixel data input

d00cd85

refactor: add or remove empty lines

972d25c

fix: errors after rebase

c0f0347

fix: remove redundant preprocessing step

0c26a5c

refactor: changes suggested in review

c09c653

fix: not existing error type, add comments to JSI code

6491f24

feat: add new PlatformNotSupported error

658c3e9

fix: compilation JSI error

f68318f

feat: add tests for generateFromPixels method

6b89da3

feat: add example screen with vision camera to computer vision app

e3d5c8a

feat: suggested changes / improve comments

87adfcf

fix(android): object detection not working on android

a8bf2be

docs: add correct api references

bff75b5

fix: requested changes

a1e8e39

docs: update api reference

6cd5519

fix: application doesn't start

732441e

chore: clear apps metro configs and make them run smoothly

6ee9f4d

msluszniak force-pushed the @nk/vision-camera branch from 02bb7aa to 6ee9f4d Compare February 27, 2026 13:19

msluszniak reviewed Feb 27, 2026

View reviewed changes

packages/react-native-executorch/common/rnexecutorch/models/VisionModel.h Outdated Show resolved Hide resolved

Update packages/react-native-executorch/common/rnexecutorch/models/Vi…

2159eac

…sionModel.h

Conversation

NorbertKlockiewicz commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

msluszniak left a comment

Choose a reason for hiding this comment

Uh oh!

chmjkb Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

chmjkb Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

chmjkb Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

chmjkb Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

chmjkb Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

msluszniak commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

NorbertKlockiewicz commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

NorbertKlockiewicz commented Feb 16, 2026 •

edited

Loading

msluszniak commented Feb 26, 2026 •

edited

Loading