Skip to content

About

PICO Camera Pack Sample

Resources

Code of conduct

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

PICO Camera Pack Sample

Templates and reference samples for PICO headsets, showing how to build AR/MR vision, tracking, and spatial interaction with PICO Camera Pack passthrough RGB camera access.

English | δΈ­ζ–‡


This project is based on the sample design from the open-source community project QuestCameraKit (v81) and adds PICO-developed samples to showcase PICO Camera Pack capabilities. QuestCameraKit comes from the Meta Passthrough Camera API ecosystem and covers scenarios such as color sampling and QR code tracking, making it an important reference for Quest MR developers. This project migrates those scenes to PICO through PXR_CamTextureManager and adds PICO-specific Camera Pack and environment depth samples.

Contents

Adaptation Status Overview

Status labels

  • βœ… Adapted: Migrated to PICO Camera Pack (PXR_CamTextureManager and/or Environment Depth Extension) and runnable on device.
  • 🧩 PICO-developed: Built specifically around PICO capabilities.

Device notes: sparrow and swan are target PICO headset devices. Environment depth capabilities require swan.

# Sample Capabilities Status Devices
β€” Environment Depth Extension (Preview) Depth preview + environment depth raycast 🧩 PICO-developed swan
β€” HandJointsOverlay Accurate hand skeleton overlay on camera frames, with frame sync and filtering 🧩 PICO-developed sparrow, swan
β€” PICOFeature Video Seethrough / Occlusion / Camera Pack Visualizer 🧩 PICO-developed Video Seethrough / Visualizer: sparrow, swan; Occlusion: swan
1 ColorPicker 3D-to-2D pixel mapping and color sampling βœ… Adapted sparrow (Manual), swan (Depth)
2 QRCodeTracking ZXing single/multi-code detection + ray projection βœ… Adapted swan

PICO-Developed Samples

🧩 Environment Depth Extension (Preview)

  • Purpose: Lets virtual rays interact with real physical surfaces through PICO environment depth.
  • Implementation: com.bytedance.pico.environmentdepthextension extends PICO Unity (Swan) environment depth capabilities with depth preview, frame data access, and depth raycasting.
  • Current status: 🧩 PICO-developed (Preview, not for production use)
  • Supported devices: swan
  • Scene file: DepthPreviewRaycast.unity
How to run
  • Make sure the com.bytedance.pico.environmentdepthextension package is installed. It depends on com.unity.burst.
  • Add EnvironmentDepthDataAccess to the XR Rig. It resolves the depth provider in Awake and broadcasts frame data through DepthFrameReceived.
  • Add EnvironmentDepthRaycaster, which depends on EnvironmentDepthDataAccess, to raycast against environment depth.
  • Build the DepthPreviewRaycast scene to a swan device.
var raycaster = GetComponent<EnvironmentDepthRaycaster>();
var ray = new Ray(transform.position, transform.forward);
if (raycaster.Raycast(ray, out EnvironmentRaycastHit hit, maxDistance: 5f))
{
    // hit.Point, hit.Normal, hit.NormalConfidence, and hit.Distance are valid.
}

Environment Depth Extension demo

🧩 HandJointsOverlay

  • Purpose: Draws the hand skeleton accurately on top of the PICO passthrough camera image, aligning hands and camera pixels.
  • Implementation: This sample uses camera frame gating (CameraFrameGate) and timestamped history buffering (TimedHandJointHistory) to synchronize hand joint data with camera frames. Z-score filtering (ZScoreJointFilter) suppresses tracking jitter and outliers, while the built-in hand tracking restart policy (HandTrackerRestartPolicy) handles tracking loss. It projects 26 hand joints into the camera image and draws a skeleton overlay on a copy of the frame: joints are solid points, bones are solid lines, and left/right hands use different colors. Each frame outputs the composited texture through TextureSkeletonPainter.ResultTexture; to record the processed image, read m_painter.ResultTexture after m_painter.Apply() in HandJointsCameraOverlay.LateUpdate. Editor tests cover time synchronization, history interpolation, and border drawing.
  • Capabilities: 3D-to-2D pixel mapping, camera frame and hand timestamp synchronization, Z-score outlier filtering, joint history interpolation, and camera image processing.
  • Current status: 🧩 PICO-developed
  • Supported devices: sparrow, swan
  • Scene file: HandJointsOverlayVisualizer.unity
How to run
  • Open the HandJointsOverlayVisualizer scene.
  • Make sure the scene is configured with PXR_CamTextureManager, HandJointProvider, and a display Quad.
  • Optional Inspector settings:
    • Smooth Joints: Enables Z-score temporal filtering.
    • Smoothing Window: Filter window size. The default is 8 frames.
    • Z Score Threshold: Outlier threshold. The default is 2 sigma.
    • Show Left Hand / Show Right Hand: Toggles left/right hand display.
  • Build and deploy to a PICO device, then place your hands in front of the headset to see the skeleton overlay.

HandJointsOverlay demo

🧩 PICOFeature

  • Purpose: Provides sample scenes for PICO passthrough, environment depth occlusion, and basic Camera Pack features.
  • Implementation: This group contains directly runnable scenes such as PICO Video Seethrough, PICO Occlusion, and PICO Camera Pack Visualizer. They verify passthrough rendering, occlusion materials, and basic Camera Pack texture visualization. PICO Occlusion supports only swan devices.
  • Current status: 🧩 PICO-developed
  • Supported devices: Video Seethrough / Camera Pack Visualizer support sparrow and swan; Occlusion supports only swan.
  • Scene files: PICO Video Seethrough.unity Β· PICO Occlusion.unity Β· PICO Camera Pack Visualizer.unity

QuestCameraKit-Based Samples

1. 🎨 ColorPicker β€” βœ… Adapted

  • Purpose: Maps a 3D point in space to a 2D image pixel and samples its color.
  • Implementation: Supports two sampling modes. Environment mode uses the PICO-developed EnvironmentDepthRaycaster to compute where a ray hits a real surface, then projects that 3D point to RGB texture UVs for color sampling. This mode is based on environment depth and supports only swan. Manual mode samples a fixed point in front of the controller and can run on both sparrow and swan. The sampled color drives the cube material, with optional brightness correction.
  • Capabilities: 3D-to-2D pixel mapping.
  • Current status: βœ… Adapted, using PXR_CamTextureManager + EnvironmentDepthRaycaster.
  • Supported devices: sparrow (Manual mode), swan (Environment / Depth mode).
  • Scene file: ColorPicker.unity
How to run
  • Open the ColorPicker scene and select the SamplingMode on the component: Environment or Manual.
  • Environment mode requires EnvironmentDepthRaycaster and the Environment Depth Extension package, and supports only swan.
  • Build to a PICO device, aim at a real surface, and press Trigger to sample color.

ColorPicker Environment mode ColorPicker Manual mode

2. πŸ“± QRCodeTracking β€” βœ… Adapted

  • Purpose: Detects and tracks QR codes in real time, then projects them into 3D space.
  • Implementation: Uses ZXing.Net to detect QR codes. It supports single-code and multi-code modes, and uses PICO EnvironmentDepthRaycaster for CenterOnly and PerCorner placement modes. Detection code is controlled by the ZXING_ENABLED scripting define.
  • Capabilities: ZXing.Net + single/multi-code detection + CenterOnly/PerCorner rays.
  • Current status: βœ… Adapted, using PXR_CamTextureManager + EnvironmentDepthRaycaster, with a dependency on the Environment Depth Extension package.
  • Supported devices: swan
  • Scene file: QRCodeTracking.unity
How to run
  • In the Unity Editor, open PICO β–Έ Camera Sample β–Έ Configure Dependencies. The panel shows the install status, source, target path, and purpose for optional dependencies.
  • Click Import / Configure All Dependencies to import official Unity dependency samples and configure the ZXing library.
  • Open the QRCodeTracking scene, then choose the detection mode (Single/Multiple) and ray mode (CenterOnly/PerCorner).
  • Build to a swan device and aim at a QR code.

QRCodeTracking demo

Quick Start

Requirements

  • PICO device: A PICO headset (sparrow / swan) is required. Environment depth capabilities, including ColorPicker Environment mode, QRCodeTracking, Depth Extension, and Occlusion, require swan.
  • Unity: Use Unity 6000.3.13f1 (Unity 6).
  • Render pipeline: Universal Render Pipeline (com.unity.render-pipelines.universal@17.3.0).
  • PICO Unity SDK: Install PICO Unity Integration / XR SDK (com.bytedance.pico.xr, Swan).
  • XR Hands: com.unity.xr.hands@1.8.1, required for hand joint tracking.
  • Camera Pack: Provided by the PICO SDK through ByteDance.PICO.CameraPack and PXR_CamTextureManager.
  • Environment Depth Extension (Preview): com.bytedance.pico.environmentdepthextension@0.5.0, which depends on com.unity.burst and PICO swan environment depth capability.
  • Device testing: The passthrough RGB camera is unavailable in the Editor or simulator. Test on a real device.
  • Permissions: Grant headset camera permission on first launch.

Installation

  1. Clone the repository:
    git clone <this-repo-url>
    
  2. Configure the PICO Unity SDK: This project references the PICO SDK (com.bytedance.pico.xr) through a local path in Packages/manifest.json. Make sure the PICO SDK is located at ../PICO-Unity-SDK-6.0.0-20260729/XR.
  3. Open the project: Open Unity-QuestVisionKit with Unity 6000.3.13f1.
  4. Configure XR: Enable PICO in Project Settings β–Έ XR Plug-in Management, then grant camera permission on the device.
  5. Configure project dependencies: After cloning and opening the project for the first time, run PICO β–Έ Camera Sample β–Έ Configure Dependencies. The configuration panel shows each optional dependency's install status, source, target path, and purpose. Click Import / Configure All Dependencies to import official dependency samples and install the ZXing library required by QRCodeTracking.

Troubleshooting and Known Issues

  • QRCodeTracking cannot detect QR codes: Run Import / Configure All Dependencies in PICO β–Έ Camera Sample β–Έ Configure Dependencies, and make sure the ZXing DLL exists and ZXING_ENABLED is defined.
  • Environment depth supports only swan: ColorPicker Environment mode, QRCodeTracking, Depth Extension, and Occlusion all depend on environment depth and are available only on swan.
  • Hand skeleton and camera image are not synchronized or appear jittery: Make sure Smooth Joints is enabled on HandJointsCameraOverlay, and check that CameraFrameGate and TimedHandJointHistory initialize correctly. If tracking loss continues, inspect logs from HandTrackerRestartPolicy.
  • No camera image in the Editor: The passthrough RGB camera is available only on a real device. Build to a PICO headset.
  • Camera permission errors: Make sure headset camera permission has been granted. Otherwise, PXR_CamTextureManager cannot start.

Community Contribution References

The following references come from the upstream QuestCameraKit ecosystem and are kept as context and inspiration.

News References

Acknowledgements

  • Thanks to PICO for providing Camera Pack, environment depth capabilities, and XR Hands support for the PICO-developed samples in this project.
  • This project is based on QuestCameraKit by Roberto Coviello (xrdevrob). The original sample designs and scenes come from that project.
  • Thanks to Meta for providing the Passthrough Camera API and Passthrough Camera API Samples.
  • Thanks to Michael Jahn for the ZXing.Net library used by the QR code tracking sample.

License

This project is licensed under the MIT License. See the LICENSE file for details. The original QuestCameraKit project is also licensed under MIT; keep the upstream acknowledgements when reusing related samples.

Contact

For PICO-related issues, use PICO developer channels. For questions about the upstream QuestCameraKit samples, contact the original author Roberto Coviello (X / LinkedIn).


Happy coding, and keep exploring what PICO Camera Pack can do.

About

PICO Camera Pack Sample

Resources

Code of conduct

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages