Templates and reference samples for PICO headsets, showing how to build AR/MR vision, tracking, and spatial interaction with PICO Camera Pack passthrough RGB camera access.
English | δΈζ
This project is based on the sample design from the open-source community project QuestCameraKit (v81) and adds PICO-developed samples to showcase PICO Camera Pack capabilities. QuestCameraKit comes from the Meta Passthrough Camera API ecosystem and covers scenarios such as color sampling and QR code tracking, making it an important reference for Quest MR developers. This project migrates those scenes to PICO through
PXR_CamTextureManagerand adds PICO-specific Camera Pack and environment depth samples.
- Adaptation Status Overview
- PICO-Developed Samples
- QuestCameraKit-Based Samples
- Quick Start
- Troubleshooting and Known Issues
- Community Contribution References
- News References
- Acknowledgements
- License
- Contact
Status labels
- β Adapted: Migrated to PICO Camera Pack (
PXR_CamTextureManagerand/or Environment Depth Extension) and runnable on device.- π§© PICO-developed: Built specifically around PICO capabilities.
Device notes:
sparrowandswanare target PICO headset devices. Environment depth capabilities requireswan.
| # | Sample | Capabilities | Status | Devices |
|---|---|---|---|---|
| β | Environment Depth Extension (Preview) | Depth preview + environment depth raycast | π§© PICO-developed | swan |
| β | HandJointsOverlay | Accurate hand skeleton overlay on camera frames, with frame sync and filtering | π§© PICO-developed | sparrow, swan |
| β | PICOFeature | Video Seethrough / Occlusion / Camera Pack Visualizer | π§© PICO-developed | Video Seethrough / Visualizer: sparrow, swan; Occlusion: swan |
| 1 | ColorPicker | 3D-to-2D pixel mapping and color sampling | β Adapted | sparrow (Manual), swan (Depth) |
| 2 | QRCodeTracking | ZXing single/multi-code detection + ray projection | β Adapted | swan |
- Purpose: Lets virtual rays interact with real physical surfaces through PICO environment depth.
- Implementation:
com.bytedance.pico.environmentdepthextensionextends PICO Unity (Swan) environment depth capabilities with depth preview, frame data access, and depth raycasting. - Current status: π§© PICO-developed (Preview, not for production use)
- Supported devices: swan
- Scene file: DepthPreviewRaycast.unity
How to run
- Make sure the
com.bytedance.pico.environmentdepthextensionpackage is installed. It depends oncom.unity.burst. - Add
EnvironmentDepthDataAccessto the XR Rig. It resolves the depth provider inAwakeand broadcasts frame data throughDepthFrameReceived. - Add
EnvironmentDepthRaycaster, which depends onEnvironmentDepthDataAccess, to raycast against environment depth. - Build the
DepthPreviewRaycastscene to aswandevice.
var raycaster = GetComponent<EnvironmentDepthRaycaster>();
var ray = new Ray(transform.position, transform.forward);
if (raycaster.Raycast(ray, out EnvironmentRaycastHit hit, maxDistance: 5f))
{
// hit.Point, hit.Normal, hit.NormalConfidence, and hit.Distance are valid.
}- Purpose: Draws the hand skeleton accurately on top of the PICO passthrough camera image, aligning hands and camera pixels.
- Implementation: This sample uses camera frame gating (
CameraFrameGate) and timestamped history buffering (TimedHandJointHistory) to synchronize hand joint data with camera frames. Z-score filtering (ZScoreJointFilter) suppresses tracking jitter and outliers, while the built-in hand tracking restart policy (HandTrackerRestartPolicy) handles tracking loss. It projects 26 hand joints into the camera image and draws a skeleton overlay on a copy of the frame: joints are solid points, bones are solid lines, and left/right hands use different colors. Each frame outputs the composited texture throughTextureSkeletonPainter.ResultTexture; to record the processed image, readm_painter.ResultTextureafterm_painter.Apply()inHandJointsCameraOverlay.LateUpdate. Editor tests cover time synchronization, history interpolation, and border drawing. - Capabilities: 3D-to-2D pixel mapping, camera frame and hand timestamp synchronization, Z-score outlier filtering, joint history interpolation, and camera image processing.
- Current status: π§© PICO-developed
- Supported devices: sparrow, swan
- Scene file: HandJointsOverlayVisualizer.unity
How to run
- Open the
HandJointsOverlayVisualizerscene. - Make sure the scene is configured with
PXR_CamTextureManager,HandJointProvider, and a display Quad. - Optional Inspector settings:
Smooth Joints: Enables Z-score temporal filtering.Smoothing Window: Filter window size. The default is 8 frames.Z Score Threshold: Outlier threshold. The default is 2 sigma.Show Left Hand/Show Right Hand: Toggles left/right hand display.
- Build and deploy to a PICO device, then place your hands in front of the headset to see the skeleton overlay.
- Purpose: Provides sample scenes for PICO passthrough, environment depth occlusion, and basic Camera Pack features.
- Implementation: This group contains directly runnable scenes such as PICO Video Seethrough, PICO Occlusion, and PICO Camera Pack Visualizer. They verify passthrough rendering, occlusion materials, and basic Camera Pack texture visualization. PICO Occlusion supports only
swandevices. - Current status: π§© PICO-developed
- Supported devices: Video Seethrough / Camera Pack Visualizer support
sparrowandswan; Occlusion supports onlyswan. - Scene files: PICO Video Seethrough.unity Β· PICO Occlusion.unity Β· PICO Camera Pack Visualizer.unity
- Purpose: Maps a 3D point in space to a 2D image pixel and samples its color.
- Implementation: Supports two sampling modes. Environment mode uses the PICO-developed
EnvironmentDepthRaycasterto compute where a ray hits a real surface, then projects that 3D point to RGB texture UVs for color sampling. This mode is based on environment depth and supports onlyswan. Manual mode samples a fixed point in front of the controller and can run on bothsparrowandswan. The sampled color drives the cube material, with optional brightness correction. - Capabilities: 3D-to-2D pixel mapping.
- Current status: β
Adapted, using
PXR_CamTextureManager+EnvironmentDepthRaycaster. - Supported devices:
sparrow(Manual mode),swan(Environment / Depth mode). - Scene file: ColorPicker.unity
How to run
- Open the
ColorPickerscene and select theSamplingModeon the component: Environment or Manual. - Environment mode requires
EnvironmentDepthRaycasterand the Environment Depth Extension package, and supports onlyswan. - Build to a PICO device, aim at a real surface, and press Trigger to sample color.
- Purpose: Detects and tracks QR codes in real time, then projects them into 3D space.
- Implementation: Uses ZXing.Net to detect QR codes. It supports single-code and multi-code modes, and uses PICO
EnvironmentDepthRaycasterfor CenterOnly and PerCorner placement modes. Detection code is controlled by theZXING_ENABLEDscripting define. - Capabilities: ZXing.Net + single/multi-code detection + CenterOnly/PerCorner rays.
- Current status: β
Adapted, using
PXR_CamTextureManager+EnvironmentDepthRaycaster, with a dependency on the Environment Depth Extension package. - Supported devices: swan
- Scene file: QRCodeTracking.unity
How to run
- In the Unity Editor, open PICO βΈ Camera Sample βΈ Configure Dependencies. The panel shows the install status, source, target path, and purpose for optional dependencies.
- Click Import / Configure All Dependencies to import official Unity dependency samples and configure the ZXing library.
- Open the
QRCodeTrackingscene, then choose the detection mode (Single/Multiple) and ray mode (CenterOnly/PerCorner). - Build to a
swandevice and aim at a QR code.
- PICO device: A PICO headset (
sparrow/swan) is required. Environment depth capabilities, including ColorPicker Environment mode, QRCodeTracking, Depth Extension, and Occlusion, requireswan. - Unity: Use
Unity 6000.3.13f1(Unity 6). - Render pipeline: Universal Render Pipeline (
com.unity.render-pipelines.universal@17.3.0). - PICO Unity SDK: Install PICO Unity Integration / XR SDK (
com.bytedance.pico.xr, Swan). - XR Hands:
com.unity.xr.hands@1.8.1, required for hand joint tracking. - Camera Pack: Provided by the PICO SDK through
ByteDance.PICO.CameraPackandPXR_CamTextureManager. - Environment Depth Extension (Preview):
com.bytedance.pico.environmentdepthextension@0.5.0, which depends oncom.unity.burstand PICOswanenvironment depth capability. - Device testing: The passthrough RGB camera is unavailable in the Editor or simulator. Test on a real device.
- Permissions: Grant headset camera permission on first launch.
- Clone the repository:
git clone <this-repo-url> - Configure the PICO Unity SDK: This project references the PICO SDK (
com.bytedance.pico.xr) through a local path in Packages/manifest.json. Make sure the PICO SDK is located at../PICO-Unity-SDK-6.0.0-20260729/XR. - Open the project: Open
Unity-QuestVisionKitwithUnity 6000.3.13f1. - Configure XR: Enable PICO in Project Settings βΈ XR Plug-in Management, then grant camera permission on the device.
- Configure project dependencies: After cloning and opening the project for the first time, run PICO βΈ Camera Sample βΈ Configure Dependencies. The configuration panel shows each optional dependency's install status, source, target path, and purpose. Click Import / Configure All Dependencies to import official dependency samples and install the ZXing library required by QRCodeTracking.
- QRCodeTracking cannot detect QR codes: Run Import / Configure All Dependencies in PICO βΈ Camera Sample βΈ Configure Dependencies, and make sure the ZXing DLL exists and
ZXING_ENABLEDis defined. - Environment depth supports only swan: ColorPicker Environment mode, QRCodeTracking, Depth Extension, and Occlusion all depend on environment depth and are available only on
swan. - Hand skeleton and camera image are not synchronized or appear jittery: Make sure
Smooth Jointsis enabled onHandJointsCameraOverlay, and check thatCameraFrameGateandTimedHandJointHistoryinitialize correctly. If tracking loss continues, inspect logs fromHandTrackerRestartPolicy. - No camera image in the Editor: The passthrough RGB camera is available only on a real device. Build to a PICO headset.
- Camera permission errors: Make sure headset camera permission has been granted. Otherwise,
PXR_CamTextureManagercannot start.
The following references come from the upstream QuestCameraKit ecosystem and are kept as context and inspiration.
- Tutorials
- Environment understanding and reconstruction
- Takashi Yoshinaga: Convert images to colored point clouds
- Alireza Bahremand: Feed Quest passthrough images into MAST3R-SLAM scene point cloud distribution
- γγγ γγ: 3D scanning
- Sander Sneek: Mixed reality voxel demo
- Takashi Yoshinaga: Quest real-time point cloud data
- Bastion: Sobel edge detection + passthrough camera + scan sound effects
- Takashi Yoshinaga: Map color images to point clouds with the depth API and passthrough camera
- Lighting and reflection estimation
- Environment sampling
- QR code tracking
- 2025-03-21 The Mysticle: Quest major update introduction
- 2025-03-18 Road to VR: Meta releases Quest Camera Access for developers
- 2025-03-17 MIXED Reality News: Quest developers get new mixed reality API
- 2025-03-14 UploadVR: Quest Passthrough Camera API released
- Thanks to PICO for providing Camera Pack, environment depth capabilities, and XR Hands support for the PICO-developed samples in this project.
- This project is based on QuestCameraKit by Roberto Coviello (xrdevrob). The original sample designs and scenes come from that project.
- Thanks to Meta for providing the Passthrough Camera API and Passthrough Camera API Samples.
- Thanks to Michael Jahn for the ZXing.Net library used by the QR code tracking sample.
This project is licensed under the MIT License. See the LICENSE file for details. The original QuestCameraKit project is also licensed under MIT; keep the upstream acknowledgements when reusing related samples.
For PICO-related issues, use PICO developer channels. For questions about the upstream QuestCameraKit samples, contact the original author Roberto Coviello (X / LinkedIn).
Happy coding, and keep exploring what PICO Camera Pack can do.



