🤖 Human-to-humanoid skill transfer for coordinated whole-body loco‑manipulation.
Human demonstrations capture rich whole-body motion across diverse scenes without robot teleoperation. EgoHumanoid-V2 transfers these coordinated skills to a humanoid through coarse-to-fine action alignment: kinematic reference correction improves motion targets, and dynamics-aware refinement improves their execution through a frozen whole-body controller.
Visual alignment replaces human arms with rendered robot arms, while training-time image augmentation improves viewpoint robustness. A π₀.₅ vision-language-action policy learns from the aligned demonstrations and executes through Unitree G1 + BrainCo Revo2 + the SONIC whole-body controller. The paper evaluates zero-shot transfer on four real-world tasks without target-task robot demonstrations.
Coming soon: The remaining unchecked items are expected to be released within the next two weeks. Stay tuned!
- Hardware Setup
- Environment Setup
- Data Collection
- Data Alignment
- Model Training
- Training pipeline
- Policy Deployment
- Deployment pipeline
🔴 PICO 4 Ultra
🔵 5× PICO Trackers
🟠 GoPro
|
🟢 Unitree G1
🔵 BrainCo Revo2
|
We use three separate environments: one for robot control; one for collecting human demonstrations; and one for policy training and serving.
🤖 Robot Control
Ubuntu 22.04 · NVIDIA driver 580+.
1. Install dependencies.
sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl build-essential cmake git git-lfs libgl1 libegl1 libegl-mesa0 libglib2.0-0 ffmpeg libyaml-cpp-dev libeigen3-dev libmsgpack-dev libzmq3-dev nlohmann-json3-dev libssl-dev zlib1g-dev
curl -LsSf https://astral.sh/uv/0.11.14/install.sh | sh
source "$HOME/.local/bin/env"
curl -fL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb -o /tmp/cuda-keyring.deb
sudo dpkg -i /tmp/cuda-keyring.deb
sudo apt-get update
sudo apt-get install -y --no-install-recommends cuda-minimal-build-12-42. Install TensorRT and ONNX Runtime. Download TensorRT-10.13.3.9.Linux.x86_64-gnu.cuda-13.0.tar.gz from NVIDIA to the repository root, then run:
sudo tar -xzf TensorRT-10.13.3.9.Linux.x86_64-gnu.cuda-13.0.tar.gz -C /opt
curl -fL https://github.com/microsoft/onnxruntime/releases/download/v1.16.3/onnxruntime-linux-x64-1.16.3.tgz -o /tmp/onnxruntime.tgz
sudo tar -xzf /tmp/onnxruntime.tgz -C /opt3. Install environment. Repeat the exports when opening a new terminal.
export TensorRT_ROOT=/opt/TensorRT-10.13.3.9
export onnxruntime_ROOT=/opt/onnxruntime-linux-x64-1.16.3
export CUDAToolkit_ROOT=/usr/local/cuda-12.4
export LD_LIBRARY_PATH="$TensorRT_ROOT/lib:$onnxruntime_ROOT/lib:$CUDAToolkit_ROOT/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
bash setup/robot/install_host.sh4. Activate environment.
source sonic/.venv/bin/activateARM64 · Ubuntu 20.04 · ROS 2 Foxy · Python 3.8.
1. Install environment.
sudo apt-get update
sudo apt-get install -y python3-dev python3-venv build-essential cmake git python3-colcon-common-extensions ros-foxy-rmw-cyclonedds-cpp ros-foxy-rosidl-generator-dds-idl libyaml-cpp-dev libssl-dev libgl1 libglib2.0-0
bash setup/robot/install_robot.sh2. Activate environment.
source /opt/ros/foxy/setup.bash
source .venv-robot/ros_ws/install/setup.bash
source .venv-robot/bin/activate👤 Human Data Collection
Ubuntu 22.04 · Python 3.10.
1. Install environment.
sudo apt-get update
sudo apt-get install -y ca-certificates python3.10-dev python3.10-venv build-essential cmake git ffmpeg bluez network-manager
bash setup/human/install.sh2. Activate environment.
source .venv-human/bin/activate🧠 Policy Training & Serving
Ubuntu 22.04 · NVIDIA GPU and driver.
1. Install dependencies.
sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl git build-essential libgl1 libglib2.0-0
curl -LsSf https://astral.sh/uv/0.11.14/install.sh | sh
source "$HOME/.local/bin/env"2. Install OpenPI.
bash setup/policy/install.sh3. Activate environment.
source openpi/.venv/bin/activate1. Prepare devices.
Connect the host and PICO headset to the same network. Start the XRoboToolkit PC service on the host and the XRoboToolkit app on the headset, then enable body and hand tracking. Turn on the GoPro, enable Bluetooth pairing and select Video mode; the collector connects it to the host's WiFi network in the next step.
2. Start recording.
eh2-collect-human --task-name "{task_name}" --task "{task_instruction}"Open http://127.0.0.1:8080. Enter the host's WiFi credentials if needed, then click Prepare collection → Start recording → Save episode / Discard episode. Wait for saving before recording again. Episodes are saved in LeRobot format to data/{task_name}/; new recordings append to the dataset.
3. Check and synchronize data.
eh2-sync-human --root "data/{task_name}"Checks the recordings, synchronizes motion and video at 50 Hz, then checks the output. Results are saved to data/{task_name}_synced/, with synchronization and data quality reports in meta/.
We provide one example trajectory for each of the four tasks in examples. The simulation replays below compare these trajectories before (left) and after (right) alignment.
close_laptop.mp4 |
open_fridge.mp4 |
pull_curtain.mp4 |
close_drawer.mp4 |
1. Align motion.
Place the input recording at {task_name}/input.parquet. Set {endpoints} to hands for hand alignment or feet for foot alignment.
eh2-align \
--input "{task_name}/input.parquet" \
--endpoints {endpoints} \
--output "results/{task_name}"Results are saved under results/{task_name}/: data.parquet contains the human SMPL motion and aligned tokens; run.json records per-round metrics and the selected round.
2. Replay motion.
SMPL replay: encode the human motion online to simulate before alignment.
eh2-replay \
--input "{task_name}/data.parquet" \
--mode smpl \
--output "results/{task_name}/before"Token replay: replay the aligned motion tokens to simulate after alignment.
eh2-replay \
--input "{task_name}/data.parquet" \
--mode token \
--output "results/{task_name}/after"Each command saves replay.mp4 in its output directory. Results may vary slightly between runs.
The pipeline segments human arms and wrist trackers with SAM 3.1, fills the removed regions with DiffuEraser, and renders G1 arms from synchronized joint trajectories. The original gloves are retained to preserve hand–object contact; pixels outside the editing region remain unchanged before video encoding. DiffuEraser uses its ProPainter prior.
We provide the same Close Laptop, Open Fridge, and Pull Curtain recordings shown on the project page, reprocessed with this pipeline. Each row shows the human RGB, SAM 3.1 removal mask, inpainted background, and robot-arm rendering. Download the inputs and full-resolution outputs from Vision Alignment Examples.
Close Laptop
| Human RGB | SAM 3.1 Mask | Inpainting | Robot-arm Rendering |
|---|---|---|---|
 |
 |
 |
 |
Open Fridge
| Human RGB | SAM 3.1 Mask | Inpainting | Robot-arm Rendering |
|---|---|---|---|
 |
 |
 |
 |
Pull Curtain
| Human RGB | SAM 3.1 Mask | Inpainting | Robot-arm Rendering |
|---|---|---|---|
 |
 |
 |
 |
1. Set up the environment.
Tested on Linux with a 48 GB NVIDIA GPU. Install uv, FFmpeg, and an NVIDIA driver compatible with CUDA 13.0; allow about 35 GB of disk space. Request access to SAM 3.1, then authenticate with Hugging Face:
uvx --from huggingface_hub hf auth login
bash setup/vision/install.shThe installer creates separate segmentation/rendering and inpainting environments under runtime/vision, downloads pinned model versions, and verifies their checksums.
2. Align a recording.
Extract an example archive to {task_name}/. These examples use the blue gloves and wrist trackers from the capture setup above. Each input contains rgb.mp4 and joints.npz: body_q stores one 29-DOF G1 pose in radians per RGB frame, joint_names specifies the MuJoCo joint order, and fps matches the video. The supplied camera parameters are for these recordings; pass --calibration camera.json for a different camera setup.
runtime/vision/.venv-sam/bin/eh2-visual-align \
--input "{task_name}" \
--runtime "runtime/vision" \
--output "results/{task_name}"An optional segmentation.json beside rgb.mp4 specifies text prompts and zero-based prompt frames for arms, gloves, and wrist trackers. The Open Fridge archive includes its configuration and uses independent forward and backward propagation sessions.
Results include aligned.mp4, segmentation masks, fitted geometry, the inpainted video, and run.json with input hashes, settings, and completed stages. Repeating the command skips completed stages with matching inputs and settings. Model versions and checksums are recorded in models.json; floating-point differences across GPUs may affect the generated pixels.
Coming soon.
Coming soon.
If you use this work, please cite:
@article{chen2026egohumanoidv2,
title={{EgoHumanoid-V2}: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation},
author={Chen, Jin and Jiang, Yiming and Xu, Chongyang and Shi, Modi and Peng, Shijia and Chen, Li and Li, Tianyu and Xu, Mu and Chen, Yilun and Hoi, Steven and Li, Hongyang},
journal={arXiv preprint arXiv:2609.37181},
year={2026}
}We sincerely thank the following projects and teams:
 |
 |
 |
 |
|---|---|---|---|
| VLA models | Whole-body control | PICO tracking | GoPro interface |
Questions, ideas, or interested in collaborating? We'd love to hear from you—feel free to connect with Jin Chen on WeChat!
