DexSeedExpanding Human Hand–Object Interactions into Robot Demonstrations for Dexterous Policy Learning
01 / Real-world VLA results
One policy. Eight ways to grasp.
Dexterous pick-and-place, fine-tuned on generated demonstrations without collecting real-robot action demonstrations.
Real-robot executions across all eight grasp modes.
Click any video to enlarge ↗02 / From interaction to demonstration
Keep the grasp. Expand the possibilities.
First, organize human interactions by how the hand grasps. Then follow one selected interaction through retargeting, motion augmentation, physics, and rendering into robot training data.
03 / Explore the generated data
A grasp is more than an object.
Choose a grasp mode, then explore its source interactions, geometry variants, augmented trajectories, physical rollouts, and rendered observations.
04 / Generated demonstrations
Small seeds. A much larger repertoire.
Generated robot demonstrations across objects, geometries, and trajectories.
05 / Physics-based realization
From a motion reference to physical execution.
Four policies follow the same reference arm motion. The learned finger policy must maintain the interaction through grasping, transport, and placement.
FR3 rollout success in the physics-based imitation experiment. ManipTrans, Dexplore, and w/o Arm train with a floating hand; Ours incorporates arm tracking during training.
06 / Simulation VLA results
Learning to act from generated experience.
The fine-tuned vision–language–action policy jointly predicts arm and finger actions, conditioned on the requested grasp.
What makes the generated data effective?
Compare the effect of removing data ingredients, or reducing the amount of generated training data.
What happens when an ingredient is removed?
How much generated data is needed?
Pick-and-place success (%): lift the object and place it in the target region. Each setting has 80 trials (8 grasp modes × 10 trials). Seen and unseen conditions are defined relative to the full training set. Results from Tables IV and VI.
07 / Failure case
When the interaction slips.
The mug handle slips along the fingers during execution, and the placement is not completed.
BibTeX
@misc{pi2026dexseed,
title = {DexSeed: Expanding Human Hand--Object Interactions into Robot Demonstrations for Dexterous Policy Learning},
author = {Huaijin Pi and Zehong Shen and Yong Wang and Kechun Xu and Changpeng Wang and Dingkun Wei and Xueyang Wang and Wenjia Wang and Xing Zhu and Yujun Shen and Taku Komura},
year = {2026},
url = {https://robbyant-research.github.io/DexSeed/}
}