HALO

Human-Aligned Data Collection for WhoLe-body MObile Manipulation

Shuai Yang^1,2,* Fangyi Xu^2,* Yishu Shen¹ Kecheng Zheng² Yinghao Xu^1,†

¹ The Hong Kong University of Science and Technology ² Robbyant

^* Equal contribution ^† Corresponding author (e-mail:justimyhxu@ust.hk)

HALO combines kinesthetic teaching with a low-resistance mobility interface to collect coordinated whole-body demonstrations. Operators guide the robot end-effectors directly and move the collection platform through natural locomotion, with the mobile base removed. The platform preserves the deployed base's nonholonomic motion space. Deployment-aligned demonstration processing calibrates and temporally aligns platform motion with the base response and masks and inpaints demonstrator hands in recorded observations. Compared with Mobile ALOHA and VR teleoperation, HALO enables up to 2.9× faster demonstration collection with comparable or higher success rates. Learned policies perform Zip-Tie Fastening and achieve 73.3% success on a six-stage long-horizon mobile manipulation task.

How HALO Works

Leader–follower systems spatially separate the operator's hands, viewpoint and robot end-effectors, introducing a perception gap; the powered mobile base also constrains natural operator mobility, creating a mobility gap. HALO uses kinesthetic teaching to align the operator's perception–action space with the robot and removes the base during data collection to preserve natural locomotion. Deployment-aligned demonstration processing addresses the resulting visual and control discrepancies.

Side-view sketch comparing Mobile ALOHA, with perception and mobility gaps, to HALO, where the operator stands at the arms and walks naturally. — Fig. 1 · Mobile ALOHA's perception and mobility gaps, and HALO's aligned interface.

Mechanical Design

HALO combines a manipulation interface, a mobility interface, deployment-aligned demonstration processing, and policy learning. Explore the deployment system and collection platform through the five steps; on desktop, drag the model to look around.

Annotated photo of the HALO deployment system: two 6-DoF arms with wrist cameras, a head camera, the aluminium frame, battery and computer, the differential-drive base with its drive wheel, an encoder and a fixed-direction wheel.

System overview

Drag to rotate

01 · System overview

HALO deployment system

The framework supports two 6-DoF arms, one head camera, two wrist cameras, and the battery and computer. The deployment system uses an AgileX Tracer AGV 2 differential-drive base. The base is removed during data collection.

Arms: 2 × 6-DoF arms
Cameras: 1 head + 2 wrist
Base: AgileX Tracer AGV 2
Framework: 880 × 710 mm

02 · Mobility interface

Mobility interface

Removing the powered mobile base preserves natural locomotion and reduces the horizontal force required for repositioning to 2.3 N. Natural body motion drives the low-resistance collection platform, enabling implicit base control during bimanual manipulation. Passive wheels and two fixed-direction wheels along the drive-wheel axis preserve the deployed base's nonholonomic motion space, without post-hoc retargeting.

Horizontal force to reposition

Deployment system47.3 N

Mobile ALOHA13.0 N

Collection platform2.3 N

03 · Manipulation interface

Kinesthetic end-effector redesign

HALO uses kinesthetic teaching to align the operator's perception and action space with the robot while preserving direct visual and force feedback. The end-effector removes ALOHA's handle-and-scissor mechanism and adopts a compact, high-contact design with increased hand contact area. The redesign reduces end-effector size by more than half and increases the available pitch angle by about 40°.

	ALOHA-style	HALO
End-effector size	242 mm	105 mm
Angle illustrated in Fig. 3	60.0°	17.7°

04 · Data processing

Deployment-Aligned Data Processing

Base motion alignment uses two encoders to measure collection-platform motion. Their readings are converted into left and right platform wheel velocities, u_L and u_R. An affine mapping converts the measured motion to deployment-compatible base linear and angular velocities, while temporal alignment accounts for the deployed base's response delay.

\begin{bmatrix}v\\\omega\end{bmatrix}=\begin{bmatrix}k_{vL}&k_{vR}\\k_{\omega L}&k_{\omega R}\end{bmatrix}\begin{bmatrix}u_L\\u_R\end{bmatrix}+\begin{bmatrix}d_v\\d_\omega\end{bmatrix}

The coefficients and offsets are estimated from paired platform measurements and commanded base velocities. The estimated action–state delay is used for temporal alignment. Viewer readouts are illustrative.

Visual processing removes demonstrator-specific cues: CaRE-Ego detects the operator's hands, SAM2 propagates masks across frames, and DiffuEraser reconstructs the masked regions for observations that better match autonomous execution.

05 · Policy learning

Policy learning

The pretrained π_0.5 vision-language-action model is fine-tuned on HALO demonstrations for each task. It receives one head-camera image, two wrist-camera images, robot joint positions, and base linear and angular velocities. The action vector concatenates arm actions with base linear and angular velocities. Asynchronous real-time action chunking (RTC) predicts the next action chunk while the current chunk is executed, conditioning on the committed action prefix to maintain continuity.

Collecting Demonstrations

The operator works with their own hands and walks at a natural pace. The robot's head camera sees those hands, so they are masked and inpainted afterwards, and the training images match what the robot sees on its own.

Operator: natural locomotion + kinesthetic teaching

Robot head camera (inset: after hand removal)

Implicit base control

The low-resistance collection platform allows base motion to arise naturally from the operator's movement, without explicit base control that could disrupt hand manipulation, thereby capturing coordinated mobile manipulation behaviors.

Three photos of an operator inside the HALO frame at a shelf, stepping and leaning while the platform follows. — Fig. 5 · Base motion arises from the operator's movement.

Efficiency against VR teleoperation and Mobile ALOHA

Same operator, same tasks, same time budget; the operator had not used any of the interfaces before. Counters in the video track the demonstrations collected with each interface on Push Chair (4× speed).

Successful demonstration time

Average duration of successful demonstrations, seconds
Task	HALO	Mobile ALOHA	VR
Push Chair	11	32	30
Disassemble Lego	23	64	37

Demonstration success rate

Demonstration success rate, percent
Task	HALO	Mobile ALOHA	VR
Push Chair	100	100	25
Disassemble Lego	100	80	20

VR's perception gap makes precise gripper control hard; Mobile ALOHA's powered base constrains the operator, who pauses before grasping the chair.

Deployment-aligned demonstration processing

Deployment-aligned demonstration processing addresses visual and control discrepancies between demonstration and deployment. In visual processing, CaRE-Ego detects the operator's hands, SAM2 propagates masks across frames, and DiffuEraser reconstructs the masked regions; hand removal improves Push Chair success from 55% to 85%. Base motion alignment fits an affine mapping from platform motion to deployment-compatible base velocities and estimates the action–state delay for temporal alignment.

Two example frames processed by ego hand detection, video segmentation and inpainting. — Fig. 7 · Ego hand detection → video segmentation → inpainting.

Failure cases: arm motion faster than base motion, and base travelling too far under synchronous inference. — Fig. 6 · Failures without latency compensation (a) and with synchronous inference (b).

What the Policies Learn

Precise bimanual manipulation, coordinated base and arm motion over long horizons, and generalization beyond the layout seen during collection.

Generalization of mobile manipulation

Push Chair asks the robot to grasp a chair with both grippers while moving at about walking speed (~0.7 m/s) and push it under a table. The table never moved during data collection, yet the policy handles 2–3 m changes in its starting pose (Mobile ALOHA used 5–10 cm) and unseen environments. Velocity-based base actions favour a reactive pattern that keeps driving the chair toward the table rather than memorizing a global pose.

2–3 m initial-pose variation

Unseen environment

Dexterity

Zip-Tie Fastening chains picking, handover, folding, insertion, fastening and placement on a short, small zip tie. The HALO policy succeeds in 30% of 10 trials (41 s on average); policies trained on Mobile ALOHA and VR demonstrations achieve 0% success. In Disassemble Lego the policy succeeds 30 of 40 times, adapts its separation strategy to randomized structures, and generalizes to unseen configurations.

Zip-Tie Fastening

Disassemble Lego

Long-horizon mobile manipulation

In the long-horizon mobile manipulation task, the robot retrieves a box from a shelf, loads objects into it, lifts it with both arms, drives to a conveyor, places it, and returns to repeat. Despite many forward, backward and turning base motions, the policy brings the base back to its starting pose so the loop can run again: 11 of 15 trials succeed (73.3%).

Long-horizon mobile manipulation · Retrieve, load, lift, transport, place, and return. 3×.

Quantitative Results

Policy success rates on real hardware, and the ablations from the paper. With the same collection budget, policies trained on Mobile ALOHA or VR data fail on Disassemble Lego and the long-horizon mobile manipulation task and score 0% on Zip-Tie Fastening.

HALO policy success by task

Task success rate of HALO-trained policies, percent
Task	HALO policy
Push Chair	85
Disassemble Lego	75
Long-horizon mobile manipulation	73.3
Zip-Tie Fastening	30

Push Chair: async inference, horizon 8, hands removed · Disassemble Lego 30/40 · Long-horizon mobile manipulation 11/15 · Zip-Tie Fastening 10 trials, 41 s average.

Inference strategy · Push Chair

Success rate by execution horizon under synchronous and asynchronous inference, percent
Setting	Success rate
Sync 8	0
Sync 25	35
Async 4	75
Async 6	75
Async 8	85
Async 10	70

Sync = synchronous inference; Async = asynchronous inference. Numbers are execution horizons. Asynchronous real-time action chunking (RTC) hides inference pauses; an intermediate horizon balances reactivity and consistency.

Visual post-processing · Push Chair

Success rate with and without visual post-processing, percent
Training images	Success rate
Raw (hands visible)	55
Hands removed	85

Same demonstrations, best inference setting; only the visual processing differs.

Action representation · Disassemble Lego

Success rate by action space and initialization, percent
Action space	From VLM	Pretrained VLA
End-effector (EEF)	12.5	47.5
Joint-space	7.5	75

End-effector (EEF) actions do better from scratch; joint-space actions do better after VLA pretraining.

Supplementary Video

BibTeX

@inproceedings{halo2026,
  title     = {HALO: Human-Aligned Data Collection for Whole-body Mobile Manipulation},
  author    = {TODO: author list},
  booktitle = {TODO: venue},
  year      = {TODO}
}