Dual-arm manipulation · ACT · Corrective HIL

Bimanual Towel Folding with Corrective Human-in-the-Loop Fine-Tuning

Xiang Meng (孟想)¹ · Mingzhe Liu (刘明哲)¹

¹ East China University of Science and Technology

Advisor: Ning Guo (郭宁)²

² School of Artificial Intelligence, Shanghai Jiao Tong University

Summer research project · HIROL Lab

A staged learning pipeline for precise corner grasping and complete towel folding on two Piper arms. The final policy succeeds in 35 of 50 trials across five towels placed at random positions.

Experiment record Physical AI Runtime

70%fine-tuned policy

150evaluation trials

101corrective HIL episodes

5test towels

Video slot static/videos/finetune10k.mp4

A complete autonomous fold by the fine-tuned policy. The towel is placed at a random test position.

Overview

Abstract

Precise cloth manipulation is difficult to learn from a small set of successful demonstrations: a policy may approach the cloth while still missing the corners required for a stable bimanual grasp. We study a staged alternative. An ACT policy is first trained on 100 direct corner-grasp demonstrations using a blue square towel. The policy is then deployed, and a human operator takes over at states where the grippers are misaligned or likely to miss. These interventions produce 101 full-task trajectories on red and green towels. Fine-tuning from the corner-grasp checkpoint reaches 70% full-task success over 50 random-placement trials, compared with 26% for the pretrained policy and 20% for a policy trained from scratch on the HIL data. The results support corrective HIL fine-tuning as a practical route for this system, while leaving position-controlled robustness and cross-instance generalization open.

System

One runtime from sensing to intervention

physical_ai_runtime connects the cameras, policy, RMI authority layer, dual-arm control and episode recorder. The same runtime is used for autonomous evaluation and for switching to leader-arm control during HIL collection.

Architecture of physical_ai_runtime, corrective teleoperation and the training loop — The complete loop: three-camera sensing, ACT inference, RMI authority switching, dual-Piper execution, corrective recording and dataset conversion.

Video slotstatic/videos/three_camera_inputs.mp4

The synchronized policy observation: Orbbec top RGB together with the left- and right-wrist RGB streams.

Custom elongated slim gripper fingers on the Piper robot — The stock Piper fingers were replaced with longer, narrower fingers to reach towel corners with less geometric interference.

Observation Top RGB + two wrist RGB 14-D robot state

Runtime physical_ai_runtime ACT · RMI · recorder

Execution Dual Piper followers 14-D joint references

Dual leadersTeleop takeoverRecorded HIL episode

The custom fingers are part of the final hardware configuration. Their contribution was not isolated in a gripper ablation, so no separate performance gain is claimed.

Method

A curriculum defined by policy failures

Learn the prerequisite

Train base20k for 20k steps on 100 blue-square demonstrations of direct corner grasping.

Visit failure states

Run the policy on the robot. Misalignment and near-miss states are reached by the policy rather than scripted in advance.

Correct and finish

The operator takes control with two leader arms, recovers the grasp and completes lifting and folding. This yields 101 HIL episodes.

Fine-tune the full task

Continue from base20k for 10k HIL steps. Compare against a 20k-step scratch model trained on the same HIL data.

Video slotstatic/videos/pretraining_teleop.mp4

Human demonstrations. The 100 pretraining trajectories were collected entirely through bilateral leader-arm teleoperation.

Video slotstatic/videos/policy_hil_takeover.mp4

Corrective HIL episode. The policy reaches a difficult state; the operator takes authority, recovers the grasp and completes the task.

Results

Fine-tuning closes most of the gap

Each model is evaluated in 50 trials: ten trials on each of five towels, with the towel position randomized before every run.

Basecorner grasp, 20k

26%

ScratchHIL-101, 20k

20%

Fine-tunedbase + HIL-101, 10k
70%

Towel	Training exposure	Base 20k	Scratch 20k	Fine-tuned 10k
Blue square	Pretraining	5 / 10	0 / 10	8 / 10
Green square	HIL	3 / 10	2 / 10	9 / 10
Red square	HIL	2 / 10	3 / 10	8 / 10
Yellow square	Unseen	1 / 10	2 / 10	6 / 10
White rectangle	Unseen color, instance and shape	2 / 10	3 / 10	4 / 10

+44 ppover the pretrained base policy

+50 ppover scratch training on HIL-101

17 / 20on red and green HIL towels

10 / 20across the two unseen towels

The five towels used for evaluation — Evaluation towels: blue (pretraining), red and green (HIL), yellow (unseen square), and white (unseen rectangle).

Qualitative examples

Representative rollouts

These complete trials contrast the corner-grasp checkpoint with the fine-tuned full-task policy. The low-performing scratch model remains in the quantitative comparison, but is omitted here to keep the qualitative evidence focused.

Video slotstatic/videos/base20k.mp4

Base 20k. Direct corner-grasp pretraining; representative failure at a difficult random position.

Video slotstatic/videos/finetune10k.mp4

Fine-tuned 10k. Inherits the corner-grasp skill and completes the full folding task.

Scope

What the experiment does not establish

All models use one training seed and one robot/camera configuration.
Random placement was not logged as position or orientation, so placement difficulty cannot be balanced or stratified after the fact.
Only binary condition-level outcomes were retained; stage failures, task time and fold error are unavailable.
The white towel changes color, instance and geometry together. Its 4/10 result is not a shape-only ablation.
The custom fingers, corrective-state sampling and full-task labels were not isolated in separate ablations.

Within these limits, the result is clear: preserving the prerequisite corner-grasp skill and fine-tuning on corrective full-task trajectories is substantially more effective than either control used in this study.

Acknowledgements

Laboratory and team

We thank the Human-Inspired Robotics Learning Lab (HIROL), School of Artificial Intelligence, Shanghai Jiao Tong University, and its team, led by Prof. Wenzhao Lian (连文昭), for their support of this summer research project.

Citation

Cite this project page

Xiang Meng and Mingzhe Liu. Bimanual Towel Folding with Corrective Human-in-the-Loop Fine-Tuning. Summer research project website. Advisor: Ning Guo. Project page.

See citation details and BibTeX.