From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations

原文：From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations

作者：Bangjun Wang, Longyan Wu, Yukun Wei, Shenghe Shao, Chaoyi Huang, Wenze Cui, Zetong Xu, Hanlin Wu, Long Chen, Yi Ma, Hongyang Li

来源：arXiv cs.RO（机器人）

正文

Computer Science > Robotics

arXiv:2609.35375v1 (cs)

[Submitted on 28 Sep 2026]

Title:From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations

Authors:Bangjun Wang, Longyan Wu, Yukun Wei, Shenghe Shao, Chaoyi Huang, Wenze Cui, Zetong Xu, Hanlin Wu, Long Chen, Yi Ma, Hongyang Li

View a PDF of the paper titled From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations, by Bangjun Wang and 10 other authors

View PDF

HTML (experimental)

Abstract:Scaling up robotic manipulation is primarily bottlenecked by the scarcity of real-world robot data. While recent approaches leverage human video demonstrations to mitigate this shortage, they remain computationally expensive and still rely on paired human-robot data for domain alignment. Although current state-of-the-arts excel at long-horizon tasks, they struggle with the delicate and precise control required for complex tool manipulation. To overcome these limitations, we introduce P2P-T, from Pixel to Poses for Tool Manipulation, a data-efficient, object-centric framework that learns tool use directly from human demonstrations. P2P-T bridges the cognitive and physical execution gap through a two-stage approach. First, pretraining an object-centric world model to extract stable pose priors; second, integrating these priors into an efficient, pose-aware low-level policy. By utilizing a robust automated data processing pipeline powered by modern foundation models, P2P-T completely bypasses the need for human-robot aligned data. This reduces overall training overhead drastically. With minimal per-task fine-tuning, our framework achieves a 73% improvement over the previous state of the art in execution performance on complex, real-world tool manipulation tasks that currently remain out of reach for standard large-scale pretrained models.

Subjects:

Robotics (cs.RO); Artificial Intelligence (cs.AI)

Cite as:

arXiv:2609.35375 [cs.RO]

(or

arXiv:2609.35375v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2609.35375

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器人 · 人工智能

由「前沿雷达」于 2026-09-28 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。