Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

原文：Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

作者：Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu

来源：arXiv cs.RO（机器人）

正文

Computer Science > Robotics

arXiv:2610.18352v1 (cs)

[Submitted on 13 Oct 2026]

Title:Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

Authors:Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu

View a PDF of the paper titled Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation, by Youpeng Wen and 7 other authors

View PDF

HTML (experimental)

Abstract:A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0%$ over Orchestra using the base $\pi_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at this https URL.

Subjects:

Robotics (cs.RO); Artificial Intelligence (cs.AI)

Cite as:

arXiv:2610.18352 [cs.RO]

(or

arXiv:2610.18352v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2610.18352

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器人 · 人工智能

由「前沿雷达」于 2026-10-13 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。