Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization

原文：Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization

作者：Xinhao Yang, Wenhao Wu, Ning Lv, Yanshen Ding, Zhenhong Sun, Daoyi Dong, Chunlin Chen, Zhi Wang

来源：arXiv cs.RO（机器人）

正文

Computer Science > Robotics

arXiv:2609.39599v1 (cs)

[Submitted on 30 Sep 2026]

Title:Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization

Authors:Xinhao Yang, Wenhao Wu, Ning Lv, Yanshen Ding, Zhenhong Sun, Daoyi Dong, Chunlin Chen, Zhi Wang

View a PDF of the paper titled Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization, by Xinhao Yang and 7 other authors

View PDF

HTML (experimental)

Abstract:3D visuomotor policies provide a strong foundation for spatially precise manipulation, yet current text-to-3D policies struggle to follow unseen fine-grained behavioral specifications beyond those covered by demonstrations. We study this challenge as unseen specification generalization, where language specifies behaviorally significant variations, such as target position, displacement, or articulated state, that are absent from policy training. We find that pretrained language representations and conventional global behavior-language alignment capture coarse task semantics but often blur nearby specifications that require distinct behaviors. We introduce T3DP, a Text-to-3D Policy framework for fine-grained language-behavior alignment. Rather than compressing each instruction and demonstration into a single global embedding, T3DP preserves their local structures and establishes bidirectional token-level correspondence between linguistic elements and behavioral segments. This directly grounds subtle linguistic variations in the behavior components they affect, preventing closely related specifications from collapsing in the representation space. The resulting specification-sensitive language representation conditions a point-cloud-based 3D diffusion policy, enabling more precise control over unseen behavioral specifications without modifying the underlying policy architecture. Across Meta-World, ManiSkill, and RoboTwin, T3DP improves average held-out-specification success over global language-behavior alignment by +11.0-14.2 points, with gains on all 15 task families; on real-robot tasks, it further raises average success from 47.5% to 65.0% (+17.5 points). Representation and action-probe analyses show that fine-grained alignment better preserves specification geometry and action-relevant variation, linking local behavior grounding to downstream control.

Comments:

24 pages, 8 figures, 8 table

Subjects:

Robotics (cs.RO); Artificial Intelligence (cs.AI)

Cite as:

arXiv:2609.39599 [cs.RO]

(or

arXiv:2609.39599v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2609.39599

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器人 · 人工智能

由「前沿雷达」于 2026-09-30 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。