JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

原文：JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

作者：Gangyi Li, LuYao Wang, Jingfu Sun, Jingyi Yuan, Hang Liu, Hao Zong, Yujia Hu, Tingting Gao, Jianyong Wang, Xu Yang

来源：arXiv cs.RO（机器人）

正文

Computer Science > Robotics

arXiv:2610.12957v1 (cs)

[Submitted on 9 Oct 2026]

Title:JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

Authors:Gangyi Li, LuYao Wang, Jingfu Sun, Jingyi Yuan, Hang Liu, Hao Zong, Yujia Hu, Tingting Gao, Jianyong Wang, Xu Yang

View a PDF of the paper titled JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents, by Gangyi Li and 9 other authors

View PDF

HTML (experimental)

Abstract:Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals, and execution feedback. We present JoyInside-VLM-Agent, a ReAct-style multimodal agent developed through product-conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware. A closed-loop sandbox generates multimodal interaction trajectories covering goal revisions and failure recovery, which are checked against execution outcomes and semantic criteria. Supervised fine-tuning and multi-teacher on-policy distillation precede reinforcement learning with DPER (Dual-Source Privileged On-Policy Self-Distillation with Token-Level Evidence Routing). DPER routes token-level evidence between a frozen privileged teacher and a privileged student sharing the rollout policy’s parameters, modulating reward-derived token advantages while preserving their signs. Training rewards combine generative process evaluation, multi-judge rankings, and task-queue consistency. JoyInside-VLM-Agent achieves a BFCL v4 overall score of 77.44, exceeding the evaluated baselines; results on the other public benchmarks are mixed. On JoyInside-Devices and JoyInside-Robotics, the evaluated system, which includes domain-specific skill refinement, leads the baselines on all reported metrics. Two real-world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.

Comments:

23 pages, 6 figures, 5 tables; includes references and appendices

Subjects:

Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

Cite as:

arXiv:2610.12957 [cs.RO]

(or

arXiv:2610.12957v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2610.12957

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器人 · 人工智能 · 机器学习

由「前沿雷达」于 2026-10-09 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。