"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

原文：“Dear LLaVA, Please Drive”: A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

作者：Sebastian Berger, Katharina Winter, Fabian B. Flohr

来源：arXiv cs.RO（机器人）

正文

Computer Science > Robotics

arXiv:2609.22925v1 (cs)

[Submitted on 19 Sep 2026]

Title:“Dear LLaVA, Please Drive”: A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

Authors:Sebastian Berger, Katharina Winter, Fabian B. Flohr

View a PDF of the paper titled “Dear LLaVA, Please Drive”: A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control, by Sebastian Berger and 2 other authors

View PDF

HTML (experimental)

Abstract:Vision-language models (VLMs) provide a compelling foundation for reasoning-driven mobile navigation, offering rich contextual understanding and strong generalization from large-scale pretraining. Most existing navigation frameworks rely on imitation learning and therefore require substantial labeled trajectory data, limiting their scalability and robustness. In this work, we propose a parameter-efficient approach to fine-tune a pretrained VLM for autonomous navigation using an Imperative Learning paradigm. By optimizing against differentiable geometric cost fields rather than labeled trajectories, our model learns to generate collision-free paths exclusively from stereoscopic depth observations. We introduce a unified end-to-end navigation pipeline for natural-language-driven robotic control. This system leverages a shared VLM backbone with task-specific Low-Rank Adaptation (LoRA) modules, effectively bridging the gap from semantic target selection to low-level trajectory planning. Our approach achieves competitive Success weighted by Path Length (SPL) in unseen environments while updating less than 1% of the model’s total parameters. Qualitative real-world experiments validate sim-to-real generalization and stable path planning without fine-tuning on real-world data. These results highlight a practical approach for deploying VLM-based agents on mobile robots, enabling high-level semantic navigation without the prohibitive requirement for large-scale, labeled trajectory data.

Subjects:

Robotics (cs.RO)

Cite as:

arXiv:2609.22925 [cs.RO]

(or

arXiv:2609.22925v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2609.22925

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器人

由「前沿雷达」于 2026-09-19 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。