Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

原文：Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

作者：Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. Zhang

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Machine Learning

arXiv:2610.35733v1 (cs)

[Submitted on 26 Oct 2026]

Title:Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

Authors:Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. Zhang

View a PDF of the paper titled Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution, by Zhenshuo Zhang and 2 other authors

View PDF

HTML (experimental)

Abstract:Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.

Comments:

30 pages. To appear at NeurIPS 2026 (extended version)

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Robotics (cs.RO); Machine Learning (stat.ML)

Cite as:

arXiv:2610.35733 [cs.LG]

(or

arXiv:2610.35733v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.35733

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · 人工智能 · 多智能体 · 机器人 · 统计机器学习

由「前沿雷达」于 2026-10-27 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。