EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

原文：EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

作者：Zhengxiang Huang, Yujie Sun, Shengheng Chen, Zhaode Wang, Chaoyue Niu, Zeyu Zhao, Chengfei Lv, Fan Wu, Guihai Chen

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Operating Systems

arXiv:2611.14356v1 (cs)

[Submitted on 10 Nov 2026]

Title:EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

Authors:Zhengxiang Huang, Yujie Sun, Shengheng Chen, Zhaode Wang, Chaoyue Niu, Zeyu Zhao, Chengfei Lv, Fan Wu, Guihai Chen

View a PDF of the paper titled EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities, by Zhengxiang Huang and 8 other authors

View PDF

HTML (experimental)

Abstract:Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host–device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine–backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.

Subjects:

Operating Systems (cs.OS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE)

Cite as:

arXiv:2611.14356 [cs.OS]

(or

arXiv:2611.14356v1 [cs.OS] for this version)

https://doi.org/10.48550/arXiv.2611.14356

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

cs.OS · 人工智能 · 自然语言处理 · 软件工程

由「前沿雷达」于 2026-11-11 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。