作者：Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina, Alejandro Maté, Juan Trujillo
来源：arXiv cs.DB（数据库）
正文
Computer Science > Software Engineering
arXiv:2610.11482v1 (cs)
[Submitted on 8 Oct 2026]
Title:Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows
Authors:Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina, Alejandro Maté, Juan Trujillo
View a PDF of the paper titled Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows, by Jorge Garc'ia-Carrasco and 4 other authors
View PDF
HTML (experimental)
Abstract:Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasize textual responses or isolated code generation rather than the validity of complete engineering artifacts.
Objectives: We evaluate whether local LLM agents can produce correct and reproducible data-engineering artifacts, quantify the effect of a closed-loop workspace condition, and examine trade-offs in model scale, architecture, quantization, runtime, tool use, and failure.
Methods: We introduce a benchmark of fifteen mobility-workflow tasks covering data discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Deterministic checkers assess generated scripts, tables, structured files, figures, and reports. Ten local configurations are evaluated in one-shot and closed-loop conditions, with five repetitions per model, mode, and task, yielding 1,500 scored attempts on a consumer-grade GPU.
Results: Among models larger than two billion parameters, the workspace condition increases pass rates by 26.7-52.0 percentage points over one-shot generation. The strongest configuration reaches 85.3% artifact-level success, and a quantized 9-billion-parameter model reaches 69.3% with an approximately 6.5 GB memory footprint. Gains are largest when intermediate artifacts expose errors the agent can inspect and repair.
Conclusion: Local open-weight agents can support a meaningful subset of software-intensive data-engineering work, but reliability depends on model capability, task verifiability, and deterministic validation. The benchmark provides a reproducible method for evaluating complete agent configurations before adoption in engineering workflows.
Comments:
Preprint, under review. 26 pages, 5 figures, 5 tables. Dataset: this https URL
Subjects:
Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Databases (cs.DB); Machine Learning (cs.LG)
ACM classes:
D.2.8; I.2.7
Cite as:
arXiv:2610.11482 [cs.SE]
(or
arXiv:2610.11482v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2610.11482
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
主题
软件工程 · 人工智能 · 数据库 · 机器学习
由「前沿雷达」于 2026-10-08 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。
评论
加载中…