作者：Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge
来源：arXiv cs.DB（数据库）
正文
Computer Science > Computation and Language
arXiv:2609.06527v1 (cs)
[Submitted on 6 Sep 2026]
Title:ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
Authors:Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge
View a PDF of the paper titled ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language, by Hang Zhang and 14 other authors
View PDF
HTML (experimental)
Abstract:Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Databases (cs.DB)
Cite as:
arXiv:2609.06527 [cs.CL]
(or
arXiv:2609.06527v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.06527
Focus to learn more
arXiv-issued DOI via DataCite
主题
自然语言处理 · 人工智能 · 数据库
由「前沿雷达」于 2026-09-06 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。
评论
加载中…