Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

原文：Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

作者：Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Software Engineering

arXiv:2610.01847v1 (cs)

[Submitted on 1 Oct 2026]

Title:Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

Authors:Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang

View a PDF of the paper titled Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning, by Zichen Xie and 4 other authors

View PDF

HTML (experimental)

Abstract:Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet these specifications may themselves contain defects: two individually reasonable principles may prescribe incompatible behavior when applied to the same situation, leaving no response that satisfies both. Detecting such inconsistencies is challenging. Formalizing natural-language specifications risks losing subtle distinctions, while behavior-based testing cannot reliably distinguish specification defects from differences in model behavior. We introduce VeriSpec, the first approach to directly detect inconsistencies in model specifications by auditing the specification text itself. Our key insight is to preserve the specification in natural language while using an LLM as a verifier. VeriSpec extracts structured, context-aware rules, constructs a topic-guided graph to cluster behaviorally related rules at the same authority level, and applies LLM-as-verifier reasoning to detect inconsistencies. Applying VeriSpec to the OpenAI Model Spec, we extract 405 rules and manually validate five inconsistencies, all reported to its developers, who responded positively and have initiated internal discussions. Compared with five baselines, VeriSpec identifies the most validated inconsistencies, achieves the highest precision (38.5%), and incurs the lowest cost per validated inconsistency ($11.12). These results establish direct specification auditing as a practical complement to behavioral alignment evaluation, catching defects at the source before they shape any model. The code is available at this https URL.

Subjects:

Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

Cite as:

arXiv:2610.01847 [cs.SE]

(or

arXiv:2610.01847v1 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2610.01847

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

软件工程 · 人工智能 · 自然语言处理

由「前沿雷达」于 2026-10-01 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。