Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs

原文：Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs

作者：Niklas Wagner, Alexander Mädche

来源：arXiv cs.AI（人工智能）

正文

Computer Science > Human-Computer Interaction

arXiv:2610.23304v1 (cs)

[Submitted on 16 Oct 2026]

Title:Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs

Authors:Niklas Wagner, Alexander Mädche

View a PDF of the paper titled Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs, by Niklas Wagner and 1 other authors

View PDF

Abstract:Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit’s data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.

Comments:

Published in the Proceedings of the 34th European Conference on Information Systems (ECIS 2026)

Subjects:

Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Databases (cs.DB)

Cite as:

arXiv:2610.23304 [cs.HC]

(or

arXiv:2610.23304v1 [cs.HC] for this version)

https://doi.org/10.48550/arXiv.2610.23304

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

人机交互 · 人工智能 · 数据库

由「前沿雷达」于 2026-10-16 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。