Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

原文：Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

作者：Jiawen Du, Arshan Ali Khan, Chenhao Zhang, Zachary Plotkin, Li Shen, Qi Long, Yun Li, Can Chen, Tianlong Chen, Nicholas Konz

来源：arXiv cs.LG（机器学习）

正文

Computer Science > Machine Learning

arXiv:2610.06685v1 (cs)

[Submitted on 5 Oct 2026]

Title:Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

Authors:Jiawen Du, Arshan Ali Khan, Chenhao Zhang, Zachary Plotkin, Li Shen, Qi Long, Yun Li, Can Chen, Tianlong Chen, Nicholas Konz

View a PDF of the paper titled Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs, by Jiawen Du and 9 other authors

View PDF

HTML (experimental)

Abstract:Clinical questions often depend on linking a patient’s multimodal evidence to external biomedical knowledge, yet existing predictive systems rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions. We present MM-KG (Multimodal Knowledge Graph), which represents heterogeneous, multimodal patient observations and biomedical concepts as separate layers in one typed graph, joined by explicit alignment edges. First, modality-specific harmonizers convert EHR text, imaging, genomic, and biospecimen data into typed observations mapped to UMLS concepts, which a route-prioritized aligner links to a biomedical knowledge graph. Query-conditioned retrieval then selects a compact subgraph for downstream use by a large language model or a graph neural network. We build MM-KGs for MIMIC-IV and ADNI, and evaluate them with a 2x2 design that separates patient evidence, biomedical knowledge, and their interaction. On questions that require both sources, neither source alone performs far above chance, whereas their combination yields a drug-controlled AUROC interaction of +0.194 on MIMIC and +0.299 on ADNI. On held-out five-candidate ranking, MM-KG outperforms MindMap by +0.131 Hits@1 and leads an adapted GraphCare on the items that require consulting the patient, and deleting the single answer-bearing relation from the retrieved packet returns Hits@1 to the no-knowledge baseline. Finally, query-conditioned retrieval reaches 0.731 AUROC with 6.8x less context than the strongest generic policy, whereas static knowledge graph context gives no consistent gain on ordinary outcome prediction. Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.

Subjects:

Machine Learning (cs.LG); Computation and Language (cs.CL)

Cite as:

arXiv:2610.06685 [cs.LG]

(or

arXiv:2610.06685v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.06685

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · 自然语言处理

由「前沿雷达」于 2026-10-06 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。