RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

原文：RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

作者：Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec

来源：arXiv cs.DB（数据库）

正文

Computer Science > Machine Learning

arXiv:2610.17867v1 (cs)

[Submitted on 13 Oct 2026]

Title:RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

Authors:Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec

View a PDF of the paper titled RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions, by Rishabh Ranjan and 7 other authors

View PDF

HTML (experimental)

Abstract:Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.

Comments:

Accepted to NeurIPS 2026. Website: this https URL

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Databases (cs.DB)

Cite as:

arXiv:2610.17867 [cs.LG]

(or

arXiv:2610.17867v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.17867

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

主题

机器学习 · 人工智能 · 数据库

由「前沿雷达」于 2026-10-13 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。