The Life of a Token: from Words to Bits on the Wire

原文：The Life of a Token: from Words to Bits on the Wire

作者：Davide Avesani, Pengwenlong Gu, Sotiris Skaperas, Stefano Secci

来源：arXiv cs.DC（分布式并行）

正文

Computer Science > Distributed, Parallel, and Cluster Computing

arXiv:2609.19924v1 (cs)

[Submitted on 17 Sep 2026]

Title:The Life of a Token: from Words to Bits on the Wire

Authors:Davide Avesani (CEDRIC - ROC), Pengwenlong Gu (CEDRIC - ROC), Sotiris Skaperas (Cnam), Stefano Secci (CEDRIC - ROC)

View a PDF of the paper titled The Life of a Token: from Words to Bits on the Wire, by Davide Avesani (CEDRIC - ROC) and 3 other authors

View PDF

Abstract:Large Language Models (LLMs) transform vast collections of unstructured text into semantic patterns used for language generation and reasoning tasks. Behind their ease of use lies a complex process: words become tokens, tokens become vectors, and vectors ultimately give rise to streams of bits that flow through High-Performance Computing (HPC) systems. As modern LLMs grow to billions or trillions of parameters, this path increasingly unfolds across thousands of interconnected accelerators, making the underlying communication fabric a critical and often opaque component of model training. This tutorial aims to walk the reader through the journey from words to network traffic, shedding light on how language is translated into communication flows within HPC training systems. Using concrete examples from Dante’s Divine Comedy, we illustrate how model architecture, tokenization, embeddings, and parallelization strategies shape the volume, structure, and timing of data exchanged across the network. We combine architectural analysis with analytical traffic models and numerical examples to characterize the communication requirements of LLM training. We try to demystify how words travel across the network and provide practical insights into the network requirements needed to support the journey from text to trained model.

Subjects:

Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI); Performance (cs.PF)

Cite as:

arXiv:2609.19924 [cs.DC]

(or

arXiv:2609.19924v1 [cs.DC] for this version)

https://doi.org/10.48550/arXiv.2609.19924

Focus to learn more

arXiv-issued DOI via DataCite

主题

分布式计算 · 机器学习 · cs.NI · cs.PF

由「前沿雷达」于 2026-09-17 采集。正文取自原文页面，已保留出处链接。标题与正文版权归原作者所有。