Skip to content

Tokens And Text Representation

PART 2 — Tokens and Text Representation

AI models do not read English words; they read numbers.

  • Token: A fragment of a word. A token can be a single character, a syllable, or a whole word. (e.g., "hamburger" might be split into "ham", "bur", "ger"). Roughly, 1 token ≈ 0.75 English words.
  • Tokenization: The process of chopping text into tokens.
  • Token IDs: The unique integer assigned to each token in the vocabulary (e.g., "ham" = 453, "bur" = 992).
  • Vocabulary: The fixed list of all possible tokens the model knows.
  • Context: The sequence of tokens provided to the model.
  • Context Window: The maximum number of tokens a model can process at once (e.g., 128k tokens).
  • Input vs Output tokens: Input tokens are what the user sends (the prompt). Output tokens are what the model generates.

Why does an HLD engineer care?

Everything in AI serving is priced, constrained, and measured in tokens. * Latency: More output tokens = more time spent generating. * Cost: APIs charge per input and output token. * Memory: Processing more tokens requires exponentially or linearly more GPU memory. * Context Window: If a user uploads a book that exceeds the context window, the system crashes or drops data. HLD must account for chunking or summarizing text to fit this window.