Skip to content

Tokens And Text Representation

PART 2 — Tokens and Text Representation

AI models don't process strings. They process fixed-length arrays of integers.

  • Tokenizer: A pre-processing dictionary (often implemented as a Trie or via Byte Pair Encoding - BPE). It maps subwords to integers.
    • Example: "hamburger" -> ["ham", "bur", "ger"] -> [412, 88, 902].
  • Vocabulary Size: The size of this dictionary (e.g., 32k to 128k entries).
  • Context Window: The maximum array length the model's memory can hold at once.

SDE Takeaway: Tokens are the fundamental unit of cost and latency. * Cost: APIs charge per array element ($ per 1k tokens). * Latency: Output latency scales linearly with output array length O(N). Memory usage scales linearly or quadratically O(N^2) with the context window size.