Skip to content

Tokens And Text Representation

PART 2 — Tokens and Text Representation

AI models process fixed-length integer arrays, not strings. The conversion is done by a Tokenizer.

Example Trace (Byte Pair Encoding): Let's tokenize the string: "unfriendliness"

  1. The tokenizer looks at its dictionary (vocabulary).
  2. It breaks the string into known chunks: ["un", "friend", "li", "ness"]
  3. It maps these to integer IDs: 
    vocab = {"un": 101, "friend": 5042, "li": 89, "ness": 302}

input_string = "unfriendliness"
token_ids = [101, 5042, 89, 302] # This array is what actually goes to the GPU

SDE Implication: If you change "friend" to a word the tokenizer doesn't know well, it might split it into 5 characters [f, r, i, e, n, d]. This means 1 word suddenly costs 5x more compute and API cost to process!