1. Pass the entire sequence to the model

PART 8 — LLMs (Autoregressive Generation)

LLMs generate text exactly one token at a time in a while loop.

Example:

prompt = [101, 500] # "The cat"

while True:
    # 1. Pass the entire sequence to the model
    # The model outputs a probability distribution over the 32,000 token vocabulary
    probabilities = model.forward(prompt) 

    # Example probabilities: { "sat": 0.8, "slept": 0.15, "barked": 0.001 }

    # 2. Pick the next token (temperature 0 means pick the max)
    next_token = get_max_probability(probabilities) # Returns "sat" (ID 892)

    # 3. Check for stop condition
    if next_token == STOP_TOKEN_ID or len(prompt) > MAX_LEN:
        break

    # 4. APPEND and repeat! (This is why it's slow)
    prompt.append(next_token) 

    print(decode(next_token)) # Streams "sat" to the user UI

---