Input: The embedding of our token (size 3)
PART 4 — Neural Networks¶
A Neural Network layer is just a matrix multiplication followed by an activation function (like stripping negative numbers to 0).
Example Pseudo-code:
import numpy as np
# Input: The embedding of our token (size 3)
x = np.array([0.5, 0.2, 0.1])
# Weights: The "learned" parameters of the model (size 3x3)
W = np.array([
[ 0.1, -0.2, 0.5],
[ 0.8, 0.1, 0.0],
[-0.3, 0.4, 0.9]
])
# Bias
b = np.array([0.1, 0.1, 0.1])
# Forward Pass (One Layer)
# 1. Multiply and add
y_raw = np.dot(W, x) + b
# 2. Activation Function (ReLU: turn negatives to 0)
y_out = np.maximum(0, y_raw)
print(y_out) # This vector is passed to the next layer