Skip to content

Embeddings

PART 3 — Embeddings

Text needs to be converted into a format where computers can understand meaning.

  • Embedding: A list of numbers (a dense vector) that represents the semantic meaning of a piece of text.
  • Vector: An array of floats, e.g., [0.12, -0.45, 0.89, ...].
  • Semantic similarity: Concepts that mean similar things have vectors that point in similar directions in mathematical space.
  • Embedding Models: Specialized, smaller AI models whose only job is to convert text into vectors.
  • Embedding dimensions: The length of the vector array (e.g., 1536 numbers).

Intuition: Imagine a 2D graph with "Royalty" on the Y-axis and "Gender" on the X-axis. The vector for "king" might be [0.9, 0.9]. The vector for "queen" might be [0.9, -0.9]. The vector for "apple" might be [-0.8, 0.0]. Because "king" and "queen" are close in the "Royalty" dimension, the computer knows they are related.

Where it appears in production: 

Document
 ↓
Chunk (Paragraph)
 ↓
Embedding Model
 ↓
Vector [0.12, 0.88, ...]
 ↓
Vector Database

Why does an HLD engineer care?

Embeddings are the foundation of Search and Retrieval-Augmented Generation (RAG). Instead of searching for exact keyword matches (SQL LIKE '%king%'), embeddings allow your system to search for concepts. Understanding embeddings is required to design semantic search and recommendation systems.