Phase 8: Extensions, C Modules API & Modern Ecosystem¶
Audience: Principal Distributed Systems Architect & Module Developer
Source References:src/module.c,src/redismodule.h,src/eval.c,src/functions.c
1. The Redis Modules C/Rust API Architecture¶
Introduced in Redis 4.0, the Modules API allows engineers to extend Redis with native C, C++, or Rust code running at bare-metal memory speeds inside the engine process.
┌────────────────────────────────────────────────────────┐
│ redis-server Core │
├────────────────────────────────────────────────────────┤
│ aeEventLoop │ Dict Engine │ Jemalloc Heap │ RDB Engine │
└─────────┬──────────────┬─────────────┬────────────┬────┘
│ │ │ │
▼ ▼ ▼ ▼
┌────────────────────────────────────────────────────────┐
│ RedisModule ABI / API Layer │
├────────────────────────────────────────────────────────┤
│ RedisModule_CreateCommand() RedisModule_CreateDataType│
│ RedisModule_Alloc() RedisModule_OpenKey() │
└───────────────────────────┬────────────────────────────┘
│
▼
[Custom Shared Library (.so file)]
- Native Vector Index (HNSW)
- Time Series Ring Buffers
- Graph Engine (Adjacency Matrix)
1.1 Anatomy of a Minimal C Module¶
#include "redismodule.h"
int MyCustomCommand_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
if (argc != 2) return RedisModule_WrongArity(ctx);
RedisModule_AutoMemory(ctx);
/* Zero-copy access to key in Redis Dict */
RedisModuleKey *key = RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);
if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) {
RedisModule_StringSet(key, argv[1]);
RedisModule_ReplyWithSimpleString(ctx, "CREATED");
} else {
RedisModule_ReplyWithSimpleString(ctx, "EXISTS");
}
return REDISMODULE_OK;
}
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
if (RedisModule_Init(ctx, "mycustommod", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, "mycustom.set", MyCustomCommand_RedisCommand, "write deny-oom", 1, 1, 1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
return REDISMODULE_OK;
}
1.2 Defining Custom Native Data Types¶
A module can register an entirely new object type (RedisModule_CreateDataType) by providing function pointers for engine lifecycle events:
- rdb_save: Serialize custom memory graphs into binary RDB streams.
- rdb_load: Deserialization during server boot.
- aof_rewrite: Generate RESP commands to reconstruct state during AOF rewrites.
- free: Memory reclamation callback executed when the key is deleted or evicted.
2. RediSearch & Vector Similarity Search (HNSW)¶
RediSearch transforms Redis into a secondary indexing engine and high-dimensional vector database for LLM retrieval-augmented generation (RAG).
Vector Similarity Search Architecture:
[High-Dimensional Embedding (e.g. 1536 floats)]
│
▼
[HNSW Hierarchical Graph]
Level 2: [Node A] ─────────────────────────► [Node M]
│ │
▼ ▼
Level 1: [Node A] ────────► [Node G] ──────► [Node M]
│ │ │
▼ ▼ ▼
Level 0: [Node A] ──► [B] ──► [G] ──► [K] ──► [M] ──► [Z]
2.1 Flat vs. HNSW Indexing¶
- Flat Index (
FLAT): Brute-force exact nearest neighbor search (\(O(N)\) dot-product / cosine similarity scans). Perfect accuracy; high latency for large datasets (\(> 100k\) vectors). - HNSW (
HNSW): Hierarchical Navigable Small World graphs (\(O(\log N)\) approximate nearest neighbors). Delivers sub-millisecond retrieval across millions of vectors.
# Create Vector Index on Hash / JSON
FT.CREATE idx:documents ON HASH PREFIX 1 doc: SCHEMA \
title TEXT \
embedding VECTOR HNSW 6 \
TYPE FLOAT32 \
DIM 1536 \
DISTANCE_METRIC COSINE \
M 16 \
EF_CONSTRUCTION 200
3. Lua Scripting vs. Modern Redis Functions (v7.0+)¶
3.1 The Legacy Lua Scripting Limitations (EVAL / EVALSHA)¶
In Redis 2.6 to 6.2, atomic multi-key logic relied on Lua scripts:
- EVAL "redis.call('set', KEYS[1], ARGV[1])" 1 mykey myval
- Pain Points:
- Scripts were transient and stored in a volatile RAM cache (eval_scripts dict).
- If a Redis instance restarted or failed over to a replica, the script cache was cleared. The application layer threw NOSCRIPT No matching script. Please use EVAL. errors and was forced to reload scripts.
3.2 Modern Redis Functions (Redis 7.0+)¶
Functions are first-class citizens stored permanently in the database and synchronized across replication links and RDB snapshots.
#!lua name=inventory_engine api_version=1.0
local function reserve_stock(keys, args)
local item_key = keys[1]
local quantity_demanded = tonumber(args[1])
local current_stock = tonumber(redis.call('GET', item_key) or 0)
if current_stock >= quantity_demanded then
redis.call('DECRBY', item_key, quantity_demanded)
return { 1, current_stock - quantity_demanded }
else
return { 0, current_stock }
end
end
redis.register_function('reserve_stock', reserve_stock)