Skip to content

Search Indexing Strategy

1. Decision

Implement a robust Search Indexing Strategy by combining Change Data Capture (CDC via Debezium/Kafka) for near-real-time primary DB sync, Index Aliases for zero-downtime mapping updates, Index State Management (ISM/ILM) for automated lifecycle tiering, and optimal shard sizing (20GB–50GB per shard).

2. 30-Second Answer

Search indexing transforms raw application data into structured, searchable inverted index segments inside search engines (Elasticsearch / OpenSearch). A production-grade indexing strategy resolves three core challenges: Data Synchronization (using Change Data Capture via Debezium & Kafka to stream primary DB WAL changes asynchronously, avoiding dual-write race conditions), Index Mapping Evolution (using Index Aliases products_live pointing to versioned indices products_v2 for zero-downtime schema migrations via _reindex), and Storage Lifecycle Management (using ILM/ISM to rollover time-series indices through Hot, Warm, Cold, and Freeze storage tiers). Proper shard allocation (20–50GB target size) prevents heap exhaustion and query latency spikes.

3. Decision Matrix

Option Best For Avoid When Latency Scalability Consistency Operational Complexity
CDC Async Pipeline (Debezium + Kafka + Indexer) Production microservice architectures, zero dual-write bugs, decoupled primary DB sync Micro-scale applications where running Kafka/Debezium infrastructure is overkill Near Real-Time (10-50ms pipeline lag + 1s NRT refresh) Extremely High (Horizontal stream worker scale) Eventual Consistency (Async pipeline stream) Medium to High
Dual-Writing in Application Code Simple low-traffic CRUD applications, basic prototype setups High-scale production systems (Dual writes fail under network glitches, creating permanent index drift) Low Write latency; High risk of index drift Poor (Tightly couples search indexing to HTTP request) Weak (Inconsistent when 1 write fails) Low
Batch Bulk Ingestion (_bulk API) Historical backfills, daily analytical log ingestion, database initial seeding Real-time immediate search indexing needs High per batch; Sub-second total execution Extremely High (Optimized for throughput >50,000 docs/sec) Eventual Batch Commit Low to Medium

4. Selection Rules

Requirement Best Choice Rationale / Example
Decoupled Primary Database Search Synchronization Debezium CDC + Kafka Reads Postgres WAL changes, converts to JSON, and updates Elasticsearch asynchronously
Zero-Downtime Mapping Schema Migration Index Alias (_reindex) Point app code to alias products_live; reindex to products_v2 in background and swap alias
Automated Log Storage Cost Reduction ISM Lifecycle Policies Hot (NVMe SSD) \(\rightarrow\) Warm (HDD) \(\rightarrow\) Cold (S3 Blob) \(\rightarrow\) Delete after 90 days
High Velocity Data Seeding / Backfills Bulk API (_bulk) Ingest 5,000 documents per HTTP payload instead of issuing 5,000 single INDEX API calls
Prevent Over-Sharding Heap Exhaustion 20GB–50GB Shard Sizing Rule Maintain total shard count below 20 shards per 1GB JVM heap

5. Tradeoffs

  1. CDC Eventual Consistency vs Application Dual-Writing Risk
  2. Gain (CDC): Guarantees 100% index accuracy; primary DB WAL is the ultimate single source of truth.
  3. Lose (CDC): Introduces 10-50ms replication lag before updates become searchable.
  4. Index Aliases Zero-Downtime vs Temporary Double Storage
  5. Gain (Alias): Instantaneous, atomic zero-downtime index mapping updates via _aliases API.
  6. Lose (Alias): Requires 2x storage disk capacity during the _reindex process.
  7. Bulk Ingestion Throughput vs Memory Buffer Spikes
  8. Gain (Bulk): 10x-50x faster write throughput by amortizing HTTP and segment flush overhead.
  9. Lose (Bulk): Excessive bulk payload sizes (e.g. >50MB) trigger JVM CircuitBreakingException errors.
  10. Time-Based Rollover (ISM) vs Fixed Index Sizing
  11. Gain (Rollover): Automatically manages index sizes based on document count, age, or byte size.
  12. Lose (Rollover): Queries spanning historical ranges must search multiple index aliases.
  13. Dynamic Mapping Flexibility vs Strict Mapping Control
  14. Gain (Dynamic): Automatically infers data types for new fields on the fly.
  15. Lose (Dynamic): High risk of mapping explosion or misinterpreting text vs keyword fields.

6. Interview Counter Questions

  1. How do you implement Zero-Downtime Schema Reindexing using Elasticsearch Index Aliases?
  2. What is the difference between text and keyword field mappings in Elasticsearch/OpenSearch?
  3. How does the _bulk API optimize indexing throughput compared to single-document index requests?
  4. What is Over-Sharding and why does having thousands of small shards destroy JVM heap performance?
  5. How does Index State Management (ISM / ILM) automate Hot-Warm-Cold-Delete storage transitions?
  6. How do you handle mapping explosions caused by dynamic field indexing?
  7. What is refresh_interval and how does tuning it from 1s to 30s optimize indexing throughput?
  8. Why should you disable _source storing in special log analytics scenarios?
  9. How do you calculate optimal Shard and Replica counts for a new search index?
  10. How do you handle missing or out-of-order CDC indexing events?

7. Gold Standard Answers

  1. Create index_v1, point alias products_live to v1. When schema changes: create index_v2 with new mapping, call POST /_reindex from v1 to v2, atomic swap alias products_live from v1 to v2, delete v1.
  2. text: Analyzed (tokenized, lowercased, stemmed) for full-text search. keyword: Un-analyzed exact binary string for filtering, sorting, and aggregations.
  3. _bulk packages thousands of JSON mutations into a single HTTP payload, amortizing network TCP RTT, thread context switches, and Lucene memory segment commit flushes.
  4. Over-sharding: Every shard is a full Lucene index consuming JVM heap memory for segment headers, term dictionaries, and FSTs. Over 20 shards/GB heap causes Stop-The-World GC pauses and cluster instability.
  5. ISM defines automated state rules: Hot (Fast SSD, write-heavy) \(\rightarrow\) Rollover at 30GB \(\rightarrow\) Warm (Read-only, shrink shards) \(\rightarrow\) Cold (S3 offloaded) \(\rightarrow\) Delete after 90 days.
  6. Dynamic mapping creates new fields automatically when new JSON keys arrive. Unchecked nested JSON causes "Mapping Explosion" (>1,000 fields), crashing cluster state memory. Disable dynamic mapping ("dynamic": "strict" or "false").
  7. refresh_interval controls how often memory buffers flush to OS page cache as searchable Lucene segments. Increasing from 1s to 30s (or -1 during bulk loads) eliminates CPU-intensive segment merges, doubling write throughput.
  8. Disabling _source stops storing the raw original JSON blob on disk, saving ~50% storage space for high-volume logs where raw JSON reconstruction is not required.
  9. Target primary shard size = 20GB to 50GB. Formula: Primary Shards = Total Expected Index Size / 30GB. Set Number of Replicas = 1 for High Availability across data nodes.
  10. Attach version numbers or WAL sequence IDs (version field) to CDC events. Indexing workers use version_type=external, rejecting out-of-order events with lower version numbers.

8. Common Mistakes

  • Writing directly to specific index names (POST /products/doc/1) instead of writing through an Index Alias (POST /products_live/doc/1).
  • Allowing dynamic mapping on multi-tenant APIs, resulting in mapping explosions that crash cluster state.
  • Creating 50 primary shards for a tiny 2GB index, causing severe over-sharding and JVM heap exhaustion.
  • Mapping all string fields as text only, breaking sorting, aggregations, and exact match filtering.
  • Indexing high-volume bulk historical data with refresh_interval=1s and number_of_replicas=1 enabled during ingestion.

9. Keywords

Search Indexing, Inverted Index, Debezium CDC, Index Alias, _reindex, _bulk API, text vs keyword, Over-Sharding, 20GB-50GB Shard Rule, ISM, ILM, Mapping Explosion, refresh_interval, Segment Merging, External Versioning.

10. Cheat Sheet

  • CDC Sync Rule: Never dual-write! PostgreSQL WAL \(\rightarrow\) Debezium \(\rightarrow\) Kafka \(\rightarrow\) Search Indexer.
  • Index Aliases: Always query and write through Index Aliases (products_live). Use _reindex + atomic alias swap for zero-downtime mapping updates.
  • Field Mappings: Use text for full-text search; use keyword for exact filtering, sorting, and aggregations.
  • Shard Sizing Rule: Target 20GB to 50GB per Primary Shard. Keep total shards \(< 20\) per 1GB JVM heap.
  • Bulk Ingestion: Use _bulk API (5MB-15MB payload size) + disable replicas + set refresh_interval=-1 during heavy backfills.
  • Disable Dynamic Mapping: Set "dynamic": "strict" in production templates to prevent Mapping Explosions.

11. Related Decisions