Skip to content

Blob Storage vs Metadata Database

1. Decision

Implement a decoupled dual-storage architecture by choosing Blob Storage (S3, GCS) for storing heavy binary content payloads (video, audio, images, documents) and a Metadata Database (PostgreSQL, Cassandra, DynamoDB) for indexing object attributes (file names, permissions, tags, owner IDs, byte offsets, S3 URI references).

2. 30-Second Answer

In high-scale system designs (e.g., YouTube, Netflix, Google Drive, Dropbox), raw binary content and file metadata must never be stored in the same database engine. Blob Storage (AWS S3, Azure Blob Storage) is optimized for storing immutable, flat binary payloads with infinite scale, multi-AZ durability, and cheap cost per gigabyte, but lacks multi-attribute searching, indexing, or relational authorization controls. The Metadata Database (PostgreSQL, DynamoDB, Cassandra) stores lightweight entity metadata (file ID, owner ID, size, ACL permissions, checksum, upload status, S3 storage path), providing sub-millisecond query filtering, relational access control, and transactional consistency. Decoupling payload from metadata allows both tiers to scale independently.

3. Decision Matrix

Option Best For Avoid When Latency Scalability Consistency Operational Complexity
Blob Storage (S3/GCS) Raw binary file content (videos, images, audio, PDFs, database backups) Storing searchable attributes, high-frequency metadata filtering, small key-value state Medium (20-50ms TTFB) Virtually Unlimited Strong Read-After-Write Low
Metadata Database (Postgres/DynamoDB) Storing file attributes, ACL permissions, tags, search indexes, upload status, S3 URIs Storing heavy binary file payloads (>1MB) Low (<5ms point lookups) Scale-up (SQL); Linear Scale-out (NoSQL) Strict ACID (SQL); Eventual/Strong (NoSQL) Medium

4. Selection Rules

Requirement Best Choice Rationale / Example
Raw Binary Storage (>100KB Payload) Blob Storage Offloads file payload disk I/O and network bandwidth from application infrastructure
Fast Sub-Millisecond Attribute Search Metadata DB B-Tree / Secondary index queries filter files by user, tag, or creation date
File Access Control List (ACL) Permissions Metadata DB Enforce user authorization policies, tenant isolation, and RBAC before returning file URLs
Infinite Content Scale at Low Cost Blob Storage S3 provides petabyte-scale storage at ~$0.023/GB/mo without disk provisioning
File Versioning & Deduplication Metadata Metadata DB Store SHA-256 content hashes to point identical duplicate files to a single S3 object
Direct Client Stream Processing Blob Storage Stream video bytes directly to CDN (CloudFront) using byte-range headers

5. Tradeoffs

  1. Decoupled Architecture Scalability vs Distributed State Coordination
  2. Gain: Metadata DB remains tiny, fast, and easily cacheable; Blob Storage handles heavy throughput traffic.
  3. Lose: Potential consistency drift if Blob upload succeeds but Metadata DB write fails (or vice versa).
  4. Metadata Query Expressiveness vs Raw Payload Storage
  5. Gain: Query, filter, and sort file attributes instantly without downloading binary file contents.
  6. Lose: Must execute 2-step lookup: (1) Fetch S3 URI & ACL from Metadata DB, (2) Fetch content from S3.
  7. Storage Cost Economics
  8. Gain: High-volume raw bytes reside on cheap cloud object storage; expensive database SSD disk stores only bytes of text metadata.
  9. Lose: Incur minor network API request costs between compute nodes, metadata DB, and S3.
  10. Deduplication Capability
  11. Gain: Storing content hashes (SHA-256) in Metadata DB allows multiple metadata records to point to a single S3 blob key, saving terabytes of storage.
  12. Lose: Deleting a metadata record requires reference-counting logic before safely deleting the underlying S3 blob object.
  13. Authorization Enforcement
  14. Gain: Metadata DB evaluates user security permissions before generating short-lived signed S3 URLs.
  15. Lose: S3 buckets must be kept completely private to prevent unauthorized direct bucket access bypassing the metadata layer.

6. Interview Counter Questions

  1. How do you design Google Drive / Dropbox file storage architecture using Blob Storage and Metadata DB?
  2. How do you handle file deduplication across millions of user uploads using SHA-256 content hashing?
  3. How do you prevent orphan blobs in S3 if the file upload succeeds but the Metadata DB transaction fails?
  4. What happens when a user deletes a file that is shared among 10 users (Reference Counting in Metadata DB)?
  5. How does CDN (CloudFront) integrate with Blob Storage and Metadata DB for secure video streaming?
  6. How do you handle large file chunking (e.g., 10GB file split into 4MB chunks) in Blob Storage vs Metadata DB?
  7. What database engine should you choose for the Metadata DB (PostgreSQL vs DynamoDB vs Cassandra)?
  8. How do you implement instant folder renaming in Google Drive without updating millions of metadata rows?
  9. How do you handle multipart upload resumption after network failure?
  10. How do you search file contents (Full-Text Search) when file metadata is in Postgres and raw content is in S3?

7. Gold Standard Answers

  1. Decouple metadata (File ID, Parent Folder ID, Owner, Permissions, Hash, S3 URL) stored in PostgreSQL/Cassandra from raw file chunks stored in AWS S3. API servers check permissions in DB, then return S3 Presigned URLs for direct client downloads.
  2. Calculate SHA-256 hash of the file payload on client side before upload. Query Metadata DB for content_hash. If match exists, skip S3 upload entirely and create a new metadata record pointing to the existing S3 blob key (deduplication).
  3. Upload files to S3 under an unconfirmed prefix (/pending/<upload_id>). Upon DB transaction completion, mark status as ACTIVE and move S3 object to /files/. A daily cron job deletes /pending/ objects unconfirmed after 24 hours.
  4. Use a reference count column (ref_count) or mapping table in Metadata DB. File deletion decrements ref_count. Only when ref_count == 0 is the underlying S3 blob key enqueued for deletion.
  5. Client requests media access from API Server. API Server verifies ACL in Metadata DB and generates a signed CloudFront URL with a short TTL (e.g., 15 mins). CloudFront fetches and caches video segments from private S3 origin.
  6. Split 10GB file into 4MB chunks. Metadata DB maintains a chunks table mapping file_id, chunk_index, chunk_hash, and s3_chunk_url. Client uploads chunks in parallel; Metadata DB reassembles logical file view.
  7. Choose PostgreSQL if strict hierarchical folder relationships, ACID updates, and complex ACL joins dominate. Choose DynamoDB/Cassandra if scale exceeds >100k metadata write QPS with simple key-value lookups (user_id + file_id).
  8. Store folder structure using Materialized Path or Ltree in PostgreSQL, or use absolute parent_folder_id pointers. Renaming a folder updates a single row entry for that folder in Metadata DB; child file records remain untouched.
  9. Metadata DB stores upload_id and uploaded chunk_indices. Upon network failure and client reconnect, client queries Metadata DB for missing chunk indices and resumes uploading only un-received chunks to S3.
  10. S3 Event Notification triggers asynchronous text extraction pipeline (AWS Textract / Tika inside Lambda), indexing extracted raw text content into Elasticsearch/OpenSearch along with the file_id for full-text search querying.

8. Common Mistakes

  • Storing binary file data directly inside database tables alongside metadata columns.
  • Making S3 buckets publicly readable to simplify client access, completely bypassing database ACL security checks.
  • Synchronously uploading 5GB files to S3 inside an API HTTP request thread while holding an open database transaction lock.
  • Deleting S3 blob objects immediately upon user file deletion without checking reference counts for duplicate files.
  • Updating millions of child metadata records in a database when a parent folder is renamed or moved.

9. Keywords

Blob Storage, Metadata Database, Decoupled Architecture, Presigned URL, SHA-256 Deduplication, Reference Counting, CloudFront Signed URL, Chunking, Materialized Path, Orphan Blobs, S3 Event Notification, AWS Textract.

10. Cheat Sheet

  • Core Rule: Raw Bytes (>100KB) \(\rightarrow\) Blob Storage (S3); Attributes, ACLs, Search & S3 Paths \(\rightarrow\) Metadata Database.
  • 2-Step Access: API checks permissions in Metadata DB \(\rightarrow\) returns signed S3/CloudFront URL \(\rightarrow\) client fetches content directly from S3.
  • Deduplication: Calculate content hash (SHA-256). If hash exists in Metadata DB, link new metadata record to existing S3 blob.
  • Deletion Safety: Use ref_count in Metadata DB. Delete S3 object only when ref_count == 0.
  • Folder Modeling: Store parent pointers (parent_id) in Metadata DB. Folder rename = 1 row update in DB.
  • Large Files: Split large files into 4MB chunks. Store chunk map in Metadata DB; upload chunks in parallel to S3.

11. Related Decisions