Skip to content

Blue-Green vs Canary vs Rolling Deployment

1. Decision

Choose Rolling Deployment as the low-cost default for stateless microservices tolerating temporary version co-existence; choose Blue-Green Deployment for high-risk releases requiring instantaneous zero-downtime cutover and immediate 1-step rollback capability; choose Canary Deployment for mission-critical applications requiring progressive traffic validation (e.g. 1% \(\rightarrow\) 5% \(\rightarrow\) 100%) against real production metrics before full rollout.

2. 30-Second Answer

Deployment strategies dictate how new software versions replace legacy code in production. Rolling Deployment replaces old instances with new instances incrementally (pod-by-pod). It requires zero extra hardware capacity, but exposes production to mixed version co-existence during the rollout. Blue-Green Deployment provisions two complete identical physical environments: "Blue" (live production) and "Green" (new version). Once Green passes validation, the router switches 100% of traffic instantly to Green. It provides instant rollback, but doubles infrastructure hardware cost. Canary Deployment shifts a tiny fraction of live user traffic (e.g., 1-5%) to a small new version subset, monitoring metrics (error rate, p99 latency) before automatically promoting to 100% or rolling back.

3. Decision Matrix

Option Best For Avoid When Hardware Cost Rollback Speed Version Co-existence Operational Complexity
Rolling Deployment (K8s Default) Standard stateless microservices, routine updates, systems with limited hardware budgets Applications where old and new versions cannot run simultaneously due to breaking database schema changes Low (Zero additional hardware required) Slow (Must incrementally reverse rolling update pod-by-pod) High (Mixed versions active simultaneously during rollout) Low
Blue-Green Deployment Mission-critical monolithic releases, major API updates, zero-downtime instant cutovers Tight hardware budgets (requires 2x full cluster capacity), long-lived stateful database connections High (Requires 100% duplicate parallel environment capacity) Instantaneous (Single router / DNS switch) Zero (Environment is 100% Blue or 100% Green) Medium
Canary Deployment (Flagger / Argo Rollouts) High-scale mission-critical systems, validating new code against real production user traffic, automated metric analysis Applications lacking automated metrics and observability infrastructure Low to Medium (Provisions small canary pod subset e.g. +5-10%) Fast & Automated (Rolls back automatically if error metrics spike) High (Canary subset runs alongside production baseline) High (Requires Prometheus metrics + Argo Rollouts)

4. Selection Rules

Requirement Best Choice Rationale / Example
Low-Cost Standard Microservice Update Rolling Deployment Incrementally updates 25% of Kubernetes pods at a time (maxSurge: 25%)
Instantaneous Single-Step Rollback Gate Blue-Green Deployment If Green fails post-switch, flip load balancer back to Blue instantly
Progressive Traffic Validation against Real Metrics Canary Deployment Shift 2% traffic to Canary; evaluate Prometheus HTTP 5xx error rate for 10 mins
Monolithic Database Schema Migration Release Blue-Green / Canary Pair deployment with Non-Breaking Backward-Compatible Database Schema Migrations

5. Tradeoffs

  1. Hardware Cost vs Deployment Safety
  2. Gain (Blue-Green): 100% isolated green testing before live traffic cutover.
  3. Lose (Blue-Green): Requires 2x total cluster compute and memory capacity.
  4. Instant Rollback vs Incremental Rollback
  5. Gain (Blue-Green): Flipping router back to Blue takes <1 second.
  6. Lose (Rolling): Reverting a failed rolling update requires rolling backward pod-by-pod, prolonging outage.
  7. Real Production Validation vs Synthetic Testing
  8. Gain (Canary): Validates real user traffic patterns and edge-case inputs on 1% traffic before full blast radius.
  9. Lose (Canary): Complex traffic routing setup (Istio / Linkerd / Argo Rollouts).
  10. Version Co-existence & Schema Evolution
  11. Gain (Blue-Green): Eliminates mixed-version code execution in production.
  12. Lose (Rolling/Canary): Database schema MUST support both old and new code simultaneously.
  13. Automation & Observability Dependency
  14. Gain (Canary): Argo Rollouts automatically aborts and rolls back if p99 latency spikes above 200ms.
  15. Lose (Canary): Requires highly accurate Datadog/Prometheus metric pipelines.

6. Interview Counter Questions

  1. How do Kubernetes maxSurge and maxUnavailable parameters control Rolling Deployments?
  2. What is the fundamental requirement for Database Schemas when executing Blue-Green or Canary deployments?
  3. How does Argo Rollouts or Flagger automate Canary Deployments on Kubernetes?
  4. How do you handle long-lived WebSocket connections during a Blue-Green traffic cutover?
  5. What is the difference between Feature Flags (Dark Launching) and Canary Deployments?
  6. How does a Service Mesh (Istio / Envoy) execute precise percentage-based traffic splitting for Canaries?
  7. What happens if a database schema change is destructive (e.g. dropping a column) during a Rolling Deployment?
  8. How do you test the Green environment in Blue-Green deployment before switching public traffic?
  9. When would a Canary deployment fail to catch a critical bug?
  10. What is a Recreate Deployment Strategy and when is it acceptable?

7. Gold Standard Answers

  1. maxSurge: Max pods created above desired pod count during rollout (e.g. 25%). maxUnavailable: Max pods that can be unavailable during rollout (e.g. 25%). Controls rollout speed and capacity.
  2. Database schemas MUST be Backward-Compatible. Both old (Blue/Baseline) and new (Green/Canary) code versions access the database simultaneously. Use Expand-Contract (Parallel Change) schema patterns.
  3. Argo Rollouts replaces K8s Deployment. It defines steps (setWeight: 5 \(\rightarrow\) pause: {duration: 10m}) and queries Prometheus metrics (AnalysisTemplate). If error rates exceed thresholds, it aborts automatically.
  4. Issue a graceful shutdown signal (SIGTERM / WS Close frame). Clients receive the close frame and automatically reconnect using exponential backoff + jitter, landing on the new Blue/Green load balancer target.
  5. Canary: Deploys new code binary to a small server subset, splitting network traffic at the router layer. Feature Flags: Deploys code to all servers, toggling features on/off at runtime per user ID.
  6. Istio VirtualService defines HTTP weight routing: route: [{destination: v1, weight: 95}, {destination: v2, weight: 5}]. Envoy sidecars split traffic precisely at the L7 proxy layer.
  7. Dropping a column breaks legacy running pods immediately! Always use the 4-phase Expand-Contract pattern: (1) Add new column, (2) Write to both, (3) Migrate code, (4) Drop old column months later.
  8. Use a private host header (Host: green.internal) or test cookie routed by the load balancer to the Green environment, allowing QA teams to run automated integration tests before flipping public traffic.
  9. If the bug occurs only under specific high-concurrency loads, background async cron jobs, or rare edge-case user inputs that did not hit the 1% canary traffic sample during the evaluation window.
  10. Recreate terminates 100% of old pods before launching new pods. It incurs brief downtime, but is acceptable for internal dev environments or monolithic apps that cannot tolerate version co-existence.

8. Common Mistakes

  • Executing destructive database schema changes (e.g. dropping/renaming columns) before a Rolling or Canary rollout finishes.
  • Implementing Blue-Green deployment without automated health checks on the Green environment before cutover.
  • Running Canary deployments manually without automated metric analysis (Argo Rollouts/Flagger), requiring human monitoring.
  • Setting maxUnavailable: 100% in Kubernetes Rolling Updates, causing complete service downtime during deployments.
  • Failing to handle long-lived WebSocket / gRPC persistent connections during environment traffic shifts.

9. Keywords

Blue-Green Deployment, Canary Deployment, Rolling Deployment, Recreate Deployment, maxSurge, maxUnavailable, Argo Rollouts, Flagger, Expand-Contract Pattern, Backward Compatibility, Feature Flags, Dark Launching, Istio VirtualService, Traffic Splitting, AnalysisTemplate.

10. Cheat Sheet

  • Rolling: K8s default. Pod-by-pod replacement (maxSurge: 25%). Low cost, but version co-existence occurs.
  • Blue-Green: 2 identical environments (Blue=Live, Green=New). 100% instant cutover/rollback via router. Requires 2x hardware capacity.
  • Canary: Progressive traffic shift (1% \(\rightarrow\) 5% \(\rightarrow\) 100%) validated against real Prometheus metrics via Argo Rollouts. Minimal blast radius.
  • Database Rule: Database schema changes MUST be Backward-Compatible (Expand-Contract pattern) for ALL deployment strategies!
  • Feature Flags vs Canary: Canary = Traffic split at router layer; Feature Flags = Runtime code toggle in app layer.
  • Automatic Rollback: Configure Argo Rollouts AnalysisTemplate to abort Canary deployment if HTTP 5xx error rate exceeds 1%.

11. Related Decisions