Downtime during deployments is a solved problem, in theory. In practice, most engineering teams still experience deployment-related incidents because they chose a strategy that doesn't match their architecture, or they implemented the right strategy with incomplete safeguards. Here's how to think about zero-downtime deployments from first principles.
Blue-Green Deployments
Blue-green is the simplest model to reason about. You maintain two identical production environments. At any given time, one (blue) serves live traffic while the other (green) sits idle. You deploy the new version to the idle environment, run your smoke tests, and then switch the load balancer to route traffic to it. If something goes wrong, you switch back.
The appeal is instant rollback. The cost is infrastructure: you're paying for two full production environments. For stateless applications behind a load balancer, blue-green is often the right default. For systems with persistent connections, long-running jobs, or complex state, the cutover can introduce subtle issues that require careful handling of in-flight requests.
Database migrations deserve special attention in blue-green setups. Both environments must be compatible with the same database state, which means migrations need to be backward-compatible. The expand-and-contract pattern, where you add the new schema first, deploy the new code, then remove the old schema, is essential.
Canary Releases
Canary deployments route a small percentage of traffic to the new version while the majority continues hitting the stable release. You monitor error rates, latency, and business metrics on the canary, and gradually increase its traffic share if everything looks healthy.
This approach excels at catching issues that only manifest under real production load, such as race conditions, performance regressions, and integration failures that test environments miss. The tradeoff is complexity. You need traffic splitting at the load balancer or service mesh layer, robust observability to compare canary metrics against the baseline, and automated rollback triggers that respond faster than a human can.
Canary releases also introduce version skew. For a period of time, two versions of your service are handling requests simultaneously. Your system must tolerate this gracefully, particularly around API contracts and cached data.
Rolling Updates
Rolling updates replace instances incrementally. Kubernetes performs rolling updates by default: it spins up new pods while terminating old ones, respecting configured surge and unavailability limits. This requires no duplicate infrastructure and works well for horizontally scaled stateless services.
The risk is more distributed. Unlike blue-green, there's no single cutover moment. If the new version has a subtle bug, it propagates gradually across your fleet. Readiness probes and health checks are your primary defense, and they must be comprehensive enough to catch real failures, not just "the process started successfully."
Rolling updates also require careful configuration of pod disruption budgets and graceful shutdown handling. Connections must be drained before termination, and the application must handle SIGTERM signals properly to finish in-flight work.
Choosing the Right Strategy
The decision depends on three factors: your tolerance for infrastructure cost, the complexity of your state management, and how quickly you need to detect regressions.
Blue-green optimizes for rollback speed. Canary optimizes for risk detection. Rolling updates optimize for resource efficiency. Many mature organizations use a combination: canary releases for critical services, rolling updates for internal tooling, and blue-green for database-heavy systems where rollback speed is paramount.
Whatever you choose, the deployment pipeline must be tested as rigorously as the application code. A deployment strategy that has never been rolled back in production is a deployment strategy that has never been tested.
Ready to build a deployment pipeline that doesn't wake anyone up at night? Start the conversation.