Progressive Delivery

Definition

Progressive delivery is the practice of releasing changes to production in stages, where each stage exposes the change to a controlled subset of traffic or users before the next stage. The defining property is that deployment is decoupled from release: code is shipped to production servers without being immediately visible to all users; visibility ramps up over time as confidence grows. Variants include canary release, blue-green deployment, ring deployment, dark launching, and feature-flagged rollouts.

Why It Matters

The risk of any change is unknown until that change runs under real production load on real production data with real users. Pre-production testing reduces risk but cannot eliminate it (→ Testing For Reliability). Progressive delivery makes the production environment itself the final test bench, limiting blast radius to a controllable fraction of traffic and providing fast rollback when an issue is detected. From Site Reliability Engineering ch. 36: "Canary rigorously even for 'low-risk' changes — the combination of a change with an unlikely configuration keyword triggered the crash-loop outage."

The economic argument is in Release It ch. 1: routine, low-risk deployment via progressive delivery transforms release from a $100K event (40-person deployment armies, midnight UAT, all-hands rollback) into a one-button operation that's safe enough to do during business hours.

The Core Decoupling: Deployment vs Release

This distinction is the cognitive shift that makes progressive delivery possible.

• Deployment: placing built artefacts on production infrastructure. Routine, scriptable, low-stakes.
• Release: exposing the deployed code to traffic. Reversible, observable, controlled.

Without this decoupling, every deployment is a release and every release is a binary risk event. With it, a deployed-but-not-released change can be observed in production (via shadow traffic, internal users, feature flags) and released gradually. From Building Evolutionary Architectures: feature toggles make deployment "boring" by removing the all-or-nothing characteristic.

Mechanisms

Canary Release

A small fraction of traffic is routed to the new version while the rest continues to the old version. Metrics are compared; if the canary holds (error rate, latency percentiles, business KPIs within bounds), traffic is incrementally shifted toward the new version.

From Site Reliability Engineering ch. 17: "Canary tests... expose new version to live traffic; exponential rollout with order-of-fault estimation (U=1: scales linearly with traffic; U=2: randomly damages data a future user will see; U=3: damaged data is also a valid identifier; most bugs are U=1)." The order-of-fault model justifies exponential rollout: U=1 bugs are found cheaply at small canary fractions, so most bugs are caught before broad exposure.

Operational requirements:

• Identical observability for canary and baseline (same dashboards, same alerts)
• Comparable workload routing (random sample, not "European traffic only")
• Automated abort on metric deviation (see Chaos Engineering's ChAP for an extreme case)
• Sufficient canary traffic to achieve statistical power within the rollout window

Blue-Green Deployment

Two identical production environments (blue = current, green = new). Deploy to green; smoke-test; switch the router to point at green. Old environment remains as instant rollback target.

Trade-offs vs canary:

• Faster cutover, simpler routing logic
• Either-or; no gradual ramp
• Requires double the production capacity during the switch
• Database/state migrations are harder (both environments may share the DB)
• Cleaner for stateless tiers; canary is usually better for stateful systems

Hybrid pattern: blue-green for the infrastructure, canary for the routing once green is live.

Ring / Geographic Deployment

A progression of deployment targets ordered by risk tolerance: internal users → beta users → small region → larger region → global. Standard practice at Google and Microsoft. From Site Reliability Engineering: "Progressive rollouts: staged, supervised, across geographies; roll back first, diagnose second."

The "roll back first, diagnose second" rule is the key cultural property: rollback is a routine response to deviation, not a defeat. Diagnosis happens once the production fire is out.

Dark Launching

Deploy the change and route real traffic to it, but discard the result and serve users from the old path. The new code path runs in production under real load before any user depends on it. Used heavily for performance validation, cache warming, and verifying that downstream dependencies can absorb the new traffic.

Distinct from canary: in dark launching, no users see the new behaviour; the goal is performance/load characterisation.

Parallel Run

Both old and new code run on every request; results are compared; one result is returned. The other is logged for offline correctness analysis. See Parallel Run for the full pattern — particularly useful for refactoring or platform migration where correctness equivalence is the primary risk.

GitHub's Scientist library is the canonical implementation. Distinct from canary (correctness comparison, not load-based metrics) and from dark launching (both paths execute and are compared, vs old serving and new being discarded).

Feature Flags

The release control lives in application code: a flag determines whether the new behaviour is visible to a given request. Allows runtime control, per-user/per-cohort targeting, and instant disable without redeployment. See Feature Flags for the full treatment. From Building Evolutionary Architectures: feature toggles decouple deployment from release at the application layer rather than the infrastructure layer.

The Pattern Spectrum

Real organisations combine multiple: infrastructure-level canary deployment with application-level feature flags is common.

Prerequisites

Progressive delivery rests on a stack of preceding capabilities. From Accelerate's Continuous Delivery Practices:

1. Trunk-based development — long-lived branches make progressive delivery harder, not easier
2. Automated deployment pipeline — manual deployment is incompatible with frequent staged rollout (→ Deployment Pipelines)
3. Comprehensive automated tests — pre-canary tests reduce the rate of canary aborts
4. Production observability — without metrics comparable across versions, "did the canary hold?" is unanswerable (→ Observability)
5. Loose architectural coupling — services that can be released independently (→ Architecture Quantum)
6. Database schema evolution practices — expand/contract, dual-write, never-rename (→ Evolutionary Database Design)

Anti-patterns

Canary without comparable observability. If the new version has different telemetry than the baseline, deviation detection is unreliable. Decision quality is bounded by metric quality.

Stale-flag accumulation. Feature flags that were never removed after rollout become dead conditionals. From Building Evolutionary Architectures: feature flags are technical debt with a use-by date. See Feature Flags for the lifecycle discipline.

Rollback fear. If teams treat rollback as failure, they will hesitate, diagnose first, and let the incident grow. The SRE principle is "roll back first, diagnose second" — rollback must be a low-status routine action.

Schema changes that can't be progressively delivered. A breaking database migration cannot be canaried — old code can't read new schema. Use expand/contract patterns (→ Evolutionary Database Design) so the schema accommodates both versions during the rollout window.

Insufficient canary traffic for signal. A 1% canary on a low-traffic service may never generate enough samples to detect a P99 regression within the rollout window. Either scale up the canary fraction, lengthen the window, or accept that low-traffic services need different deployment patterns.

Coupled rollback. If the new version writes data the old version can't read, rolling back the code is not enough — you have to roll back data too. Design rollback paths during feature design, not during the incident.

How Different Sources Treat It

Related Concepts

• Feature Flags — application-layer release control
• Deployment Pipelines — the automation substrate
• Continuous Delivery Practices — the broader capability set
• Parallel Run — correctness-comparison variant
• Strangler Fig — long-running architectural progressive delivery
• Evolutionary Database Design — schema patterns that survive partial rollout
• Observability — prerequisite for canary decisions
• Chaos Engineering — ChAP as automated canary + experiment combined
• Error Budgets — budget exhaustion as a release-rate brake
• Four Key Metrics — deployment frequency, change failure rate, MTTR all influenced by progressive delivery
• Migration Pattern Selection — progressive delivery as the umbrella for migration-specific rollout patterns

Key Quotes

"Canary rigorously even for 'low-risk' changes." — Site Reliability Engineering ch. 17

"Roll back first, diagnose second." — Site Reliability Engineering

"$50,000 invested in a zero-downtime deployment pipeline returns 18× over five years." — Release It ch. 1