Skip to the solution
MediumSystem Design14 min read

What are blue-green and canary deployments?

285 views

The problem

Question presented to candidate: "What are blue-green and canary deployments? How do they differ from a normal rolling update, and when would you pick each one?"

What a strong answer should cover:

  • Blue-green: two full environments; deploy and test the idle one, switch all traffic at once, switch back to roll back. Fast, all-or-nothing, needs double capacity during the switch
  • Canary: send a small, growing share of real traffic to the new version, compare its metrics with the old version (the control), and promote or roll back automatically. Small blast radius, slower rollout
  • Rolling update as the baseline (replace instances a few at a time, as a Kubernetes Deployment does by default) and why its rollback is slow and its mixed fleet still hits everyone
  • What makes canaries work: sticky assignment of users, enough traffic for a statistically meaningful comparison, automated analysis gates, and metrics that capture the user experience (errors, latency, business KPIs)
  • The parts that break both: database schema changes (expand and contract), long-lived connections and sessions, background jobs and queues, and why feature flags complement deployments

Code / implementation expected: Optional — a simulation of a bad release under blue-green, rolling and canary, counting failed requests before rollback; the example does that, and shows a naive canary gate blocking a good release.

  • deployment

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

14 min read · 13 sections

Read the code

Blue-green vs rolling vs canary: blast radius of a bad release and the cost of a noisy canary gate
Open in playground
// Blue-green vs rolling vs canary for a bad release and a good release.
// Simulated traffic: 10,000 requests per minute from 5,000 users. Every
// version fails 0.2% of requests; the bad v2 also fails 3% more.
const RPM = 10000, USERS = 5000, INSTANCES = 10;
function mix(x) { // 32-bit integer hash (murmur3 finalizer): deterministic pseudo-randomness
  x = x >>> 0; x ^= x >>> 16; x = Math.imul(x, 0x85ebca6b) >>> 0; x ^= x >>> 13; x = Math.imul(x, 0xc2b2ae35) >>> 0; x ^= x >>> 16;
  return x >>> 0;
}
const bucket = (u) => mix(u + 7777) % 100;      // sticky canary bucket 0..99 per user
const roll = (i) => mix(i) % 10000;              // per-request dice, 0..9999
const baseFail = (i) => roll(i) < 20;            // 0.2% in every version
const bugFail = (i) => roll(i) >= 20 && roll(i) < 320; // 3% more in the bad v2

function simulate(strategy, buggy, rule) {
  let i = 0, bad = 0, minute = 0, alertRun = 0;
  const hit = new Set();
  let stage = 0, stageMin = 0, rolled = 0, rollingBack = false, detected = -1;
  const STAGES = [1, 5, 25, 50, 100], STAGE_LEN = 10;
  let cReq = 0, cErr = 0, ctlReq = 0, ctlErr = 0; // canary analysis window
  while (minute < 120) {
    minute++;
    let err = 0;
    for (let k = 0; k < RPM; k++, i++) {
      const u = mix(2 * i + 1) % USERS;
      let v2;
      if (strategy === "blue-green") v2 = detected < 0;
      else if (strategy === "rolling") v2 = i % INSTANCES < rolled;
      else v2 = detected < 0 && bucket(u) < STAGES[stage];
      const failed = baseFail(i) || (v2 && buggy && bugFail(i));
      if (v2 && buggy && bugFail(i)) { bad++; hit.add(u); }
      if (failed) err++;
      if (v2) { if (failed) cErr++; cReq++; } else { ctlReq++; if (failed) ctlErr++; }
    }
    if (strategy === "canary") {
      const naive = cReq >= 100 && cErr * ctlReq > 2 * ctlErr * cReq;                           // "twice the control rate"
      const margin = cReq >= 500 && cErr * ctlReq * 100 > ctlErr * cReq * 100 + cReq * ctlReq; // 1 point above control, 500+ samples
      if (detected < 0 && (rule === "naive" ? naive : margin)) { detected = minute; break; }
      if (++stageMin === STAGE_LEN && STAGES[stage] < 100) { stage++; stageMin = 0; cReq = cErr = ctlReq = ctlErr = 0; }
      if (STAGES[stage] === 100 && stageMin >= 1) return { bad, users: hit.size, detected, full: minute - 1 };
      continue;
    }
    alertRun = err * 100 > RPM ? alertRun + 1 : 0; // alert: error rate above 1% for 3 minutes in a row
    if (alertRun >= 3 && detected < 0) { detected = minute; rollingBack = true; if (strategy === "blue-green") break; }
    if (strategy === "rolling") {
      if (rollingBack) { rolled--; if (rolled === 0) break; }
      else if (rolled < INSTANCES) rolled++;
      else if (!buggy) return { bad, users: hit.size, detected, full: minute - 1 };
    } else if (!buggy) return { bad, users: hit.size, detected, full: 0 };
  }
  return { bad, users: hit.size, detected, full: -1 };
}

console.log("== bad release: v2 fails 3% of requests ==");
for (const [s, rule] of [["blue-green"], ["rolling"], ["canary", "margin"]]) {
  const r = simulate(s, true, rule);
  console.log(s.padEnd(11) + " detected in minute " + String(r.detected).padStart(2) + ", failed requests caused by v2: " + String(r.bad).padStart(4) + ", users hit: " + String(r.users).padStart(4) + " of " + USERS);
}
console.log("");
console.log("== good release: time until 100% of traffic is on v2 ==");
for (const [s, rule] of [["blue-green"], ["rolling"], ["canary", "margin"], ["canary", "naive"]]) {
  const r = simulate(s, false, rule);
  console.log((s + (rule ? " (" + rule + " gate)" : "")).padEnd(22) + " " + (r.full < 0 ? "never" : String(r.full).padStart(2) + " minutes") + ", false alarms: " + (r.detected < 0 ? "none" : "minute " + r.detected));
}

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 47 of 106 in System design

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All System design questions
Up nextMedium

What is a message queue and why use one?

Continue