What is a load balancer and what strategies does it use?

Q: What is a load balancer and what strategies does it use?

**Target Audience:** Beginners to system design — you know what a web server is; no networking background assumed. **Difficulty:** Medium **Verified with:** Node 22.22, Python 3.12, Go 1.24 and Java 21 (all four print identical output). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the exact words an interviewer expects. All numbers come from actually running the example; the **Verified** section quotes the full output. ## 1. Why This Even Matters — A Story First A supermarket has six checkout lanes. Without anyone directing traffic, shoppers pick the lane nearest the entrance and it backs up while lane six is empty. So the store puts a person at the front who points each shopper to a lane. How should that person choose? "Next lane in turn" is fair if every cart is the same size — but the shopper with two hundred items lands in lane three and everyone sent there after them waits. "The lane with t

The problem

Question presented to candidate: "Explain what a load balancer actually does in front of a fleet of servers. Then walk me through the main algorithms it can use to pick a server — round robin, least connections, hashing and so on — and tell me when each one is the right choice and when it goes wrong."

What a strong answer should cover:

The jobs: spread requests over a backend pool, run health checks and stop sending to failed servers, give clients one stable address, often terminate TLS, and drain connections during deploys
Layer 4 vs Layer 7: L4 balances TCP/UDP connections by address and port; L7 understands HTTP and can route by path, header or cookie, at more CPU per request
Static algorithms (round robin, weighted round robin, random, hash) vs dynamic ones (least connections / least outstanding requests, power of two choices), and the failure mode of each: round robin piles up on uneven request costs, plain hash mod N reshuffles almost every key when N changes
Why power of two random choices is the default in modern proxies: nearly as good as least connections with no global state, and it avoids herding when many balancers share stale load data
The load balancer itself must not be a single point of failure: redundant pairs, multiple addresses in DNS, anycast, or a managed multi-zone service

Code / implementation expected: Optional — describing the algorithms precisely is enough; implementing one or two (as in the example) is a strong signal.

load-balancing

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Beginners to system design — you know what a web server is; no networking background assumed. Difficulty: Medium Verified with: Node 22.22, Python 3.12, Go 1.24 and Java 21 (all four print identical output).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the exact words an interviewer expects. All numbers come from actually running the example; the Verified section quotes the full output.

1. Why This Even Matters — A Story First

A supermarket has six checkout lanes. Without anyone directing traffic, shoppers pick the lane nearest the entrance and it backs up while lane six is empty. So the store puts a person at the front who points each shopper to a lane.

How should that person choose? "Next lane in turn" is fair if every cart is the same size — but the shopper w

The solution is hidden so you can try first

16 min read · 12 sections

Read the code

Load-balancing strategies: smooth WRR, least connections, P2C, hash mod N

Open in playground

// Load-balancing strategies, simulated in-process (no real network).
// Part 1: smooth weighted round robin (the nginx algorithm).
// Part 2: 3 equal servers, uneven request durations: who piles up?
// Part 3: hash(key) mod N when a 5th server is added.

// Deterministic 32-bit LCG so every language draws the same "random" numbers.
let seed = 42;
function rand(n) {
  seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0;
  return (seed >>> 16) % n;
}

// Part 1 ----------------------------------------------------------------
function smoothWeighted(names, weights, picks) {
  const current = names.map(() => 0);
  const total = weights.reduce((a, b) => a + b, 0);
  const out = [];
  for (let p = 0; p < picks; p++) {
    let best = 0;
    for (let i = 0; i < names.length; i++) {
      current[i] += weights[i];
      if (current[i] > current[best]) best = i;
    }
    current[best] -= total;
    out.push(names[best]);
  }
  return out;
}
console.log("Part 1: smooth weighted round robin, weights A=5 B=1 C=1");
console.log("  picks: " + smoothWeighted(["A", "B", "C"], [5, 1, 1], 7).join(" "));

// Part 2 ----------------------------------------------------------------
const N = 3, REQUESTS = 300;
const duration = (i) => (i % 3 === 2 ? 30 : 3); // every 3rd request is slow

function simulate(name, choose) {
  seed = 42;
  const ends = Array.from({ length: N }, () => []); // end ticks of open requests
  const peak = new Array(N).fill(0);
  const work = new Array(N).fill(0);
  let rr = 0;
  for (let t = 0; t < REQUESTS; t++) {
    for (let s = 0; s < N; s++) ends[s] = ends[s].filter((e) => e > t);
    const active = ends.map((e) => e.length);
    const s = choose(active, () => rr++ % N);
    ends[s].push(t + duration(t));
    work[s] += duration(t);
    peak[s] = Math.max(peak[s], ends[s].length);
  }
  console.log(
    "  " + name.padEnd(18) +
    "peak open per server [" + peak.join(", ") + "]" +
    "  busy ticks [" + work.join(", ") + "]"
  );
}

function leastIndex(active) {
  let best = 0;
  for (let i = 1; i < active.length; i++) if (active[i] < active[best]) best = i;
  return best;
}

console.log("Part 2: 300 requests, 1 per tick, every 3rd takes 30 ticks, others 3");
simulate("round robin", (active, next) => next());
simulate("random", () => rand(N));
simulate("least connections", (active) => leastIndex(active));
simulate("power of two", (active) => {
  const a = rand(N);
  let b = rand(N - 1);
  if (b >= a) b++;
  return active[b] < active[a] ? b : a;
});

// Part 3 ----------------------------------------------------------------
function fnv1a(str) {
  let h = 2166136261;
  for (let i = 0; i < str.length; i++) {
    h ^= str.charCodeAt(i);
    h = Math.imul(h, 16777619) >>> 0;
  }
  return h;
}
let moved = 0;
const KEYS = 10000;
for (let i = 0; i < KEYS; i++) {
  const h = fnv1a("user-" + i);
  if (h % 4 !== h % 5) moved++;
}
console.log("Part 3: hash(key) mod N, going from 4 to 5 servers");
console.log("  keys that change server: " + moved + " of " + KEYS + " (" + (moved * 100 / KEYS).toFixed(1) + "%)");

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.