Skip to the solution
MediumSystem Design16 min read

What are the three pillars of observability?

836 views

The problem

Question presented to candidate: "People talk about the three pillars of observability. What are they, what is each one good and bad at, and how would you use them together when checkout starts failing in production?"

What a strong answer should cover:

  • Metrics: cheap numeric time series aggregated over labels; best for dashboards and alerting, weak at explaining one request; label cardinality limits the detail they can carry
  • Logs: timestamped event records, ideally structured (key=value or JSON); best for the why of one event; expensive in volume, so levels, sampling and retention matter
  • Traces: one request across services as a tree of spans sharing a trace ID; best for where time or errors happen; usually sampled
  • Correlation is the point: the same trace ID in spans and logs, exemplars from metric buckets to traces, shared resource attributes (service, version) so you can pivot from an alert to a trace to its logs
  • Monitoring vs observability: known questions vs new questions without shipping new code; OpenTelemetry as the vendor-neutral way to produce all three
  • Beyond the pillars: continuous profiles (an OpenTelemetry signal in Alpha since March 2026), wide structured events, and SLOs as the thing the signals serve

Code / implementation expected: Optional — explaining a real investigation flow matters more; the example walks one simulated incident through all three signals.

  • observability

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

16 min read · 12 sections

Read the code

One incident through all three pillars: a metric alert, a trace waterfall, the matching logs, and the cost of each signal
Open in playground
// Three pillars on one incident: metrics say WHAT, traces say WHERE, logs say WHY.
// 12 simulated minutes, 100 checkout requests per minute; payments v2 ships at minute 6.
let seed = 42;
function rand() { seed = (seed * 48271) % 2147483647; return seed / 2147483647; }
const BUCKETS = [50, 100, 250, 500, 1000, 2500];
const pad = (n, w) => String(n).padStart(w, " ");
const clock = (m, s) => `00:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`;

const traces = [], logs = [], perMin = [];
for (let m = 0; m < 12; m++) perMin.push({ n: 0, err: 0, hist: BUCKETS.map(() => 0) });
for (let i = 0; i < 1200; i++) {
  const minute = Math.floor(i / 100), sec = Math.floor(((i % 100) * 6) / 10);
  const traceId = "4bf92f3577b34da6" + (i + 1).toString(16).padStart(16, "0");
  const card = rand() < 0.2 ? "amex" : rand() < 0.5 ? "visa" : "mastercard";
  const version = minute >= 6 ? "v2" : "v1";
  const gw = 3 + Math.floor(rand() * 5), co = 10 + Math.floor(rand() * 10), pay = 15 + Math.floor(rand() * 10);
  let db = 8 + Math.floor(rand() * 8), error = false;
  if (version === "v2" && card === "amex") { db = 700 + Math.floor(rand() * 600); if (db > 1000) { db = 1000; error = true; } }
  const spans = [
    { name: "gateway POST /pay", start: 0, dur: gw + co + pay + db },
    { name: "checkout placeOrder", start: 1, dur: co + pay + db },
    { name: "payments charge", start: 2, dur: pay + db },
    { name: "payments db.query", start: 3, dur: db },
  ];
  traces.push({ traceId, card, version, error, total: spans[0].dur, spans });
  const pm = perMin[minute]; pm.n++; if (error) pm.err++;
  pm.hist[BUCKETS.findIndex((b) => spans[0].dur <= b)]++;
  const at = clock(minute, sec);
  if (error) {
    logs.push(`${at} level=ERROR service=payments trace_id=${traceId} msg="db timeout after 1000ms" query=select_card_limits card_type=${card} version=${version}`);
    logs.push(`${at} level=ERROR service=checkout trace_id=${traceId} msg="payment failed" status=502`);
  } else {
    logs.push(`${at} level=INFO service=payments trace_id=${traceId} msg="charge ok" card_type=${card} version=${version}`);
  }
}

console.log("== METRICS: the dashboard (labels: service, route, status) ==");
console.log("minute  requests  errors  error%  p99 bucket");
let breached = 0, fired = -1;
perMin.forEach((pm, m) => {
  const need = Math.ceil(pm.n * 0.99); let cum = 0, p99 = 0;
  for (let b = 0; b < BUCKETS.length; b++) { cum += pm.hist[b]; if (cum >= need) { p99 = BUCKETS[b]; break; } }
  const rate = (pm.err * 100) / pm.n;
  console.log(`${pad(m, 6)}  ${pad(pm.n, 8)}  ${pad(pm.err, 6)}  ${pad(rate.toFixed(1), 5)}%  <= ${p99} ms`);
  breached = rate > 2 ? breached + 1 : 0;
  if (breached === 2 && fired < 0) fired = m;
});
console.log(`ALERT CheckoutErrorRate (error% > 2 for 2 minutes) fired at end of minute ${fired}`);
console.log("The metric says WHAT and WHEN. It cannot say which request or why.");

console.log("\n== TRACES: where did the time go? ==");
const bad = traces.filter((t) => t.error);
console.log(`error traces: ${bad.length} of ${traces.length}`);
const t0 = bad[0];
console.log(`waterfall for trace ${t0.traceId} (${t0.total} ms):`);
for (const s of t0.spans) {
  const share = Math.round((s.dur * 100) / t0.total);
  console.log(`  ${s.name.padEnd(20)} start +${s.start} ms  ${pad(s.dur, 5)} ms  ${pad(share, 3)}% of request`);
}
const self = t0.spans[3].dur;
console.log(`slowest leaf span: payments db.query = ${self} ms (${Math.round((self * 100) / t0.total)}% of the request)`);

console.log("\n== LOGS: why? (filter by the trace_id from the trace) ==");
for (const line of logs.filter((l) => l.includes(t0.traceId))) console.log("  " + line);

console.log("\n== PIVOT: slice error traces by attributes the metric does not carry ==");
const byCard = {}, byVer = {};
for (const t of bad) { byCard[t.card] = (byCard[t.card] || 0) + 1; byVer[t.version] = (byVer[t.version] || 0) + 1; }
console.log("errors by card_type:", ["amex", "visa", "mastercard"].map((c) => `${c}=${byCard[c] || 0}`).join(" "));
console.log("errors by version:  ", ["v1", "v2"].map((v) => `${v}=${byVer[v] || 0}`).join(" "));
const amexV2 = traces.filter((t) => t.card === "amex" && t.version === "v2");
console.log(`amex on v2: ${amexV2.length} requests, ${amexV2.filter((t) => t.error).length} failed`);

console.log("\n== VOLUME of each signal for these 12 minutes ==");
const series = 2 * (BUCKETS.length + 1 + 2), samples = series * 4 * 12;
const logBytes = logs.reduce((a, l) => a + l.length + 1, 0), spanCount = traces.length * 4;
console.log(`metrics: ${series} series x 4 scrapes/min x 12 min = ${samples} samples, ${samples * 16} bytes at 16 B/sample`);
console.log(`logs:    ${logs.length} lines, ${logBytes} bytes`);
console.log(`traces:  ${spanCount} spans, ${spanCount * 400} bytes at an assumed 400 B/span`);
console.log(`metric cost per request: ${((samples * 16) / traces.length).toFixed(1)} B; log cost: ${(logBytes / traces.length).toFixed(1)} B; trace cost: ${(spanCount * 400) / traces.length} B`);

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 18 of 106 in System design

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All System design questions
Up nextMedium

What is consistent hashing?

Continue