What is chain-of-thought prompting and when does it actually help?

Q: What is chain-of-thought prompting and when does it actually help?

**Target Audience:** Engineers who have heard "add let us think step by step" as a magic fix and want to know when it is worth its tokens. **Difficulty:** Medium **Verified with:** Node 22.22.0 and Python 3.12.3 (a stub model with a fixed compute budget per token, plus a real answer extractor; no real model is called). Paper results are cited inline, checked October 2026. > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First Ask a friend "what is 17 times 24?" and demand the answer in one second. Most people guess. Give them a pencil and they write 17 × 20 = 340, 17 × 4 = 68, 340 + 68 = 408, and get it right. They did not get smarter. They used the paper to hold the intermediate results, so each small step was easy.

The problem

Question presented to candidate: "What is chain-of-thought prompting? Why does asking a model to reason step by step improve some answers, when does it not help at all, and how do you use it in a production system, especially now that reasoning models exist?"

What a strong answer should cover:

Definition: prompting the model to write intermediate reasoning before the final answer, either with worked examples that show reasoning (few-shot CoT, Wei et al. 2022) or a trigger phrase such as "Let's think step by step" (zero-shot CoT, Kojima et al. 2022)
Why it works: a model does a fixed amount of computation per generated token; writing intermediate results into its own context lets it spend more serial computation on a problem (more test-time compute)
When it helps and when it does not: large gains on multi-step maths, logic and symbolic tasks; little or no gain on lookups, simple classification and most knowledge questions (Sprague et al. 2024), while always costing output tokens and latency
Production patterns: reason first, answer last; parse the final answer from a fixed marker or a structured field placed after the reasoning; self-consistency (sample several chains, majority vote) for extra accuracy at extra cost
Caveats: the written reasoning is not guaranteed to be the real cause of the answer (unfaithful CoT), and reasoning models already think internally, so explicit "think step by step" instructions are usually unnecessary for them

Code / implementation expected: Optional — answer extraction from a reasoning trace, or a comparison of direct vs CoT accuracy and token cost on a small set.

chain-of-thought
reasoning
prompting

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers who have heard "add let us think step by step" as a magic fix and want to know when it is worth its tokens. Difficulty: Medium Verified with: Node 22.22.0 and Python 3.12.3 (a stub model with a fixed compute budget per token, plus a real answer extractor; no real model is called). Paper results are cited inline, checked October 2026.

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

Ask a friend "what is 17 times 24?" and demand the answer in one second. Most people guess. Give them a pencil and they write 17 × 20 = 340, 17 × 4 = 68, 340 + 68 = 408, and get it right. They did

The solution is hidden so you can try first

13 min read · 10 sections

Read the code

Direct answer vs chain-of-thought with a compute-limited stub model, plus answer extraction

Open in playground

// Chain-of-thought with a stub model that has a fixed amount of "compute per
// token": it can carry out at most 2 arithmetic operations before it must write
// a token. Answering directly gives it one chance; writing steps gives it many.
const TASKS = [
  { q: "A shop has 40 mugs and sells 15.", start: 40, ops: [["-", 15]] },
  { q: "Tickets cost 12 each; buy 7.", start: 12, ops: [["*", 7]] },
  { q: "Start with 50, add 30, then halve it.", start: 50, ops: [["+", 30], ["/", 2]] },
  { q: "3 boxes of 24 pens, 10 broken, rest shared by 2 teams.", start: 3, ops: [["*", 24], ["-", 10], ["/", 2]] },
  { q: "120 users, 25% churn, then 30 sign up, then double.", start: 120, ops: [["*", 3], ["/", 4], ["+", 30], ["*", 2]] },
  { q: "Budget 1000, spend 280 and 170, split by 5, add 15.", start: 1000, ops: [["-", 280], ["-", 170], ["/", 5], ["+", 15]] },
];
const BUDGET = 2; // operations the stub can do "in its head" before emitting a token
const apply = (v, [op, n]) => (op === "+" ? v + n : op === "-" ? v - n : op === "*" ? v * n : v / n);
const truth = (t) => t.ops.reduce(apply, t.start);

// Direct answer: the stub must produce the number now, so it only gets BUDGET ops.
function directAnswer(t) {
  const v = t.ops.slice(0, BUDGET).reduce(apply, t.start);
  return `Answer: ${v}`;
}
// Chain of thought: each written line is new context the next step can read.
function cotAnswer(t) {
  const lines = [];
  let v = t.start;
  for (let i = 0; i < t.ops.length; i += BUDGET) {
    for (const op of t.ops.slice(i, i + BUDGET)) {
      const before = v;
      v = apply(v, op);
      lines.push(`Step ${lines.length + 1}: ${before} ${op[0]} ${op[1]} = ${v}`);
    }
  }
  lines.push(`Answer: ${v}`);
  return lines.join("\n");
}
// Reason first, answer last: take the LAST "Answer:" line, or nothing.
function extract(text) {
  const all = [...text.matchAll(/Answer:\s*(-?\d+(?:\.\d+)?)/g)];
  return all.length ? Number(all[all.length - 1][1]) : null;
}
const tok = (s) => Math.ceil(s.length / 4); // rough output-token estimate

console.log("steps  truth  direct  cot   tok direct  tok cot  task");
const buckets = { "1-2 steps": [0, 0, 0, 0, 0], "3+ steps": [0, 0, 0, 0, 0] };
for (const t of TASKS) {
  const d = directAnswer(t), c = cotAnswer(t), want = truth(t);
  const dOk = extract(d) === want, cOk = extract(c) === want;
  const b = buckets[t.ops.length <= 2 ? "1-2 steps" : "3+ steps"];
  b[0]++; b[1] += dOk; b[2] += cOk; b[3] += tok(d); b[4] += tok(c);
  console.log(`${String(t.ops.length).padStart(5)}  ${String(want).padStart(5)}  ${(dOk ? "ok" : "WRONG").padStart(6)}  ${(cOk ? "ok" : "WRONG").padStart(4)}  ${String(tok(d)).padStart(10)}  ${String(tok(c)).padStart(7)}  ${t.q}`);
}
console.log("\nbucket     n  direct acc  cot acc  out tokens direct -> cot");
for (const [name, [n, dOk, cOk, dt, ct]] of Object.entries(buckets)) {
  console.log(`${name.padEnd(9)}  ${n}  ${`${dOk}/${n}`.padStart(10)}  ${`${cOk}/${n}`.padStart(7)}  ${dt} -> ${ct} (${(ct / dt).toFixed(1)}x)`);
}
console.log("\nCoT output for the 4-step budget task:\n" + cotAnswer(TASKS[5]));

console.log("\nanswer extraction:");
const samples = [
  ["answer last", "Step 1: 40 - 15 = 25\nAnswer: 25"],
  ["self-corrected", "Answer: 30\nWait, 40 - 15 = 25.\nAnswer: 25"],
  ["no answer line", "The shop has 25 mugs left."],
];
for (const [name, text] of samples) console.log(`  ${name.padEnd(15)} -> ${extract(text)}`);

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.