EasyAI Engineering17 min read

How do you choose the right model for a task?

Q: How do you choose the right model for a task?

**Target Audience:** Engineers adding their first LLM feature who see a long list of models and want a repeatable way to pick one. **Difficulty:** Easy **Verified with:** Node 22.22.0 and Python 3.12.3 (the selection harness below; the models in it are deterministic stubs and the prices are illustrative). Product and paper facts are cited inline with the date they were checked (October 2026). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First Imagine you need a car to deliver pizzas in a city. You could buy a Formula 1 car: it wins every race on TV. It is also wildly expensive, burns fuel, needs a specialist crew, and has no room for a pizza box. A small scooter is cheap, fast through traffic and does the job per

110 views

14 of 83

The problem

Question presented to candidate: "We are adding an LLM feature to our product. There are dozens of models out there, from tiny cheap ones to frontier models. Walk me through how you would decide which model to use, and how you would know you picked right."

What a strong answer should cover:

Start from the task and its constraints, not from a leaderboard: required quality, latency budget (time to first token and total), cost at the expected volume, context length, and must-have features (tool calling, structured output, vision, data residency)
Hard constraints filter first (a model without tool calling or over the latency budget is out no matter how smart it is); the remaining candidates are compared on your own eval set, not on public benchmarks
The rule: pick the cheapest, fastest model that clears the eval bar, with cost worked out per request and per month from real token counts
Routing and cascades: send easy traffic to a small model and escalate hard or low-confidence cases to a bigger one; know that a cascade only works if the escalation signal is reliable
Treat the choice as reversible: keep a provider-neutral interface, pin model versions, re-run the evals when a new model ships or an old one is deprecated

Clarifying questions expected:

What does "good enough" mean here, and do we have labelled examples to measure it?
What volume and latency do we expect, and is the call user-facing or background?
Are there data constraints (PII, region, on-prem) that rule out some providers?

Code / implementation expected: Optional — a small selection harness (eval pass rate, cost per request, constraint filter) or a cost calculation is a strong signal; no model call is needed.

model-selection
routing
cost
capability

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers adding their first LLM feature who see a long list of models and want a repeatable way to pick one. Difficulty: Easy Verified with: Node 22.22.0 and Python 3.12.3 (the selection harness below; the models in it are deterministic stubs and the prices are illustrative). Product and paper facts are cited inline with the date they were checked (October 2026).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

Imagine you need a car to deliver pizzas in a city. You could buy a Formula 1 car: it wins every race on TV. It is also wildly expensive, burns fuel, needs a specialist crew,

The solution is hidden so you can try first

17 min read · 11 sections

Read the code

Pick the cheapest model that passes the evals, then test a small-to-frontier cascade

Open in playground

// Choose a model the way production teams do: run every candidate on the same
// golden set, drop the ones that miss a hard requirement, keep the cheapest
// that passes, then see whether a cascade (cheap first, escalate) beats it.
// The models are stubs: a model answers a task correctly when its capability
// level is at least the task difficulty. Prices are illustrative, not a vendor list.
const models = [
  // name, capability, $ per 1M input, $ per 1M output, p95 latency ms, tools?
  { name: "tiny", cap: 2, pin: 0.05, pout: 0.2, p95: 400, tools: false },
  { name: "small", cap: 3, pin: 0.25, pout: 1.0, p95: 700, tools: true },
  { name: "mid", cap: 4, pin: 1.0, pout: 4.0, p95: 1400, tools: true },
  { name: "frontier", cap: 5, pin: 5.0, pout: 25.0, p95: 2600, tools: true },
];
// Golden set: 20 tasks with difficulty 1 (easy) to 5 (hard).
const golden = [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5];
const req = { minAccuracy: 0.9, maxP95: 3000, needTools: true };
const work = { perMonth: 1000000, inTok: 1200, outTok: 300 };

const perRequest = (m) => (work.inTok * m.pin + work.outTok * m.pout) / 1e6;
const money = (x) => "$" + x.toFixed(2).replace(/\B(?=(\d{3})+(?!\d))/g, ",");
const pad = (s, n) => String(s).padEnd(n);

console.log("workload: " + work.perMonth + " requests/month, " + work.inTok + " in + " + work.outTok + " out tokens each");
console.log("requirements: accuracy >= 90%, p95 <= 3000 ms, tool use");
console.log(pad("model", 10) + pad("acc", 6) + pad("$/request", 11) + pad("$/month", 13) + pad("p95", 7) + "verdict");
const passing = [];
for (const m of models) {
  const correct = golden.filter((d) => d <= m.cap).length;
  const acc = correct / golden.length;
  const why = [];
  if (acc < req.minAccuracy) why.push("accuracy");
  if (m.p95 > req.maxP95) why.push("latency");
  if (req.needTools && !m.tools) why.push("no tools");
  const verdict = why.length ? "FAIL (" + why.join(", ") + ")" : "pass";
  if (!why.length) passing.push(m);
  console.log(pad(m.name, 10) + pad(Math.round(acc * 100) + "%", 6) + pad("$" + perRequest(m).toFixed(5), 11) +
    pad(money(perRequest(m) * work.perMonth), 13) + pad(m.p95, 7) + verdict);
}
passing.sort((a, b) => perRequest(a) - perRequest(b));
const pick = passing[0];
console.log("cheapest model that passes: " + pick.name + " at " + money(perRequest(pick) * work.perMonth) + "/month");

// Cascade: try "small" first; a cheap validator (schema check + self-check)
// flags wrong answers on hard tasks; flagged tasks are re-run on "frontier".
// The validator misses one wrong answer (the last d=4 task) on purpose:
// real validators are not perfect.
const small = models[1], big = models[3];
let correct = 0, escalated = 0, silentWrong = 0;
golden.forEach((d, i) => {
  if (d <= small.cap) { correct++; return; }
  const caught = i !== 17;
  if (caught) { escalated++; if (d <= big.cap) correct++; }
  else silentWrong++;
});
const cascadeCost = perRequest(small) + (escalated / golden.length) * perRequest(big);
console.log("cascade small -> frontier: accuracy " + Math.round((correct / golden.length) * 100) + "%, escalated " +
  escalated + "/" + golden.length + ", silent wrong " + silentWrong + ", " + money(cascadeCost * work.perMonth) + "/month");
const frontier = perRequest(big) * work.perMonth;
console.log("vs frontier for everything: " + money(frontier) + "/month -> cascade saves " +
  Math.round((1 - (cascadeCost * work.perMonth) / frontier) * 100) + "%");
console.log("vs " + pick.name + " alone: cascade costs " + ((cascadeCost / perRequest(pick)) * 100).toFixed(1) + "% of it");

Discussion

Other approaches, follow-ups and how it went in real interviews.

No comments yet. Share how you would answer it.

Question 14 of 83 in AI engineering

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All AI engineering questions

Up nextEasy→

When should you use batch APIs instead of real-time LLM calls?

Continue