How do you choose between open-weight models and proprietary API models?

Q: How do you choose between open-weight models and proprietary API models?

**Target Audience:** Engineers and tech leads who have shipped on a model API and are asked "should we run our own model?" **Difficulty:** Medium **Verified with:** Node 22.22.0 and Python 3.12.3 (the break-even model below; every input is a labelled assumption, not a quote). Licence and product facts are cited inline with the source and the date checked (October 2026). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First A restaurant needs bread. It can buy from a bakery every morning: pay per loaf, no ovens, no bakers, and the bakery bakes better bread than the restaurant could. Or it can install ovens and hire a baker: a large fixed bill every month whether it sells 100 loaves or 10,000, but each extra loaf is a

The problem

Question presented to candidate: "Our team is debating whether to keep paying for a proprietary model API or to run an open-weight model ourselves. How would you make that decision, and what would change your mind either way?"

What a strong answer should cover:

The definitions: open-weight means the trained weights are downloadable and runnable under a licence (not necessarily open source); proprietary means you only reach the model through the vendor API
Hard constraints decide first: data that may not leave your network or region, a need to fine-tune or pin exact weights, or a quality bar that only a frontier proprietary model meets on your evals
The cost model: API cost grows linearly with tokens; self-hosting is mostly fixed (GPUs provisioned for peak, redundancy, engineering time), so there is a break-even volume, and utilisation drives it
Hidden costs of self-hosting: serving stack (vLLM or similar), autoscaling, upgrades, evals for every new release, on-call; and the middle options (open-weight models on a managed host or cloud marketplace)
Licence terms matter: some open-weight licences restrict use (for example the Llama community licence threshold for very large companies), while others are permissive (Apache 2.0, MIT)

Clarifying questions expected:

What volume and traffic shape do we have, and how fast is it growing?
Are there data-residency, privacy or air-gap requirements?
Does the team already run GPU infrastructure, and who would be on call for it?

Code / implementation expected: Optional — a break-even calculation with stated assumptions is a strong signal; no model call is needed.

open-weights
api
build-vs-buy
architecture

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers and tech leads who have shipped on a model API and are asked "should we run our own model?" Difficulty: Medium Verified with: Node 22.22.0 and Python 3.12.3 (the break-even model below; every input is a labelled assumption, not a quote). Licence and product facts are cited inline with the source and the date checked (October 2026).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

A restaurant needs bread. It can buy from a bakery every morning: pay per loaf, no ovens, no bakers, and the bakery bakes better bread than the restaurant could. Or it can install ovens and hire a b

The solution is hidden so you can try first

15 min read · 12 sections

Read the code

Break-even between a per-token API and self-hosted GPUs, plus a constraint-first decision

Open in playground

// Open-weight (self-hosted) vs proprietary API: hard constraints first,
// then break-even arithmetic. Every number below is an ASSUMPTION you would
// replace with your own quotes and load-test results.
const A = {
  inTok: 1200, outTok: 300,          // tokens per request
  apiIn: 1.0, apiOut: 4.0,           // API $ per 1M tokens (illustrative)
  gpuHour: 2.5,                      // $ per GPU-hour (illustrative cloud price)
  rpsPerGpu: 4,                      // sustained requests/s one GPU serves (from a load test)
  peakToAvg: 3,                      // traffic peak divided by average
  minGpus: 2,                        // two replicas so one can fail or be upgraded
  opsPerMonth: 5000,                 // share of an engineer for serving, upgrades, on-call
};
const HOURS = 730, SECONDS = 30 * 86400;
const apiPerReq = (A.inTok * A.apiIn + A.outTok * A.apiOut) / 1e6;
const money = (x) => "$" + Math.round(x).toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");

function selfHost(reqPerMonth) {
  const peakRps = (reqPerMonth / SECONDS) * A.peakToAvg;
  const gpus = Math.max(A.minGpus, Math.ceil(peakRps / A.rpsPerGpu));
  return { gpus, cost: gpus * A.gpuHour * HOURS + A.opsPerMonth, util: reqPerMonth / SECONDS / (gpus * A.rpsPerGpu) };
}

console.log("API cost per request: $" + apiPerReq.toFixed(4));
console.log("volume/month   API         self-host   GPUs  avg GPU util  cheaper");
for (const v of [1e6, 3e6, 10e6, 30e6, 100e6]) {
  const s = selfHost(v), api = v * apiPerReq;
  console.log((v / 1e6 + "M").padEnd(15) + money(api).padEnd(12) + money(s.cost).padEnd(12) +
    String(s.gpus).padEnd(6) + (Math.round(s.util * 100) + "%").padEnd(14) + (api < s.cost ? "API" : "self-host"));
}
let v = 0;
while (v * apiPerReq <= selfHost(v).cost) v += 100000;
console.log("break-even: about " + (v / 1e6).toFixed(1) + "M requests/month (" + money(v * apiPerReq) + " API spend)");
const noOps = { ...A, opsPerMonth: 0 };
let w = 0;
const hostNoOps = (r) => Math.max(noOps.minGpus, Math.ceil(((r / SECONDS) * noOps.peakToAvg) / noOps.rpsPerGpu)) * noOps.gpuHour * HOURS;
while (w * apiPerReq <= hostNoOps(w)) w += 100000;
console.log("if you forget the engineer: break-even looks like " + (w / 1e6).toFixed(1) + "M requests/month");

// Hard constraints decide before cost does.
function decide(s) {
  if (s.weightsMustStayOnPrem) return "open-weight (data may not leave your network)";
  if (s.needsFrontierQuality) return "API (no open-weight model met the eval bar)";
  if (s.needsCustomWeights) return "open-weight (you must fine-tune or pin the exact weights)";
  return s.reqPerMonth * apiPerReq < selfHost(s.reqPerMonth).cost ? "API (cheaper at this volume)" : "open-weight (cheaper at this volume)";
}
const scenarios = [
  { name: "startup chatbot, 2M req", reqPerMonth: 2e6 },
  { name: "hospital notes, air-gapped", reqPerMonth: 2e6, weightsMustStayOnPrem: true },
  { name: "agentic coding assistant", reqPerMonth: 40e6, needsFrontierQuality: true },
  { name: "bulk classifier, 60M req", reqPerMonth: 60e6 },
];
for (const s of scenarios) console.log(s.name.padEnd(28) + "-> " + decide(s));

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.