HardSystem Design17 min read

Fine-tune vs RAG vs prompt engineering: how do you choose in 2025-26?

Q: Fine-tune vs RAG vs prompt engineering: how do you choose in 2025-26?

**Target Audience:** Engineers and tech leads who must choose how to adapt an LLM for a real product, and want a decision process they can defend with numbers rather than opinions. **Difficulty:** Hard **Verified with:** Node 22.22.0 and Python 3.12.3 (a freshness simulation, a cost model with **assumed** prices, and a rule-based decision table; no model is called). Provider and paper facts are cited inline with the date they were checked (October 2026). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the exact words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First A new hire on your support team is smart but does not know your company. Three managers each have a fix. - The first writes **a better onboarding brief**: how to answer, what tone, three example replies. D

360 views

78 of 83

The problem

Question presented to candidate: "You are building an LLM feature for our product and the team is split three ways: better prompts, RAG, or fine-tuning. It is 2026, context windows are huge and prompt caching is cheap. How do you decide, and what would make you change your mind?"

What a strong answer should cover:

Diagnose the failure first: missing or changing knowledge points to retrieval (or, for a small static corpus, simply putting it in a long cached prompt); wrong behaviour (format, tone, a narrow task) that a good prompt cannot fix points to fine-tuning; cost or latency at high volume can justify fine-tuning a smaller model
Order of moves: prompt engineering with an eval first, then RAG for knowledge, then LoRA fine-tuning for behaviour; production systems often combine them (RAG for facts, a tuned model for format)
Freshness and governance: RAG updates when a document changes and can cite sources and enforce per-user access; fine-tuned knowledge is a snapshot that goes stale until retrained and cannot be deleted per document
Worked cost arithmetic: tokens per request × volume × price, plus fixed costs (vector store, training and eval runs), and the break-even volume where a tuned small model beats a long prompt; prompt caching moves that break-even a lot
What changed in 2025-26: long context and caching make "just put it in the prompt" viable for small corpora; fine-tuning availability depends on the provider (OpenAI's docs say its self-serve fine-tuning platform is winding down) while open-weight models plus LoRA remain fully under your control
Every option is judged by the same eval on your own data, including the cost of maintaining it when the base model changes

Clarifying questions expected:

What exactly fails today: wrong facts, wrong format or tone, or too slow and expensive?
How big is the knowledge, how often does it change, and does it need citations or per-user access control?
What volume do we expect, and do we have labelled examples and an eval set?

Code / implementation expected: Optional — a cost model with a break-even calculation, or a simulation of what happens to each approach when a fact changes, makes the reasoning concrete.

finetune
rag

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers and tech leads who must choose how to adapt an LLM for a real product, and want a decision process they can defend with numbers rather than opinions. Difficulty: Hard Verified with: Node 22.22.0 and Python 3.12.3 (a freshness simulation, a cost model with assumed prices, and a rule-based decision table; no model is called). Provider and paper facts are cited inline with the date they were checked (October 2026).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the exact words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

A new hire on your support team is smart but does not know your company. Three managers each have a fix.

The solution is hidden so you can try first

17 min read · 14 sections

Read the code

Prompt vs RAG vs fine-tune: freshness when a fact changes, monthly cost and break-even, and a decision table

Open in playground

// Prompt vs RAG vs fine-tune, made concrete: (1) what happens when a fact changes,
// (2) monthly cost at different volumes, (3) a requirements -> lever decision table.
// All prices below are ASSUMPTIONS for the arithmetic, not any vendor price list.

// (1) Freshness. The refund window changes from 30 to 14 days on day 10.
const truth = (day) => (day < 10 ? 30 : 14);
const systems = {
  "prompt (edited by hand on day 17)": (day) => (day < 17 ? 30 : 14),
  "RAG (re-indexed nightly)": (day) => truth(day - 1), // sees yesterday's documents
  "fine-tune (retrained day 0, 30)": (day) => (day < 30 ? 30 : truth(30)),
};
console.log("days 0-34 answering 'what is the refund window?':");
for (const [name, answer] of Object.entries(systems)) {
  let stale = 0;
  for (let day = 0; day < 35; day++) if (answer(day) !== truth(day)) stale++;
  console.log(`  ${name.padEnd(36)} wrong on ${String(stale).padStart(2)} of 35 days`);
}

// (2) Monthly cost. Per request: input tokens, output tokens, price per 1M tokens; plus fixed costs.
const BIG = { in: 3.0, out: 15.0 }, SMALL = { in: 0.4, out: 1.6 }; // assumed $ per 1M tokens
const options = [
  { name: "prompt only, big model", price: BIG, inTok: 6000, outTok: 300, fixed: 0 },        // long instructions + 20 examples
  { name: "prompt only + cache", price: BIG, inTok: 300, cachedTok: 5700, outTok: 300, fixed: 0 }, // cached reads at 10% (assumed)
  { name: "RAG, big model", price: BIG, inTok: 1500 + 2000, outTok: 300, fixed: 300 },      // + 2k retrieved tokens, vector DB
  { name: "fine-tuned small model", price: SMALL, inTok: 300, outTok: 300, fixed: 2000 },  // retrain + eval each month
  { name: "fine-tuned small + RAG", price: SMALL, inTok: 2300, outTok: 300, fixed: 2300 },
];
const roundHalfAway = (v) => Math.sign(v) * Math.floor(Math.abs(v) + 0.5);
const usd = (v) => "$" + roundHalfAway(v).toLocaleString("en-US");
const perRequest = (o) => (o.inTok * o.price.in + (o.cachedTok || 0) * o.price.in * 0.1 + o.outTok * o.price.out) / 1e6;
const monthly = (o, n) => o.fixed + n * perRequest(o);
const volumes = [10000, 100000, 1000000, 3000000];
console.log("monthly cost by requests/month:", volumes.map((v) => v.toLocaleString("en-US")).join(" / "));
for (const o of options) {
  const cells = volumes.map((n) => usd(monthly(o, n)).padStart(8)).join(" ");
  console.log(`  ${o.name.padEnd(24)} per request $${perRequest(o).toFixed(5)} | ${cells}`);
}
const [p, pc, , f] = options;
for (const base of [p, pc]) {
  const breakEven = (f.fixed - base.fixed) / (perRequest(base) - perRequest(f));
  console.log(`fine-tuned small model beats "${base.name}" above`, Math.ceil(breakEven).toLocaleString("en-US"), "requests/month (if quality is equal)");
}

// (3) Decision rules: each requirement points at a lever; most real systems combine them.
function choose(req) {
  const levers = [];
  if (req.privateOrChangingFacts) levers.push("RAG");
  if (req.needsCitations && !levers.includes("RAG")) levers.push("RAG");
  if (req.strictFormatOrStyle && req.promptFailedEval) levers.push("fine-tune (LoRA)");
  if (req.highVolume && req.narrowTask && req.labelledExamples >= 1000) {
    if (!levers.includes("fine-tune (LoRA)")) levers.push("fine-tune (LoRA)");
    levers.push("smaller model");
  }
  if (!levers.length) levers.push("prompt engineering only");
  else levers.unshift("prompt engineering");
  return levers.join(" + ");
}
const scenarios = [
  ["support bot over our help center", { privateOrChangingFacts: true, needsCitations: true }],
  ["write release notes in house style", { strictFormatOrStyle: true, promptFailedEval: false }],
  ["same, but prompt fails the style eval", { strictFormatOrStyle: true, promptFailedEval: true, labelledExamples: 800 }],
  ["classify 5M tickets/month into 40 labels", { highVolume: true, narrowTask: true, labelledExamples: 20000 }],
  ["contract Q&A with clause citations", { privateOrChangingFacts: true, needsCitations: true, strictFormatOrStyle: true, promptFailedEval: true }],
  ["one-off internal summarizer", {}],
];
console.log("decision table:");
for (const [name, req] of scenarios) console.log(`  ${name.padEnd(42)} -> ${choose(req)}`);

Discussion

Other approaches, follow-ups and how it went in real interviews.

No comments yet. Share how you would answer it.

Question 78 of 83 in AI engineering

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All AI engineering questions

Up nextHard→

How do you monitor LLM quality drift in production?

Continue