Skip to the solution
HardAI Engineering16 min read

What are common RAG failure modes and how do you debug them?

562 views

The problem

Question presented to candidate: "Users say our RAG assistant sometimes gives wrong or outdated answers, and sometimes says it does not know things that are definitely in the docs. What usually goes wrong in a RAG system, and how would you actually debug one of these reports?"

What a strong answer should cover:

  • Failures happen at a specific stage and should be named by it: not indexed or badly parsed, stale or duplicated versions, a fact split across chunks, retrieval miss (vocabulary mismatch, exact identifiers, wrong filter), ranked out by the reranker or the context budget, context present but ignored or contradicted, wrong citations, and answering questions the documents cannot answer
  • The debugging method: reproduce from a per-request trace (rewritten query, filter, retrieved IDs and scores, reranked and packed IDs, prompt version, answer), identify the gold chunk, and walk the stages in order; the first stage that lost the gold chunk is where the fix goes
  • Check the cheap upstream things first: is the document in the index at all, is it the current version, does one chunk actually contain the answer
  • Turn single reports into an eval set and a funnel (indexed, retrieved, packed, answered correctly) so you fix the stage losing the most questions, and add each fixed case as a regression test
  • Typical fixes per stage: parsing and sync fixes, structure-aware chunking or contextual headers, hybrid search and query rewriting, deeper reranking, fewer and better-ordered chunks, grounding and abstention instructions, citation validation

Clarifying questions expected:

  • Do we log retrieved chunk IDs and scores per request, or only the question and answer?
  • Do we have examples of the failing questions with the documents users expected?

Code / implementation expected: Optional — a triage function that classifies a logged request by the first stage that lost the answer, or a clear step-by-step debugging procedure.

  • rag
  • debugging
  • failure-modes
  • retrieval

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

The solution is hidden so you can try first

16 min read · 9 sections

Read the code

Chunk-boundary check and a first-failing-stage triage over logged requests
Open in playground
// RAG failure triage: classify each logged request by the first stage that lost the answer.
const words = (t) => t.toLowerCase().replace(/[^a-z0-9 ]/g, " ").split(" ").filter((w) => w);

console.log("1) a fact split by a chunk boundary");
const DOC = "Enterprise customers can cancel at any time. The cancellation fee is waived if notice is given 60 days before renewal. Otherwise one month is charged.";
function chunk(text, size, overlap) {
  const w = text.split(" "), out = [];
  for (let i = 0; i < w.length; i += size - overlap) { out.push(w.slice(i, i + size).join(" ")); if (i + size >= w.length) break; }
  return out;
}
function bySentence(text, max) { // whole sentences, packed up to max words
  const out = []; let cur = [];
  for (const sent of text.split(/(?<=\.) /)) {
    const n = sent.split(" ").length;
    if (cur.length && cur.join(" ").split(" ").length + n > max) { out.push(cur.join(" ")); cur = []; }
    cur.push(sent);
  }
  if (cur.length) out.push(cur.join(" "));
  return out;
}
const configs = [["fixed 10 words, overlap 0", chunk(DOC, 10, 0)], ["fixed 10 words, overlap 4", chunk(DOC, 10, 4)], ["fixed 10 words, overlap 3", chunk(DOC, 10, 3)], ["sentences, max 16 words", bySentence(DOC, 16)]];
configs.forEach(([name, cs], k) => {
  const whole = cs.findIndex((c) => words(c).includes("fee") && words(c).includes("60"));
  console.log(`   ${name.padEnd(26)} ${cs.length} chunks; "fee" and "60" together: ${whole >= 0 ? "yes, chunk " + whole : "no"}`);
  if (k === 0) cs.forEach((c, i) => console.log(`     chunk ${i}: ${c}`));
});
console.log("");
console.log("2) triage of logged requests");
// Source of truth (document store) and what the vector index currently holds.
const SOURCE = { "policy#2": "v3", "pricing#1": "v2", "sso#4": "v1", "export#0": "v1", "limits#3": "v1", "billing#7": "v1", "audit#2": "v1" };
const INDEX = { "policy#2": "v3", "pricing#1": "v1", "export#0": "v1", "limits#3": "v1", "billing#7": "v1", "audit#2": "v1" };
const LOGS = [
  { q: "What is the refund window?", gold: "policy#2", expect: "45 days", cand: ["policy#2", "billing#7"], ctx: ["policy#2"], answer: "45 days [policy#2]" },
  { q: "How do I set up SSO?", gold: "sso#4", expect: "SAML", cand: ["audit#2", "limits#3"], ctx: ["audit#2"], answer: "I do not know." },
  { q: "Price of the team plan?", gold: "pricing#1", expect: "$12", cand: ["pricing#1"], ctx: ["pricing#1"], answer: "$10 per seat [pricing#1]" },
  { q: "Can I export to CSV?", gold: "export#0", expect: "CSV", cand: ["limits#3", "billing#7", "export#0"], ctx: ["limits#3", "billing#7"], answer: "Exports are limited to 10 per day [limits#3]" },
  { q: "What is error E1043?", gold: "limits#3", expect: "rate limit", cand: ["billing#7", "audit#2"], ctx: ["billing#7"], answer: "E1043 is a billing error [billing#7]" },
  { q: "Max upload size?", gold: "limits#3", expect: "2 GB", cand: ["limits#3", "export#0"], ctx: ["export#0", "billing#7", "audit#2", "limits#3"], answer: "Uploads up to 500 MB [export#0]" },
  { q: "Who can view audit logs?", gold: "audit#2", expect: "admins", cand: ["audit#2"], ctx: ["audit#2"], answer: "Only admins [audit#9]" },
  { q: "Do you support carrier pigeons?", gold: null, expect: null, cand: ["export#0"], ctx: ["export#0"], answer: "Yes, via the export API [export#0]" },
  { q: "When is the invoice sent?", gold: "billing#7", expect: "1st of the month", cand: ["billing#7"], ctx: ["billing#7"], answer: "On the 1st of the month [billing#7]" },
];
function triage(r) {
  const cites = [...r.answer.matchAll(/\[([a-z]+#\d+)\]/g)].map((m) => m[1]);
  const abstained = r.answer.startsWith("I do not know");
  if (r.gold === null) return abstained ? "ok" : "answered the unanswerable";
  if (!(r.gold in INDEX)) return "not indexed";
  if (INDEX[r.gold] !== SOURCE[r.gold]) return "stale index";
  if (!r.cand.includes(r.gold)) return "retrieval miss";
  if (!r.ctx.includes(r.gold)) return "ranked out";
  if (!r.answer.includes(r.expect)) return r.ctx.indexOf(r.gold) > 0 ? "ignored (not first)" : "ignored context";
  if (cites.some((c) => !r.ctx.includes(c))) return "bad citation";
  return "ok";
}
const counts = new Map();
LOGS.forEach((r, i) => {
  const t = triage(r);
  counts.set(t, (counts.get(t) || 0) + 1);
  const pos = r.ctx.indexOf(r.gold);
  console.log(`   ${String(i + 1).padStart(2)}. ${r.q.padEnd(33)} gold ${String(r.gold).padEnd(10)} ${pos >= 0 ? `ctx pos ${pos + 1}` : "ctx  -   "}  -> ${t}`);
});

console.log("");
console.log("3) funnel: where answerable questions are lost");
const ans = LOGS.filter((r) => r.gold !== null);
const stages = [
  ["in source of truth", (r) => r.gold in SOURCE],
  ["indexed and current", (r) => INDEX[r.gold] === SOURCE[r.gold]],
  ["in retrieved candidates", (r) => INDEX[r.gold] === SOURCE[r.gold] && r.cand.includes(r.gold)],
  ["in packed context", (r) => INDEX[r.gold] === SOURCE[r.gold] && r.ctx.includes(r.gold)],
  ["answered correctly", (r) => INDEX[r.gold] === SOURCE[r.gold] && r.ctx.includes(r.gold) && r.answer.includes(r.expect)],
];
for (const [name, ok] of stages) {
  const n = ans.filter(ok).length;
  console.log(`   ${name.padEnd(24)} ${n} of ${ans.length}  ${"#".repeat(n)}`);
}
console.log("");
console.log("4) counts by bucket");
[...counts].sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)).forEach(([t, n]) => console.log(`   ${String(n).padStart(2)}  ${t}`));

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

Question 73 of 83 in AI engineering

Keep your momentum going.

Mark this question solved to track your progress, then take the next one while it is fresh.

All AI engineering questions
Up nextHard

What is prompt injection and how do you defend against it?

Continue