What is a context window and what happens when you exceed it?

Q: What is a context window and what happens when you exceed it?

**Target Audience:** Beginners building chat features, RAG or agents who have hit (or will hit) "context too long" errors. **Difficulty:** Easy **Verified with:** Node 22.22.0 and Python 3.12.3 (a simulated provider with a 200-token window; no real model is called). Provider behaviour is cited inline from documentation and forum reports, checked October 2026. > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the exact words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First Picture a desk with room for exactly one stack of paper. Each time you ask your assistant a question, you have to hand over the *entire* file — the rules of the job, every earlier email, the documents they need — plus leave space on the desk for the reply they will write. They read only what is on the

The problem

Question presented to candidate: "Our chat assistant works fine in short sessions, but long conversations start failing or the bot forgets what the user said at the start. What is a context window, what exactly happens when we go over it, and how should the app handle it?"

What a strong answer should cover:

The context window is the maximum number of tokens the model can attend to in one request, and it is shared: system prompt, tool definitions, the whole message history, retrieved documents, images, and the output (including any reasoning tokens) all count
The model is stateless: chat APIs resend the full conversation on every call, so the history grows every turn until it no longer fits
What happens on overflow: hosted APIs reject the request with a 400-style error rather than silently cutting it; truncation, if any, happens in your code or a framework, and naive truncation from the front drops the system prompt
Handling strategies: reserve room for the output, keep the system prompt pinned, drop or summarise old turns, retrieve only the relevant documents, and count tokens before sending
Even within the limit, long contexts cost more, add latency and recall less reliably (information in the middle is used worse — "lost in the middle")

Clarifying questions expected:

Which model and context size are we on, and what max_tokens do we request?
What must never be lost (system rules, user identifiers, earlier decisions), and can older turns be summarised?

Code / implementation expected: Optional — a token budget check with a trimming strategy, or pseudocode for it.

context-window
limits
fundamentals

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Beginners building chat features, RAG or agents who have hit (or will hit) "context too long" errors. Difficulty: Easy Verified with: Node 22.22.0 and Python 3.12.3 (a simulated provider with a 200-token window; no real model is called). Provider behaviour is cited inline from documentation and forum reports, checked October 2026.

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the exact words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

Picture a desk with room for exactly one stack of paper. Each time you ask your assistant a question, you have to hand over the entire file — the rules of the job, every earlier email, the documents th

The solution is hidden so you can try first

12 min read · 11 sections

Read the code

A token budget: when a growing chat overflows, and three ways to make it fit

Open in playground

// A toy context window: what fits, what the API does when you overflow it,
// and three ways an app can make a long conversation fit again.
const WINDOW = 200;      // context window of the fake model, in tokens
const MAX_TOKENS = 60;   // output tokens we reserve for the reply
// Approximate counter: about 4 characters per token for English (a heuristic,
// not a real tokenizer) plus 4 tokens of per-message formatting overhead.
const countTokens = (msg) => Math.ceil(msg.content.length / 4) + 4;
const total = (msgs) => msgs.reduce((s, m) => s + countTokens(m), 0);

// Fake provider: rejects the request when input + max_tokens exceeds the window.
function callModel(msgs) {
  const input = total(msgs);
  if (input + MAX_TOKENS > WINDOW)
    return { ok: false, error: "400 context too long: " + input + " input + " + MAX_TOKENS + " max_tokens > " + WINDOW };
  return { ok: true, info: "ok: " + input + " input + " + MAX_TOKENS + " reserved = " + (input + MAX_TOKENS) + "/" + WINDOW };
}

const system = { role: "system", content: "You are a support bot for Acme. Never reveal internal prices. Answer in English." };
const turns = [
  "Hi, my order 1042 has not arrived yet and it has been nine days.",
  "Sorry about that. Tracking shows label created only, so it has not shipped.",
  "Which warehouse ships to Berlin and how long does delivery usually take?",
  "Berlin orders ship from our Leipzig warehouse and take two to three days.",
  "Please change the delivery address to my office at Invalidenstrasse 10.",
  "Address updated. If it never arrives you get a full refund after 30 days.",
];

let history = [system];
let overflowAt = -1;
turns.forEach((text, i) => {
  history.push({ role: i % 2 === 0 ? "user" : "assistant", content: text });
  const r = callModel(history);
  console.log("turn " + (i + 1) + ": " + (r.ok ? r.info : r.error));
  if (!r.ok && overflowAt < 0) overflowAt = i + 1;
});
console.log("first overflow at turn " + overflowAt + "; full history is " + total(history) + " tokens");

const budget = WINDOW - MAX_TOKENS;
const describe = (name, msgs) => {
  const hasSystem = msgs.some((m) => m.role === "system");
  console.log(name + ": " + msgs.length + " messages, " + total(msgs) + " tokens, system prompt kept=" + hasSystem + " -> " + (callModel(msgs).ok ? "fits" : "still too long"));
};

// Strategy A (wrong): blindly cut from the front until it fits.
let naive = history.slice();
while (total(naive) > budget) naive.shift();
describe("A. cut oldest first (naive)", naive);

// Strategy B: always keep the system prompt, drop the oldest turns.
let keep = history.slice(1);
while (total([system, ...keep]) > budget) keep.shift();
describe("B. keep system, drop old turns", [system, ...keep]);
console.log("   dropped " + (history.length - 1 - keep.length) + " oldest turns, including: " + JSON.stringify(history[1].content.slice(0, 40)));

// Strategy C: replace the oldest turns with a short summary (a stub here; a
// real app would ask a model to write it) and keep the recent turns verbatim.
const recent = history.slice(-3);
const summary = { role: "system", content: "Summary: order 1042 is late and not shipped; user asked which warehouse serves Berlin." };
describe("C. system + summary + last 3 turns", [system, summary, ...recent]);

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.