What are tokens and why do they matter for cost and limits?

Q: What are tokens and why do they matter for cost and limits?

**Target Audience:** Beginners who are about to ship (or pay for) their first LLM feature and keep seeing "tokens" on every pricing and limits page. **Difficulty:** Easy **Verified with:** Node 22.22.0 and Python 3.12.3 (a tiny BPE tokenizer and the cost arithmetic; no real tokenizer or model is called). Product facts and prices are cited inline with the date they were checked (October 2026). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the exact words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First Imagine a taxi that charges by the kilometre, but the meter does not measure kilometres you can see. It measures "road segments", and the city decides where segments start and end: a straight motorway might be one long segment, while a twisty old-town street is chopped

The problem

Question presented to candidate: "Everything in LLM APIs is measured in tokens — the pricing page, the context limit, the rate limits. What is a token, how does text get split into them, and how do they affect what our feature costs and how fast it is?"

What a strong answer should cover:

A token is the unit a model reads and writes: a word, part of a word, a punctuation mark or a byte sequence, produced by a tokenizer with a fixed vocabulary (tens of thousands to a few hundred thousand entries)
How subword tokenizers such as BPE are built: start from characters or bytes, repeatedly merge the most frequent adjacent pair; common words become one token, rare words split into many pieces
Rules of thumb and their limits: roughly 4 characters or three quarters of an English word per token (OpenAI), but code, numbers, rare words and many non-English languages take more tokens; counts differ between model families
Cost: input and output tokens are priced separately and output is usually several times dearer; a worked monthly estimate from traffic, prompt size and answer size
Limits and latency: the context window and rate limits are counted in tokens; each output token is one sequential decode step, so answer length drives response time
Practical habits: count with the provider tokenizer or token-counting endpoint, cap max_tokens, shrink prompts, cache repeated prefixes

Code / implementation expected: Optional — a small tokenizer or a cost calculation from token counts is a good way to show the arithmetic.

tokens
tokenization
pricing
fundamentals

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Beginners who are about to ship (or pay for) their first LLM feature and keep seeing "tokens" on every pricing and limits page. Difficulty: Easy Verified with: Node 22.22.0 and Python 3.12.3 (a tiny BPE tokenizer and the cost arithmetic; no real tokenizer or model is called). Product facts and prices are cited inline with the date they were checked (October 2026).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the exact words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

Imagine a taxi that charges by the kilometre, but the meter does not measure kilometres you can see. It measures "road segments", and the city decides where segments sta

The solution is hidden so you can try first

13 min read · 11 sections

Read the code

Train a tiny BPE tokenizer, count tokens and work out a monthly bill

Open in playground

// Train a tiny byte-pair-encoding (BPE) tokenizer, tokenize text with it,
// then do the token arithmetic that drives cost, limits and latency.
const trainingWords = { low: 5, lower: 2, newest: 6, widest: 3, new: 4, wide: 2 };

// Every word starts as characters plus an end-of-word marker "_".
let vocabWords = Object.keys(trainingWords).map((w) => ({ syms: [...w, "_"], n: trainingWords[w] }));
const merges = [];
for (let step = 1; step <= 10; step++) {
  const pairs = {};
  for (const { syms, n } of vocabWords)
    for (let i = 0; i + 1 < syms.length; i++) {
      const key = syms[i] + " " + syms[i + 1];
      pairs[key] = (pairs[key] || 0) + n;
    }
  // Most frequent pair wins; ties go to the alphabetically smallest pair.
  const best = Object.keys(pairs).sort().reduce((a, b) => (pairs[b] > pairs[a] ? b : a));
  const [l, r] = best.split(" ");
  merges.push([l, r]);
  console.log("merge " + String(step).padStart(2) + ": " + l + " + " + r + " -> " + l + r + "  (seen " + pairs[best] + " times)");
  vocabWords = vocabWords.map(({ syms, n }) => {
    const out = [];
    for (let i = 0; i < syms.length; i++) {
      if (i + 1 < syms.length && syms[i] === l && syms[i + 1] === r) { out.push(l + r); i++; } else out.push(syms[i]);
    }
    return { syms: out, n };
  });
}

// Encoding new text = apply the learned merges, in the order they were learned.
function encodeWord(word) {
  let syms = [...word, "_"];
  for (const [l, r] of merges) {
    const out = [];
    for (let i = 0; i < syms.length; i++) {
      if (i + 1 < syms.length && syms[i] === l && syms[i + 1] === r) { out.push(l + r); i++; } else out.push(syms[i]);
    }
    syms = out;
  }
  return syms;
}
for (const text of ["newest lowest", "wider widest", "zebra quokka"]) {
  const toks = text.split(" ").flatMap(encodeWord);
  console.log(JSON.stringify(text).padEnd(16) + " -> " + toks.length + " tokens " + JSON.stringify(toks) + "  chars/token=" + (text.length / toks.length).toFixed(2));
}

// Cost arithmetic. Prices and traffic are ASSUMPTIONS for the exercise.
const priceIn = 3.0, priceOut = 15.0; // USD per 1M tokens
const reqPerDay = 10000, inTok = 1500, outTok = 400, days = 30;
const monthIn = reqPerDay * inTok * days, monthOut = reqPerDay * outTok * days;
const costIn = (monthIn / 1e6) * priceIn, costOut = (monthOut / 1e6) * priceOut;
console.log("month input tokens:  " + monthIn.toLocaleString("en-US") + " -> $" + costIn.toFixed(2));
console.log("month output tokens: " + monthOut.toLocaleString("en-US") + " -> $" + costOut.toFixed(2));
console.log("total $" + (costIn + costOut).toFixed(2) + " | output is " + ((100 * monthOut) / (monthIn + monthOut)).toFixed(1) + "% of tokens but " + ((100 * costOut) / (costIn + costOut)).toFixed(1) + "% of cost");
console.log("same prompts in a language that needs 2.5x the tokens: $" + (costIn * 2.5 + costOut * 2.5).toFixed(2));
const tokPerSec = 50; // assumed decode speed
console.log("decode time for " + outTok + " output tokens at " + tokPerSec + " tok/s: " + (outTok / tokPerSec).toFixed(1) + " s");

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.