How do multimodal models handle images, audio and other non-text inputs?

Q: How do multimodal models handle images, audio and other non-text inputs?

**Target Audience:** Engineers who have used text-only LLM APIs and now need to send screenshots, scans, photos or audio. **Difficulty:** Medium **Verified with:** Node 22.22.0 and Python 3.12.3 (the patch-projection demo and the token estimators below; no real model is called). Provider token rules and paper details are cited inline with the source and the date checked (October 2026). > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the **Verified** section quotes it in full. ## 1. Why This Even Matters — A Story First Imagine a brilliant analyst who only reads text, and who works in a sealed room. Someone wants their opinion on a photograph. You cannot pass the photo through the slot, only cards with writing. So you hire an assistant who looks at the photo square by square and writes a card for

The problem

Question presented to candidate: "Models like the current GPT, Claude and Gemini families accept screenshots, photos and audio, not just text. How does a model that was built around text tokens actually take in an image or a sound clip, and what does that mean for us when we build features on it?"

What a strong answer should cover:

Non-text inputs are turned into sequences of vectors in the same embedding space as text tokens: an image is cut into patches and run through a vision encoder (ViT-style), then a projector maps those features to the LLM width; the LLM then attends over image and text positions together
Audio follows the same pattern: resample, turn into a spectrogram (Whisper uses 80-channel log-Mel frames in 30-second windows), encode, project; or a pipeline of speech-to-text then a text model
Architectural options: adapter on a pretrained LLM (LLaVA-style projector), cross-attention into a frozen LM, or natively multimodal training; trade-offs in cost and quality
Practical consequences: images and audio cost tokens (each provider publishes a formula: tiles, patches or per-second rates), resolution settings trade detail for cost, large images are downscaled
Limitations and risks: small text and counting errors, hallucinated visual details, and prompt injection hidden inside images or audio

Code / implementation expected: Optional — turning a small image into patch vectors, or estimating image and audio token counts from published rules, shows real understanding.

multimodal
vision
audio
fundamentals

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers who have used text-only LLM APIs and now need to send screenshots, scans, photos or audio. Difficulty: Medium Verified with: Node 22.22.0 and Python 3.12.3 (the patch-projection demo and the token estimators below; no real model is called). Provider token rules and paper details are cited inline with the source and the date checked (October 2026).

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the words an interviewer expects. All program output shown was produced by actually running the example; the Verified section quotes it in full.

1. Why This Even Matters — A Story First

Imagine a brilliant analyst who only reads text, and who works in a sealed room. Someone wants their opinion on a photograph. You cannot pass the photo through the slot, only cards w

The solution is hidden so you can try first

16 min read · 11 sections

Read the code

From pixels to visual tokens: patchify and project an image, then estimate image and audio token counts

Open in playground

// How an image becomes "tokens" for a language model, and what that costs.
// Part 1 runs the real mechanism on a tiny image: cut it into patches,
// flatten each patch, project it with a weight matrix into the model width.
// Part 2 applies the providers' published token-count rules (estimators only).

// 1) An 8x8 grayscale image: left half dark, right half bright, a bright dot.
const H = 8, W = 8, P = 4, D = 4;                 // patch size P, model width D
const img = [];
for (let y = 0; y < H; y++) { img.push([]); for (let x = 0; x < W; x++) img[y].push(x < 4 ? 10 : 200); }
img[1][1] = 255;
// Fixed "learned" projection: D x (P*P) weights, deterministic for the demo.
const Wp = [];
for (let i = 0; i < D; i++) { Wp.push([]); for (let j = 0; j < P * P; j++) Wp[i].push((((i * 7 + j * 3) % 5) - 2) / 10); }
const tokens = [];
for (let py = 0; py < H / P; py++) for (let px = 0; px < W / P; px++) {
  const flat = [];
  for (let y = 0; y < P; y++) for (let x = 0; x < P; x++) flat.push(img[py * P + y][px * P + x] / 255);
  const vec = Wp.map((row) => row.reduce((a, w, j) => a + w * flat[j], 0) + 0.01 * tokens.length); // + position
  tokens.push(vec);
  console.log("patch (" + py + "," + px + ") mean=" + (flat.reduce((a, b) => a + b, 0) / flat.length).toFixed(3) +
    " -> visual token [" + vec.map((v) => v.toFixed(3)).join(", ") + "]");
}
const text = ["What", "is", "in", "the", "image", "?"];
console.log("sequence the LLM sees: " + tokens.length + " image tokens + " + text.length + " text tokens = " + (tokens.length + text.length) + " positions");
for (const [side, patch] of [[224, 16], [336, 14], [448, 14]]) {
  console.log("ViT-style encoder " + side + "x" + side + " with " + patch + "px patches -> " + (side / patch) ** 2 + " patch tokens");
}

// 2) Published estimators (rules as documented, checked October 2026).
function claude(w, h, maxEdge = 1568, maxTok = 1568) {
  let s = Math.min(1, maxEdge / Math.max(w, h));
  if ((w * s * h * s) / 750 > maxTok) s = Math.sqrt((maxTok * 750) / (w * h));
  const nw = Math.floor(w * s), nh = Math.floor(h * s);
  return Math.ceil((nw * nh) / 750);
}
function openaiTiles(w, h) {
  let s = Math.min(1, 2048 / Math.max(w, h));
  w = Math.floor(w * s); h = Math.floor(h * s);
  s = Math.min(1, 768 / Math.min(w, h));
  w = Math.floor(w * s); h = Math.floor(h * s);
  return 85 + 170 * Math.ceil(w / 512) * Math.ceil(h / 512);
}
console.log("image          claude-1568~  claude-2576~  openai-tiles-high  openai-low");
for (const [w, h] of [[200, 200], [512, 512], [1920, 1080], [4032, 3024]]) {
  console.log((w + "x" + h).padEnd(15) + String(claude(w, h)).padEnd(14) + String(claude(w, h, 2576, 4784)).padEnd(14) +
    String(openaiTiles(w, h)).padEnd(19) + "85");
}

// Audio: a Whisper-style encoder and a per-second API rate.
const secs = 45;
const frames = secs * 100;                        // 10 ms hop -> 100 frames per second
const windows = Math.ceil(secs / 30);             // fixed 30 s windows, zero-padded
console.log("audio " + secs + " s: " + frames + " mel frames, " + windows + " Whisper windows x 1500 encoder positions = " +
  windows * 1500 + "; Gemini at 32 tokens/s = " + secs * 32 + " tokens");
console.log("video 60 s at Gemini 263 tokens/s = " + 60 * 263 + " tokens");

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.