LLM Inference · Attention at Decode Time

The KV Cache:
A Hands-On, In-Browser Walkthrough

An interactive monograph

Why decoding feels like re-reading the whole conversation for every word — and the one trick that buys back a factor of N

Abstract

To generate the next token, an autoregressive transformer runs its whole attention stack over the entire sequence so far. Do that naively for every token and the first word of your prompt is re-processed for every word of the answer — a single decode step costs O(n²), and the whole answer costs O(n³). This walkthrough builds the intuition for the KV cache: why a token's keys and values are worth keeping but its query is not, what one cached decode step actually does, how large the cache grows, why the causal mask uses −∞ rather than ×0, why decoding is memory-bandwidth bound, and why prompt caching only reuses a contiguous prefix. It closes with a six-question assessment.

Every figure is live. Step the demos, drag the sliders, answer the checkpoints. Toggle Theme for dark mode. No code, no build step — pure in-browser JavaScript.

The KV Cache:A Hands-On, In-Browser Walkthrough

The KV Cache:
A Hands-On, In-Browser Walkthrough