Late nights fueled by code,
A technical ballet unfolds,
Heart and work in one ode.
Do you think this haiku was written by a human or a language model?
That's right.
It could be either! But here's the real question: when an LLM generates this response, why do cached tokens cost 10x less?
As of 2024, both OpenAI and Anthropic charge 10x less for cached input tokens. Anthropic claims caching can reduce latency by up to 85%.
But what exactly gets cached? Let's find out.
LLMs predict the next word in a sequence. Try it yourself:
How many times does the LLM need to run to generate "lamb"?
Exactly!
LLMs are autoregressive — they generate one token, append it to the input, and run again.
Not quite.
LLMs generate one token at a time, then use that output as input for the next token.
Inside the LLM, the attention mechanism determines how much each token should influence the next prediction.
To predict the next word after "little", how many attention scores must we compute?
That's right!
Standard attention computes an L × L matrix. This is computed every single time we generate a token!
Not quite.
Standard attention computes scores for every pair of tokens — that's L × L scores!
Attention uses three matrices derived from your input:
When we add a NEW token to the sequence, which matrices change?
Exactly!
The old rows don't change! K and V for "Mary" stay the same. This is the key insight.
Think about it:
K = embeddings @ W_K. If old embeddings don't change, their K values don't either!
Since K and V for old tokens never change, we can cache them!
💡 Key Insight
With KV caching, we only compute K and V for the new token, then append to the cache.
Let's compare compute costs. Adjust sequence length:
| Without Cache | With KV Cache | |
|---|---|---|
| FLOPs per token | 8.2B | 8.2M |
| Growth Rate | O(L²) | O(L) |
Why do providers charge ~10x less (not 1000x less)?
Exactly!
KV cache requires memory (10+ GB for long contexts). 10x is the practical savings.
Not quite.
Memory costs and other operations affect pricing.
Providers also cache across requests!
💡 This is why prompt caching matters!
100 users sharing the same prompt = pay full price once, then 10x cheaper for 99.
Let's check your understanding with 3 key questions.
What exactly gets cached in "KV caching"?
✓ Correct!
KV caching stores K = embeddings @ W_K and V = embeddings @ W_V.
✗ Not quite.
The cached data is the K and V matrices from attention.
🎯 Test Complete!
What exactly gets cached?
The K and V matrices from the attention mechanism.
Why does it work?
K and V for a token only depend on that token's position, not future tokens.
Where do the savings come from?
💡 Bonus: Temperature doesn't invalidate the cache!
Temperature only affects the final token selection. The KV cache is computed before that step.
🎓 You've completed the KV Caching tutorial!