No content blocks yet.
The Physics of Context: More Tokens, Less Intelligence
Attention is a conserved quantity. Every token you add taxes every token already there. A physical model for why bigger contexts make LLMs dumber.
Key Takeaways
- Context windows grow; attention doesn't.
- Attention is normalized - a fixed budget split across every token present - so each token you add taxes all the others.
- Quality versus context size is a peaked curve: rising while you add needed facts, falling as low-value tokens dilute the budget.
- Near-miss content is worse than noise: retrieval-shaped distractors attract real attention and interfere destructively, producing grounded-sounding wrong answers.
- Needle-in-a-haystack benchmarks test the zero-interference corner of the space and say little about production behavior.