What Is a Context Window?

Quick definition first. The context window of an LLM is how much text, counted in tokens, the model can look at at one time. Bigger window, longer inputs, more information going into each answer.

In simple terms, it's the model's working memory. Don't think "brain." Think whiteboard. Whatever's on the board, the model sees. Whatever isn't, doesn't exist for it. Session ends, board gets wiped.

What's it actually for?

Everything the model does happens inside it. It reads your question there, reasons there, writes the answer there. It's the playing field.

And here's what most people miss: the model doesn't remember your last chat by itself. Every time you hit send, the app packs the whole conversation back into the window and sends it again. That's why long chats get slower and heavier.

Tokens, not words

The window is counted in tokens. A token is a chunk of text: sometimes a whole word, sometimes just a piece of one. For English, it's about 1.3 tokens per word, or roughly 0.75 words per token. So a 128k window fits around 96,000 words.

Why do providers start small?

Cost. Attention compares every token with every other token, so compute grows roughly with the square of the length. Double the window, and that part costs about four times more.

So nobody jumps straight to a huge number. Providers went 4,096, then 8,192, then 16,384, and kept climbing to 128k, 200k and beyond as hardware and techniques got better. Also, many models are trained on shorter sequences first and stretched to longer ones later. They don't start at full length.

Input and output share the same space

The window holds what you send (input) and what the model writes (output). Together, they have to fit. There's also a separate cap on output, which is much smaller than the input limit.

Example: 128k window, you ask for a 30k-token answer. You've got about 98k left for input. Simple math, but people forget it.

So is the limit fixed?

Yes, it's a hard wall. Go over and one of two things happens: you get an error, or the app quietly chops off the oldest content. The model doesn't slowly get dumber after crossing the line. It just can't see what didn't fit.

The real problem is inside the window. The fuller it gets, the worse things get. Models miss details buried in the middle of long inputs (people call it "lost in the middle"), responses slow down, and the bill goes up. Same task at 20k tokens often beats it at 120k. A big window is a limit, not a target.

What to do about it

• Send only what's relevant. Not everything you have.
• New topic, new chat.
• Long conversation? Summarize it and carry the summary forward.
• Big document set? Use retrieval (RAG) so only the useful pieces get in.
• Put key instructions at the start or the end. Not the middle.

Bottom line

The context window is working memory. Hard limit, shared between input and output, expensive to scale, and less reliable the fuller it gets. Bigger helps. Using it smartly helps more.