What a context window is, and why your chatbot forgets
An AI model can only read so much at once. That limit explains lost instructions, rising costs and why long chats drift.
Kivorent News desk, , 2 min read
What to know
- The context window is the maximum amount of text a model can take into account in a single response.
- It is measured in tokens. In English, one token is roughly three quarters of a word.
- Everything counts against it: your messages, the model's replies, attached files and hidden instructions.
Ask a chatbot to follow a rule at the start of a long conversation, and a hundred messages later it may act as if you never said it. The cause is usually the context window.
The short version
A language model does not remember your conversation the way a person does. Each time it replies, it is handed the text of the conversation so far and writes what comes next. The context window is the cap on how much text it can be handed at once.
Tokens, the unit of measure
Models read text in pieces called tokens. A token can be a whole short word, part of a longer word, or a punctuation mark. A common rule of thumb for English is that one token is about three quarters of a word, so 1,000 tokens is around 750 words. Code, numbers and other languages can use more tokens for the same amount of text.
What fills the window
More than you might expect:
- System instructions the app adds behind the scenes.
- Every earlier message in the conversation, yours and the model's.
- Files and web pages you attach or the model looks up.
- Tool results, when the model runs a search or executes code.
- The reply itself, which has to fit in the same budget.
What happens when it fills up
Apps handle a full window in one of three ways. Some drop the oldest messages. Some replace older messages with a shorter summary. Some stop and ask you to start a new chat. The first two are why an instruction you gave early on can quietly stop being followed: the model is no longer being shown it.
Why bigger is not always better
Larger windows let a model work with whole codebases or long reports. They also have costs.
- Price. Most services charge by the token, and the whole conversation is sent again on every turn. A long chat gets more expensive with each message.
- Speed. More text to read means a slower first word.
- Attention. Models tend to use material at the start and end of a long input more reliably than material buried in the middle.
How to work with it
- Put standing instructions where the app keeps them permanently, such as a project or custom instruction setting, instead of in the first message.
- Start a new chat when the topic changes.
- Attach the three pages that matter instead of the whole 200-page document.
- If a model starts contradicting earlier decisions, restate them. It may no longer be able to see them.
A related technique, retrieval, exists to get the right few pages in front of the model. We cover it in RAG explained.
Spotted an error? Tell us and we will fix it.