You're deep into a conversation with an AI assistant, explaining your project architecture, sharing code snippets, debugging an issue — and then it responds like it has no idea what you've been talking about. It forgot. Or more accurately, it ran out of room to remember.
This happens because every AI model has a context window — a hard limit on how much text it can see at once. Understanding how context windows work isn't just academic curiosity. It directly affects how you design AI-powered applications, how you structure prompts, and why some conversations go sideways after a few exchanges.
What Is a Context Window?
A context window is the total amount of text an AI model can process in a single interaction. Think of it as the model's working memory — everything it can "see" at once when generating a response. This includes:
- System prompt: The instructions defining the model's behavior
- Conversation history: All previous messages in the chat
- Current input: Your latest message or prompt
- Generated output: The model's response (yes, this counts too)
The critical thing to understand: context windows are measured in tokens, not words or characters. And once you exceed the limit, the model doesn't gracefully summarize — it simply cannot see the content that falls outside the window.
Context Window Visualization:
|←——————— 128K tokens (GPT-4o) ———————→|
[System Prompt] [History...] [Your Input] [Response]
~500 tokens ~100K ~5K tokens ~22K tokens
When history grows too large:
[System Prompt] [... older messages DROPPED ...] [Recent History] [Input] [Response]
The model literally cannot see dropped messages.How Tokens Work: Tokenization and BPE
Tokens are the fundamental unit AI models use to process text. A token isn't a word — it's a chunk of text determined by the model's tokenizer. Most modern models use Byte Pair Encoding (BPE) to decide how to split text into tokens.
What Is a Token?
In English, a token averages about 4 characters or 0.75 words. But the exact tokenization depends on the word:
Tokenization examples (GPT-4 tokenizer): "Hello world" → ["Hello", " world"] = 2 tokens "JavaScript" → ["JavaScript"] = 1 token "tokenization" → ["token", "ization"] = 2 tokens "pneumonoultramicro" → ["pne", "um", "ono", "ultra", "micro"] = 5 tokens "const x = 42;" → ["const", " x", " =", " 42", ";"] = 5 tokens "🎉" → ["🎉"] = 1 token " " (4 spaces) → [" "] = 1 token Key insight: Common words = fewer tokens. Rare/technical words = more tokens. Code uses more tokens than prose because of symbols and formatting.
How BPE (Byte Pair Encoding) Works
BPE is the algorithm that builds a tokenizer's vocabulary. It starts with individual characters and iteratively merges the most frequent pairs:
BPE Training Process (simplified):
Step 1: Start with characters
"the cat sat" → ['t','h','e',' ','c','a','t',' ','s','a','t']
Step 2: Find most frequent pair → 't','h' appears often
Merge: 'th' becomes a token
Step 3: Next most frequent → 'th','e'
Merge: 'the' becomes a token
Step 4: Continue until vocabulary size reached (50K-100K tokens)
Result: Common words like "the", "function", "return" = 1 token
Rare words get split into sub-word piecesThis is why different languages tokenize differently. English text averages ~1.3 tokens per word, while languages like Japanese or Chinese can use 2-3x more tokens for the same semantic content. Code tokenization varies by language too — Python tends to be more token-efficient than verbose languages like Java.
Context Window Sizes: Model Comparison
Context windows have grown dramatically. Here's how the major models compare in 2026:
| Model | Context Window | ≈ Pages of Text | ≈ Lines of Code | Use Case |
|---|---|---|---|---|
| GPT-4o | 128K tokens | ~300 pages | ~8,000 lines | General development, chat |
| GPT-4o-mini | 128K tokens | ~300 pages | ~8,000 lines | Cost-effective tasks |
| Claude 4 Sonnet | 200K tokens | ~500 pages | ~12,000 lines | Large codebases, docs |
| Claude 4 Opus | 200K tokens | ~500 pages | ~12,000 lines | Complex reasoning |
| Gemini 2.0 Pro | 2M tokens | ~5,000 pages | ~125,000 lines | Entire repos, video |
| Gemini 2.0 Flash | 1M tokens | ~2,500 pages | ~62,000 lines | Fast, large context |
| Llama 4 Scout | 10M tokens | ~25,000 pages | ~600,000 lines | Open-source, long context |
| Llama 4 Maverick | 1M tokens | ~2,500 pages | ~62,000 lines | Open-source, balanced |
Important caveat: just because a model supports 200K tokens doesn't mean it performs equally well across that entire range. Most models have a "sweet spot" where accuracy is highest — typically the first 50-80% of the window. Content placed in the middle of very long contexts often gets less attention (the "lost in the middle" phenomenon).
Why Context Windows Matter for Developers
Context window size directly impacts what you can build. Here are the real-world scenarios where limits bite:
Long Document Processing
Processing legal contracts, research papers, or technical documentation often requires the full document in context for accurate summarization or Q&A. A 50-page contract is roughly 75K tokens — it fits in Claude or GPT-4o but consumes most of the window, leaving little room for instructions or conversation.
Codebase Understanding
When an AI coding agent needs to understand your project, it reads files into context. A medium-sized project might have 200 files. Even at 100 lines average, that's 20,000 lines — roughly 250K tokens. No single model can hold that all at once (except Gemini's 2M window), so agents must be selective about what they read.
Multi-Turn Conversations
Every message in a conversation history consumes tokens. A productive debugging session with code snippets can easily reach 50K+ tokens after 10-15 exchanges. Once you hit the limit, the model loses the beginning of the conversation — including the original problem description.
RAG (Retrieval-Augmented Generation)
RAG systems retrieve relevant documents and insert them into the context window. The window size limits how many documents you can include. With a 128K window, you might fit 20-30 relevant paragraphs plus instructions. With 2M tokens, you can include hundreds of documents — changing what's architecturally possible.
Strategies for Managing Context Limits
When your content exceeds the context window, you need strategies to work within the constraints:
1. Chunking
Break large documents into smaller pieces and process each independently. This works for tasks like summarization where you can summarize chunks, then summarize the summaries.
// Chunk a document for processing
function chunkText(text, maxTokens = 4000, overlap = 200) {
const words = text.split(/\s+/);
const chunks = [];
const wordsPerChunk = Math.floor(maxTokens * 0.75); // ~0.75 words per token
for (let i = 0; i < words.length; i += wordsPerChunk - overlap) {
const chunk = words.slice(i, i + wordsPerChunk).join(' ');
chunks.push(chunk);
}
return chunks;
}
// Process each chunk, then combine results
const chunks = chunkText(longDocument);
const summaries = await Promise.all(
chunks.map(chunk => summarize(chunk))
);
const finalSummary = await summarize(summaries.join('\n'));2. Summarization / Compression
Instead of dropping old context entirely, summarize it. Many AI applications maintain a "running summary" of the conversation that compresses earlier exchanges into a shorter form, preserving key facts while saving tokens.
// Conversation compression strategy
async function compressHistory(messages, maxTokens) {
const tokenCount = estimateTokens(messages);
if (tokenCount < maxTokens) return messages;
// Keep system prompt + last 5 messages intact
const systemPrompt = messages[0];
const recentMessages = messages.slice(-5);
const oldMessages = messages.slice(1, -5);
// Summarize older messages
const summary = await llm.complete({
messages: [
{ role: "system", content: "Summarize this conversation, preserving key decisions, code snippets, and unresolved questions." },
{ role: "user", content: JSON.stringify(oldMessages) }
]
});
return [
systemPrompt,
{ role: "system", content: `Previous conversation summary: ${summary}` },
...recentMessages
];
}3. Sliding Window
The simplest approach: keep the N most recent messages and drop everything else. This works for conversational UIs where recent context matters most. The downside is losing important early context like the original task description.
// Sliding window: always keep system + last N messages
function slidingWindow(messages, maxMessages = 20) {
const system = messages.filter(m => m.role === 'system');
const conversation = messages.filter(m => m.role !== 'system');
// Always keep system messages + most recent exchanges
return [
...system,
...conversation.slice(-maxMessages)
];
}4. RAG (Retrieval-Augmented Generation)
Instead of stuffing everything into context, store documents in a vector database and retrieve only the relevant pieces for each query. This is the most scalable approach for large knowledge bases.
// RAG: retrieve relevant context on-demand
async function ragQuery(question, vectorDB) {
// 1. Embed the question
const embedding = await openai.embeddings.create({
model: "text-embedding-3-small",
input: question
});
// 2. Find relevant documents (only ~5-10 chunks)
const results = await vectorDB.query({
vector: embedding.data[0].embedding,
topK: 5
});
// 3. Build focused context (fits easily in window)
const context = results.matches
.map(m => m.metadata.text)
.join('\n\n');
// 4. Generate answer with relevant context only
return await llm.complete({
messages: [
{ role: "system", content: "Answer using only the provided context." },
{ role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` }
]
});
}Code Example: Token Counting
Knowing your token count before making API calls helps you stay within limits and estimate costs.
JavaScript (using tiktoken)
import { encoding_for_model } from 'tiktoken';
// Count tokens for a specific model
function countTokens(text, model = 'gpt-4o') {
const encoder = encoding_for_model(model);
const tokens = encoder.encode(text);
encoder.free(); // Important: free memory
return tokens.length;
}
// Estimate cost before making API call
function estimateCost(inputText, estimatedOutput = 500) {
const inputTokens = countTokens(inputText);
const totalTokens = inputTokens + estimatedOutput;
// GPT-4o pricing (per 1M tokens)
const inputCost = (inputTokens / 1_000_000) * 2.50;
const outputCost = (estimatedOutput / 1_000_000) * 10.00;
return {
inputTokens,
estimatedOutputTokens: estimatedOutput,
totalTokens,
estimatedCost: `$${(inputCost + outputCost).toFixed(4)}`,
fitsInWindow: totalTokens < 128_000
};
}
console.log(estimateCost("Explain quantum computing in detail"));
// { inputTokens: 6, estimatedOutputTokens: 500, totalTokens: 506, estimatedCost: "$0.0050", fitsInWindow: true }Python (using tiktoken)
import tiktoken
def count_tokens(text: str, model: str = "gpt-4o") -> int:
"""Count tokens for a given model."""
encoder = tiktoken.encoding_for_model(model)
return len(encoder.encode(text))
def fits_in_context(messages: list, max_tokens: int = 128_000) -> dict:
"""Check if messages fit in the context window."""
encoder = tiktoken.encoding_for_model("gpt-4o")
total = 0
breakdown = []
for msg in messages:
tokens = len(encoder.encode(msg["content"]))
total += tokens + 4 # 4 tokens overhead per message
breakdown.append({"role": msg["role"], "tokens": tokens})
return {
"total_tokens": total,
"fits": total < max_tokens,
"remaining": max_tokens - total,
"usage_percent": f"{(total / max_tokens) * 100:.1f}%",
"breakdown": breakdown
}
# Example usage
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": open("large_file.py").read()},
]
result = fits_in_context(messages)
print(f"Using {result['usage_percent']} of context window")The Future of Context Windows
Context windows are growing rapidly. In 2020, GPT-3 launched with 4K tokens. By 2024, we had 128K-200K windows. In 2026, Gemini supports 2M tokens and Llama 4 Scout handles 10M. Where does this lead?
Will Infinite Context Eliminate RAG?
Probably not. Even with million-token windows, there are practical reasons to use retrieval:
- Cost: Sending 1M tokens per request is expensive. Retrieving 5 relevant chunks is 100x cheaper.
- Latency: Processing more tokens takes more time. Focused context produces faster responses.
- Accuracy: The "lost in the middle" problem worsens with scale. Focused context outperforms diluted context.
- Data freshness: Knowledge bases change hourly. You can't re-upload everything with each query.
The future is likely hybrid: large context windows for complex reasoning tasks that need extensive context, combined with RAG for efficient access to large-scale knowledge bases. The right tool depends on the task.
Context Caching
A newer development: APIs now support context caching. You upload a large document once and reference it across multiple queries without retransmitting or re-processing. This reduces cost and latency for repeated use of the same context. Both Anthropic (prompt caching) and Google (context caching) support this pattern.
// Anthropic prompt caching example
const response = await anthropic.messages.create({
model: "claude-sonnet-4-20250514",
system: [
{
type: "text",
text: largeDocs, // 100K+ tokens of reference docs
cache_control: { type: "ephemeral" } // Cache this block
},
{ type: "text", text: "Answer questions about the docs above." }
],
messages: [{ role: "user", content: "What's the auth flow?" }]
});
// First call: full price. Subsequent calls: 90% cheaper for cached portion.Common Mistakes
❌ Dumping entire files into context without filtering
Including a 2,000-line file when the question relates to one function. This wastes tokens and can confuse the model with irrelevant code.
✓ Fix: Extract only the relevant function plus necessary imports and type definitions.
❌ Not accounting for output tokens in your budget
Your 128K window must fit BOTH input and output. If you use 125K for input, the model only has 3K tokens to respond — cutting off mid-sentence.
✓ Fix: Reserve at least 20-30% of the window for the response, more for complex generation tasks.
❌ Ignoring the 'lost in the middle' effect
Models pay most attention to the beginning and end of the context. Critical information buried in the middle of a 100K-token context may be overlooked.
✓ Fix: Place important instructions and key information at the start or end of the context. Use headers to help the model navigate.
❌ Using a massive context window when a smaller one would work
Sending 100K tokens when 5K would suffice. Larger contexts are slower (more tokens to process) and more expensive. They can also reduce accuracy by diluting signal with noise.
✓ Fix: Start with minimal context and add more only if the output quality isn't sufficient.
❌ Not monitoring token usage in production
Without tracking, you won't know when conversations are hitting limits or when costs are spiking due to unnecessarily large contexts.
✓ Fix: Log token counts per request. Set alerts for conversations approaching window limits. Track cost per user session.
Best Practices
Count tokens before making API calls — use tiktoken (JS/Python) to avoid surprises
Reserve 20-30% of the context window for the model's response
Place critical instructions at the beginning and end of the context, not the middle
Use RAG for large knowledge bases instead of stuffing everything into context
Implement conversation summarization for long-running chat sessions
Monitor and log token usage per request in production to catch cost anomalies
Frequently Asked Questions
What is a context window in AI?
A context window is the maximum amount of text (measured in tokens) that an AI model can process in a single interaction. It includes both the input prompt and the generated output. Once the limit is reached, the model cannot reference earlier content.
Why does my AI forget things mid-conversation?
AI models have a fixed context window. When a conversation exceeds this limit, earlier messages are dropped or truncated. The model literally cannot see text outside its window, so it appears to forget previous context.
What is a token in AI?
A token is the basic unit of text that AI models process. Tokens are typically 3-4 characters or about 0.75 words in English. Common words are single tokens, while uncommon words may be split into multiple sub-word tokens.
How many tokens is 1,000 words?
In English, 1,000 words is approximately 1,300-1,500 tokens. The exact count depends on word complexity and the specific tokenizer. Code tends to use more tokens per character than natural language due to symbols and formatting.
What is BPE tokenization?
Byte Pair Encoding (BPE) is the algorithm most LLMs use to split text into tokens. It iteratively merges the most frequent pairs of characters or sub-words, resulting in common words being single tokens while rare words are split into pieces.
Which AI model has the largest context window?
As of 2026, Google's Gemini models support up to 2 million tokens, and Llama 4 Scout supports up to 10 million tokens. Context windows have grown rapidly — GPT-3 started with just 4K tokens in 2020.
Does a larger context window mean better performance?
Not necessarily. Models can experience 'lost in the middle' effects where they pay less attention to content in the center of long contexts. Quality of context matters more than quantity. Strategic placement and relevance are key.
What is RAG and how does it relate to context windows?
RAG (Retrieval-Augmented Generation) retrieves only relevant documents and places them in the context window before generation. It's the most scalable strategy for working with large knowledge bases without exceeding token limits.
Related Articles
Prompt Engineering for Developers
Practical techniques that work in production applications
RAG Explained for Developers
Ground your AI in real data with retrieval-augmented generation
How AI Coding Agents Work
Architecture, workflow & future of AI programming tools
AI Agents vs AI Workflows
When to use autonomous agents vs structured pipelines
Key Takeaways
Context windows include input + output — always budget for both
Tokens ≠ words. English averages ~1.3 tokens per word; code and other languages use more
Larger context doesn't mean better results — the 'lost in the middle' effect is real
RAG is the scalable solution for large knowledge bases that exceed any context window
Always measure token usage in production to control costs and catch edge cases
Conclusion
Context windows are the fundamental constraint shaping every AI application. Understanding how tokens work, how much fits in a window, and what happens when you exceed it gives you a practical advantage when building with LLMs.
The key takeaway: bigger isn't always better. A focused 10K-token context with relevant information will outperform a bloated 100K-token context full of noise. Use RAG to scale, use summarization to compress, and always measure your token usage. The best AI applications are the ones that manage context intelligently — not the ones that rely on ever-larger windows.
Context windows will keep growing. But the strategies here — chunking, summarization, RAG, and selective retrieval — will remain relevant because the principle is the same: give the model the right information, not all the information.