What the context window actually is
The context window is the maximum amount of text Claude can see in a single API request — measured in tokens. It's a hard limit baked into the model itself.
Every API call you make, Claude reads:
System prompt
+ Conversation history (all prior turns)
+ Tool schemas
+ Tool results
+ Current user message
= Total tokens consumed in this requestThat total cannot exceed the context window limit. If it does, the request fails or older content gets truncated.
Who holds it, who owns it
The model owns the limit — it's a physical constraint of how the model was trained and architected. Claude 3.5 Sonnet has a 200,000 token window for example. You cannot change that number.
But you own what goes inside it — every API call, you decide what to include in the request body. The window is not a server-side session object that persists. It's just the total size of what you send in one request.
Your HttpSession analogy — close but not quite
| HttpSession | Context Window | |
|---|---|---|
| Lives on server | Yes, persists between requests | No — exists only within one request |
| Who manages it | Server holds state | You rebuild it every request |
| Size limit | Configurable | Fixed by the model |
| Survives between calls | Yes | No |
The key difference: Claude has zero memory between API calls. There is no server-side session. You reconstruct the entire conversation history yourself on every request by passing it in the messages array. The window is just the size limit of that payload.
What the statement means concretely
In development you send small test inputs, so your total payload stays small — maybe 2,000 tokens per request. Fine.
In production:
Turn 1: System prompt (500) + user message (50) = 550 tokens
Turn 3: Above + 2 tool calls + 2 tool results = 3,000 tokens
Turn 5: Above + more turns + tool outputs = 8,000 tokens
Turn 8: Above + more tool results (3-5x larger than dev) = 195,000 tokens
Turn 9: EXCEEDS 200,000 token limit → request failsIn dev your tool results were small mock fixtures. In production they're real API responses — verbose JSON payloads 3-5x larger. The window fills at turn 8 instead of turn 50. That's the production outage.
Bottom line
The context window is the total size limit of a single API request payload, owned by the model, not a session object. You own what goes in it — and in production, tool outputs, conversation history, and tool schemas stack up fast and hit that ceiling much earlier than your dev tests ever showed.
Strategies for staying in budget
Pruning — Rewinds the conversation to an earlier point and discards everything after it. Use this when the conversation went down a wrong path and the back-and-forth since then adds noise rather than value. The cost is losing anything useful Claude figured out in the discarded stretch.
Compaction — Summarizes the full conversation history into a shorter version that preserves key facts and decisions without keeping every turn. Use this when the session is getting long but Claude has built up knowledge you want to carry forward. The cost is summarization loss — details that don't make it into the summary are permanently gone.
Clearing — Wipes the context entirely and starts fresh with an empty window. Use this when the next task is unrelated to the current one and carrying old context would bias or confuse Claude rather than help it. The cost is total — nothing survives unless you explicitly saved it somewhere persistent like a file or memory store.
Subagent Handoffs — Spins up a separate agent with its own isolated context window, gives it only what it needs for one specific subtask, and gets back just the result. Use this when a subtask is self-contained and its working steps would clutter the main conversation. The cost is visibility — you see the answer but not how the subagent got there.
Prompt Caching — Stores the processing work done on a stable, unchanging prefix of your request so subsequent requests reuse it instead of reprocessing the same tokens every time. The strongest candidates are things that don't change across turns — your system prompt, tool schemas, or a large reference document you query repeatedly. You mark what to cache using a cache_control: ephemeral breakpoint, up to four per request, on the last block you want cached. For long multi-turn sessions this is the highest-leverage cost reduction available — you pay full price once on the first request, and a fraction of that on every follow-up that hits the same prefix.
Token Counting — Lets you measure how many tokens a request will consume before you send it, using the count_tokens endpoint which takes the same request body as a normal messages call but runs no inference. Use it in development to verify your context budget assumptions hold against real tool outputs rather than small test fixtures, and in production to gate requests that would breach the window limit before they fail with an error. The four context management strategies above decide what goes in the window — token counting tells you how close to the ceiling you actually are before it's too late.
Most commonly used in Production:
Prompt Caching — Almost universal in production. If you have a system prompt, tool schemas, or reference documents that don't change across requests, there's no reason not to cache them. It's low effort to implement and directly cuts input token costs on every request. Most production teams enable this on day one.
Token Counting — Very commonly used as a defensive gate in production pipelines, especially in agentic loops where tool outputs are unpredictable in size. Teams use it to catch window overflow before it causes a live outage rather than discovering it from a failed request.
Situationally common:
Compaction — Common in long-running agentic workflows like coding assistants or multi-step planners where sessions naturally grow long. Claude Code uses it heavily. Less relevant for short request-response style APIs.
Clearing — Common in chatbot or assistant products where each new user task should start fresh. Simple to implement — just don't carry history forward.
Less common in typical production:
Pruning — Useful but operationally awkward. Requires your application to track message indices and manage conversation state carefully. More common in developer tooling than in end-user facing products.
Subagent Handoffs — Common in sophisticated multi-agent architectures but overkill for most standard production deployments. Teams reach for this only when tasks are genuinely complex enough to warrant isolated sub-contexts.
What caching saves is not token transmission — it's token processing
You're right that you still send the same content in every request. The system prompt, tool schemas — all of it still goes in the request body every time. Caching does not remove that.
What caching saves is the computational work Claude does to process those tokens on the server side.
What actually happens without caching:
Request 1: Claude reads and processes system prompt (500 tokens)
Claude reads and processes tool schemas (300 tokens)
Claude reads and processes user message (50 tokens)
→ You pay full price for all 850 tokens
Request 2: Claude reads and processes system prompt (500 tokens) ← same work again
Claude reads and processes tool schemas (300 tokens) ← same work again
Claude reads and processes user message (60 tokens)
→ You pay full price for all 860 tokensClaude is reprocessing the same system prompt and tool schemas from scratch on every single request.
What happens with caching:
Request 1: Claude processes system prompt + tool schemas (800 tokens)
→ Saves that processed state to cache
→ You pay full price this time (cache write)
Request 2: Claude retrieves cached state for system prompt + tool schemas
Claude only processes the new user message (60 tokens)
→ You pay full price for 60 tokens
→ You pay ~10% price for the 800 cached tokens
It happens entirely on Anthropic's server side — you have no infrastructure to manage for this.
Where the cache lives
Your Application Anthropic's Servers
───────────────── ──────────────────────────────
Sends full request → Receives request
(same as always) Checks: has this prefix been
cached recently?
YES → skips reprocessing it
uses saved internal state
NO → processes it fresh
saves it to cache
← Returns response
The cache is Anthropic's internal infrastructure. You don't set up Redis, a vector store, or anything else. You don't manage cache invalidation. You don't store anything on your side.
What "cached state" actually means
When Claude processes tokens, internally it computes something called KV cache (Key-Value cache) — intermediate mathematical representations of those tokens. This is the expensive computation you're paying for.
Without prompt caching: those KV computations are thrown away after each request.
With prompt caching: Anthropic saves those KV computations on their servers for 5 minutes (ephemeral cache). The next request that sends the same prefix skips recomputing them entirely.
Your only job is marking the breakpoint
json{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a support classifier...[long system prompt]",
"cache_control": { "type": "ephemeral" } ← you add this
}
]
}
That marker tells Anthropic's server: "cache everything up to this point." The actual caching infrastructure, storage, and retrieval all happen on their side. You just send the same request body as always, with that one extra field.
Bottom line
This is purely server-side on Anthropic's end. You send the same request as always — the only change is adding cache_control to mark what should be cached. Anthropic's servers store the internal processing state for 5 minutes and reuse it on matching requests. Zero infrastructure on your side.
The LLM itself is still completely stateless
The model — the neural network that does the actual thinking — has zero state between calls. Every request is independent. Nothing about your session is remembered.
What gets cached is not session state. It's preprocessed mathematical representations of specific token sequences.
How Anthropic's server identifies a cache hit
It's not mapped to you, your session, or your API key. It's mapped purely to the content itself — like a hash:
Cache key = hash of the exact token sequence up to the breakpoint
Request 1 sends: "You are a support classifier... [500 tokens]"
→ hash(those 500 tokens) = abc123
→ compute KV state, store it under key abc123
Request 2 sends: "You are a support classifier... [same 500 tokens]"
→ hash(those 500 tokens) = abc123
→ cache hit! retrieve stored KV state
→ skip recomputation
Anyone sending that exact same prefix within 5 minutes gets a cache hit — even different users, different sessions. The key is the content, not the caller.
The analogy that fits perfectly
Think of it like a CDN caching a static HTML page:
CDN doesn't know or care who requested the page
It just checks: do I have a cached copy of this URL?
YES → serve cached copy
NO → fetch fresh, cache it
Anthropic's prompt cache works the same way:
Server doesn't know or care which session this is
It just checks: do I have cached KV state for this exact token prefix?
YES → reuse it
NO → compute fresh, cache it
So what exactly is stored for 5 minutes?
Not your conversation. Not your session. Just the KV (Key-Value) attention matrices — the intermediate mathematical output of processing those specific tokens through Claude's layers. It's a computation artifact, not user state.
Without cache: tokens → [expensive matrix computation] → KV state → answer
With cache hit: tokens → [skip computation] → retrieve KV state → answer
Bottom line
LLM is still stateless in every meaningful sense — no session, no memory, no user tracking. What gets cached for 5 minutes is purely a mathematical computation artifact tied to a specific token sequence hash. If you send the same prefix, you get the same cached computation back — regardless of who you are or what session you're in. Anthropic is caching work, not state.
is 5 minutes configurable ?
It is fixed, not configurable. You cannot change it.
Cache type TTL Configurable? ephemeral 5 minutes No
That's the only cache type currently available. There is no persistent or long-lived variant you can set today.
The practical implication
5 minutes works well for high-frequency steady traffic — if requests are hitting the same prefix continuously, the cache stays warm. But if your traffic is bursty or low-volume with gaps longer than 5 minutes between requests, the cache expires and the next request pays full price to rewrite it.
Request 1 → cache write (full price)
Request 2 → cache hit (2 min later, ~10% price)
Request 3 → cache hit (4 min later, ~10% price)
--- 6 minutes pass ---
Request 4 → cache miss (cache expired, full price again)
Request 5 → cache hit (2 min later, ~10% price)
