TL;DR: LLM providers tout context windows in the millions of tokens. New research shows that most frontier models degrade significantly within ~1,000 tokens of input, long before reaching their stated limit. On-call and SRE agents accumulate logs, traces, and chat history fast. That makes them less reliable exactly when an incident gets complex. The fix isn't a bigger context window. It's not asking the LLM to reason over raw telemetry in the first place.

Why this matters now

As of mid-2026, many reliability teams have wired an LLM into some part of on-call: triage summaries, an agent that reads alerts and proposes causes, a chat interface over observability data. Most of these systems work by pushing more context into the prompt. More logs, more history, more runbooks. A study across 11 frontier models, including GPT-5, Claude 3.5 Sonnet, Gemini 2.5, and DeepSeek R1, found that this instinct is off. Accuracy on realistic tasks started dropping at a few hundred to a few thousand tokens, not at the million-token ceilings vendors publish. An on-call agent whose context grows with the incident gets less trustworthy exactly when you need it most.

How much of a model's context window can LLMs actually use?

LLMs do not effectively use nearly as much of the context window as the spec sheets claim. Paulsen's study of 11 frontier models introduced the concept of the Maximum Effective Context Window (MECW): the point at which additional input begins to degrade a model's output. That's distinct from the Maximum Context Window (MCW) that providers advertise. The gap was large. Most models showed severe accuracy degradation by roughly 1,000 tokens, a few failed with as little as 100 tokens in context, and every model tested fell short of its advertised MCW by up to 99%+.

This isn't a new observation. Liu et al. (2023) first documented that models favor information at the start and end of a prompt, while losing content buried in the middle. Researchers call this "lost in the middle." Chroma's 2025 technical report tested 18 models, including GPT-4.1, Claude 4, and Gemini 2.5, and found that all of them degraded as input length increased, even on simple retrieval tasks (Hong et al., 2025). Paulsen's contribution shows that this holds across a wider range of task types and the specific token counts in which it occurs. Product manager Teresa Torres ties this same body of research to day-to-day AI usage in her write-up on context rot, worth reading if you want the pragmatic view alongside the research.

Do more reasoning steps mean worse accuracy?

What predicts accuracy is how much data a model must actively track when it produces its final answer, not how many logical steps the task requires. Paulsen's study found that summarization, which sounds simplest since it's a single operation, performed worse than the find-and-sum task, which requires two. Find-and-sum lets the model discard irrelevant facts early, so only a short filtered list has to stay in context for the final calculation. Summarization has no discard step, so the model has to keep the entire dataset in context throughout.

That's the mechanism: fewer steps don't help if the model still has to track the full dataset at the end. Filtering out what's irrelevant before the final reasoning step is what protects accuracy.

Why are on-call and SRE agents especially exposed?

Incident triage is a multi-step, context-accumulating task. Every additional signal an agent pulls in (a new log line, a related trace, an escalation message) pushes it further past the point where the next reasoning step degrades. An agent investigating a cascading failure doesn't get a clean, bounded prompt. It collects logs, metrics, topology, runbooks, and a growing chat history, all of which are appended as the investigation continues. That's close to the worst case in Paulsen's data: a multi-step reasoning task over an expanding context window.

Orchestrate agents together, and the math gets worse fast. Three agents at 70% accuracy each, a modest and individually reasonable number, compound to a 34.3% system success rate. Each agent's 70% is often measured near or beyond its effective context limit, which makes the actual number in production likely lower.

Is this settled, or is there a counterargument?

Some newer models show little to no "lost in the middle" effect on simple fact retrieval. Reports on Gemini 2.5 Flash describe near-perfect recall regardless of where a fact sits in a long document. Epoch AI's analysis found that the input length at which top models hit 80% accuracy on long-context benchmarks has risen by more than 250x over nine months. Providers are getting better at simple retrieval over long inputs.

This depends on the task type. The counterevidence focuses on single-fact lookup, the easiest category in Paulsen's study and the one closest to being solved. The degradation that persists, and that researchers all converge on, is worst on multi-step reasoning: summarizing, sorting, correlating multiple signals into one conclusion. That's precisely what root cause analysis is.

What does a causal model do differently?

A causal model doesn't hand the LLM the raw data. In Causely's architecture, causal inference over your service and resource dependencies runs deterministically upstream of any LLM call. By the time a model is involved, it isn't reasoning over raw logs, traces, and a sprawling incident history. It's working from a small set of named diagnoses that the causal model has already isolated.

A raw incident context (logs, traces, topology dumps, chat history) can easily run into the thousands of tokens before an LLM even starts reasoning. A causal summary naming the one or two services responsible is a small fraction of that. The LLM's job shrinks from "find the cause in everything we've collected" to "remediate a cause we already found." That keeps it well inside the range where these models are still reliable.

What this means for your agentic architecture

To increase your agent's accuracy, you should treat effective context, not advertised context, as your design constraint. Scope each agent to a narrow task rather than one agent doing full triage end-to-end. Bound how much raw telemetry any single LLM call receives, regardless of the model's advertised window. Prefer structured, pre-filtered summaries (causal, symbolic, or otherwise) over raw data dumps whenever a model has to reason, not just retrieve. An agent whose accuracy drops as an incident drags on doesn't need a longer context window. It needs a shorter, engineered one.

FAQ

What is the Maximum Effective Context Window (MECW)?
MECW is the point in a model's input length where additional tokens start measurably degrading its output, as distinct from the Maximum Context Window (MCW) providers advertise. Research across 11 frontier models found MECW is often a small fraction of MCW, sometimes under 1,000 tokens even when the model supports millions.

Doesn't a bigger context window just solve this?
No. Bigger windows raise the ceiling, not the floor. Providers have made real progress on simple factoid retrieval over long inputs, but multi-step reasoning tasks like summarization and root cause correlation still degrade well before the window fills up, according to multiple independent studies.

How do I keep an on-call agent inside its effective context window?
Scope each agent to a narrow task, cap how much raw telemetry any single LLM call receives, and filter or summarize before the model reasons rather than after. An agent that reasons over a pre-filtered set of candidate causes will typically outperform one reasoning over the full incident history.

Is this a RAG problem or a context-window problem?
Both, one causes the other. Retrieval-augmented generation reduces hallucination when the retrieved content stays inside the model's effective context window, but research shows accuracy actively worsens once retrieval pulls in more than that. Naive RAG that isn't context-aware can make context rot worse, not better.

What to do next
If you want the token-cost side of this problem, read our benchmark on why AI agents burn tokens on every reliability query, or see how Causely turns an incident into a small set of named diagnoses before any LLM is involved.

Your on-call AI agent is wrong more than you think