Context engineering for production AI agents¶

Prompt engineering is the discipline of writing the right words for a single turn. Context engineering is the discipline of curating everything that ends up in the model's context window across an entire agent run — system prompt, tool definitions, retrieved documents, conversation history, scratchpad notes, sub-agent outputs. Anthropic named this discipline in September 2025¹ because production agents kept hitting the same wall: the model was smart enough, but the context it saw was wrong.

This post covers what context engineering is, the four strategies you actually use, the failure modes that show up in production, and the tooling that does each one well.

Why prompt engineering stops working at agent scale¶

For a one-shot classification or a single chat turn, prompt engineering is enough. You write the system prompt, you test it, you ship it. The model only sees what you wrote, and what you wrote is what determines the output.

Agents break this assumption. An agent in a loop generates more and more data on every turn:

• Tool definitions (function schemas, MCP servers, plugin manifests)
• Tool results (file contents, API responses, query results)
• Retrieved documents (RAG chunks, knowledge base hits)
• Conversation history (prior turns, scratchpad notes, sub-agent outputs)
• System state (current task, plan, working memory)

Each turn adds tokens. At some point you exceed the model's effective context window, or you stay under the limit but the model starts ignoring early instructions ("lost in the middle"). Anthropic's own data on long-context recall shows degradation above ~50K tokens even on Claude Sonnet 4.6¹.

Context engineering is the answer: a set of strategies for deciding what goes into the context window, when, and how much, so the model gets the information it needs to produce the right output on this turn — not a pile of historical text it has to wade through.

The four strategies¶

Anthropic's article groups context engineering into four operations¹. The names are clean; the production reality is messier. I'll cover each with concrete tooling and the failure mode it solves.

1. Write context: scratchpads, memory files, structured notes¶

What it is: the agent writes information out to a persistent or semi-persistent store, then reads it back later. The store might be a file, a vector DB, a key-value cache, or a structured note inside the conversation itself.

When you need it: when the information the agent needs is too large to keep in the active context window, or when the agent needs to remember something across sessions.

Production pattern — files-as-context:

# Agent scratchpad pattern
from pathlib import Path

class Scratchpad:
    def __init__(self, agent_id: str):
        self.path = Path(f"/tmp/agents/{agent_id}/notes.md")
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def write(self, section: str, content: str):
        with self.path.open("a") as f:
            f.write(f"\n## {section}\n\n{content}\n")

    def read_recent(self, max_chars: int = 4000) -> str:
        if not self.path.exists():
            return ""
        text = self.path.read_text()
        return text[-max_chars:]  # last N chars

The agent reads notes.md at the start of each turn, writes findings to it as it works. When the conversation exceeds the context window, the scratchpad survives; new turns pull recent notes.

Production pattern — structured memory in LangGraph:

from langgraph.graph import StateGraph, MessagesState
from langgraph.checkpoint.postgres import PostgresSaver

# Persistent memory across turns + sessions
graph = StateGraph(MessagesState)
graph.add_node("agent", call_model)
graph.add_node("tools", tool_node)

DB_URI = "postgresql://user:pass@localhost:5432/agent_state"
checkpointer = PostgresSaver.from_conn_string(DB_URI)

app = graph.compile(checkpointer=checkpointer)

# Each thread_id maps to a persistent conversation
config = {"configurable": {"thread_id": "user-12345"}}
result = app.invoke({"messages": [...]}, config=config)

LangGraph 1.2.9 (2026-07-10)³ stores the full message state in Postgres. The agent can pause, resume days later, and pick up exactly where it left off. The "context" lives in the database, not in the model window.

Failure mode it solves: the agent forgets what it was doing 30 minutes ago. Write-context keeps the durable state outside the window.

2. Select context: RAG, memory recall, just-in-time loading¶

What it is: when the agent needs information that isn't in the current context, it pulls it from a larger store. The store might be a vector database, a SQL database, an MCP server, or the filesystem.

When you need it: when the total information is too large to fit in context (a 10M-row database, a 100K-document corpus), or when the agent's needs change turn by turn.

Production pattern — LlamaIndex retrieval:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.retrievers import QueryFusionRetriever

documents = SimpleDirectoryReader("docs/").load_data()
index = VectorStoreIndex.from_documents(documents)

# Hybrid retrieval: vector + BM25 + reranking
retriever = QueryFusionRetriever(
    retrievers=[
        index.as_retriever(similarity_top_k=10),
        index.as_retriever(similarity_top_k=10),  # second retriever for BM25
    ],
    num_queries=3,        # generate 3 query variants
    use_async=True,
    similarity_top_k=5,   # final top-k after fusion
)

nodes = retriever.retrieve("How does DRA migration work?")
context = "\n\n".join(n.get_content() for n in nodes)

LlamaIndex 0.14.23 (2026-06-24)⁴ handles embedding, hybrid retrieval, and reranking. The agent calls retriever.retrieve(query) and gets back the 5 most relevant chunks for this turn — not the entire document corpus.

Production pattern — tool-based loading:

# MCP server as context source
tools = [
    {"name": "search_docs", "description": "Search internal documentation"},
    {"name": "query_database", "description": "Run SQL against the analytics DB"},
    {"name": "fetch_file", "description": "Read a file by path"},
]

The agent calls fetch_file or query_database on demand. Only the tool result lands in context, not the entire file system or database.

Failure mode it solves: the agent needs information that doesn't fit in context, or that wasn't relevant 10 turns ago.

3. Compress context: summarization, trimming, compaction¶

What it is: reduce the size of context that's already there. Summarize old conversation turns. Drop tool results that are no longer relevant. Compact the agent's scratchpad.

When you need it: when context is growing toward the window limit but the information is still potentially relevant. You don't want to drop it entirely; you want it smaller.

Production pattern — turn-level summarization:

from langchain_core.messages import HumanMessage, SystemMessage, RemoveMessage

def summarize_old_messages(state):
    messages = state["messages"]
    if len(messages) <= 10:
        return state  # nothing to compress yet

    # Keep last 6 verbatim, summarize the rest
    old = messages[:-6]
    recent = messages[-6:]

    summary_prompt = [
        SystemMessage(content="Summarize this conversation in 500 tokens or less. Preserve: decisions made, files touched, errors encountered, current task state."),
        HumanMessage(content="\n".join(m.content for m in old if isinstance(m.content, str)))
    ]
    summary = chat_model.invoke(summary_prompt)

    # Replace old messages with the summary
    return {"messages": [RemoveMessage(id=m.id) for m in old] + [
        SystemMessage(content=f"Earlier conversation summary: {summary.content}")
    ] + recent}

Run this every N turns (every 10, every 20, depending on traffic). The conversation history stays bounded; the agent still knows what happened before.

Production pattern — tool result trimming:

def trim_tool_results(messages, max_chars=2000):
    for msg in messages:
        if msg.type == "tool":
            content = msg.content if isinstance(msg.content, str) else str(msg.content)
            if len(content) > max_chars:
                msg.content = content[:max_chars] + f"\n... [{len(content)-max_chars} chars truncated]"
    return messages

Long tool outputs (a 50K-line log file, a 10MB JSON response) get trimmed before they pollute context. The full result is still available in the tool execution store; only a truncated version lives in the model's view.

Production pattern — context caching as compression:

Anthropic's prompt caching lets you mark segments as cacheable². The model provider keeps the cached prefix across calls; subsequent calls only pay the marginal cost of the new tokens. The compression here is cost, not tokens — the context is the same size, but you pay for it once instead of per-call.

Failure mode it solves: context is bloated with stale information that the agent still references.

4. Isolate context: sub-agents, scopes, sandboxes¶

What it is: split a large context problem into smaller, isolated contexts. Each sub-agent or sub-task gets its own context window. Only the relevant output flows back to the parent.

When you need it: when one big context can't hold everything, or when different parts of the task need different kinds of information (and don't need to see each other's data).

Production pattern — multi-agent with LangGraph:

from langgraph.graph import StateGraph, Send
from typing import TypedDict

class ResearchState(TypedDict):
    query: str
    sub_results: list[str]
    final_answer: str

def dispatch_research(state):
    # Spawn 3 sub-agents, each with their own context
    return [
        Send("research_subagent", {"topic": "pricing", "query": state["query"]}),
        Send("research_subagent", {"topic": "features", "query": state["query"]}),
        Send("research_subagent", {"topic": "competitors", "query": state["query"]}),
    ]

def research_subagent(state):
    # Each sub-agent has its own ~50K-token context
    # It only sees its topic + the original query
    return {"sub_results": [run_research(state["topic"], state["query"])]}

def synthesize(state):
    # Parent only sees the 3 sub-results (~5K tokens)
    # Not the full research sub-agent contexts
    return {"final_answer": synthesize_results(state["sub_results"])}

graph = StateGraph(ResearchState)
graph.add_node("synthesize", synthesize)
graph.add_conditional_edges("synthesize", dispatch_research, ["research_subagent"])
graph.add_node("research_subagent", research_subagent)
graph.add_edge("research_subagent", "synthesize")
app = graph.compile()

Each research_subagent runs in its own context window (with its own scratchpad, retrieval calls, tool results). The parent agent never sees the sub-agent's internal context — only the final summary it returns. This is how Anthropic describes multi-agent research in their own production agents¹.

Production pattern — context isolation through MCP:

# Per-task MCP server scope
async with mcp_session("research-tools") as session:
    tools = await session.list_tools()
    # Only research tools are visible to this sub-agent
    result = await agent.run(query, tools=tools)

The agent sees only the tools loaded into its MCP session. A finance sub-agent gets finance tools; a code sub-agent gets code tools. Cross-contamination is impossible because the tools aren't in scope.

Failure mode it solves: one context window can't hold the entire task without losing coherence.

What the production failure modes actually look like¶

In practice, these failure modes show up more often than the strategies themselves:

Lost-in-the-middle. The model sees 80K tokens but only attends to the first 20K and the last 5K. Symptom: it forgets the system prompt or the original task. Fix: write-context to keep critical state outside the window, or compress aggressively.

Tool result bloat. One tool call returned 50K of JSON. Symptom: subsequent turns are slow and expensive; the model can't find the one field it needed. Fix: trim tool results aggressively, or write the full result to a file and pass only the path to the model.

Retrieval pollution. The retriever returns 10 chunks, but 8 are irrelevant. Symptom: the model hallucinates based on the noisy context. Fix: tighter retrieval (hybrid + reranking), or smaller chunks, or post-retrieval filtering by the model itself.

Context thrashing. Every turn the model rewrites its plan based on new context. Symptom: the agent loops, makes no progress. Fix: isolate the planning context from the execution context — the planner shouldn't see every tool result.

Stale scratchpad. The agent wrote something to its notes 50 turns ago that's now wrong. Symptom: it references outdated state. Fix: timestamp notes, periodically re-summarize, or have the agent validate notes before using them.

Cascading sub-agents. Every sub-agent's full output flows back to the parent, blowing the parent's window. Symptom: parent context fills up despite "isolation." Fix: sub-agents return summaries, not full transcripts.

Observability for context engineering¶

You can't fix what you can't measure. For production context engineering you need to log, per turn:

• Token counts — input tokens, output tokens, cache hits/misses
• Context composition — what fraction came from system prompt, tools, retrieval, history, scratchpad
• Retrieval stats — what was retrieved, what was actually used in the response
• Tool call counts — per turn, per agent, per sub-agent
• Sub-agent spawns — when, why, how many, what they returned

Langfuse 3.218.0 (2026-07-16)⁵ does all of this out of the box with LangChain and LangGraph. OpenLLMetry and Arize Phoenix are the open-source alternatives.

A minimal Langfuse instrumentation:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

langfuse = Langfuse(
    public_key="pk-...",
    secret_key="sk-...",
    host="https://cloud.langfuse.com"
)

handler = CallbackHandler()

result = app.invoke(
    {"messages": [HumanMessage(content="Research DRA migration")]},
    config={
        "callbacks": [handler],
        "configurable": {"thread_id": "user-12345"}
    }
)

After running, Langfuse shows you the token composition, the retrieval calls, the tool calls, the latency — broken down by turn. That's how you know whether the agent's context is healthy or whether you've got a lost-in-the-middle situation brewing.

What I couldn't verify¶

• Exact token thresholds where context degradation kicks in for Claude Sonnet 4.6 / Opus 4.6. Anthropic publishes general guidance ("above ~50K tokens recall degrades")¹ but the precise cliff varies by task and prompt structure.
• Whether retrieval reranking universally beats raw vector search. Reranking costs latency and money. Published benchmarks (BEIR, TREC) suggest reranking helps for short queries, hurts or is neutral for long, specific queries — but the trade-off curve depends on your corpus.
• Multi-agent token economics in production. Anthropic notes that multi-agent research uses substantially more tokens than a single chat conversation — published numbers from their own research agent land around 4× — but the multiplier varies by architecture and task¹. Sub-agent designs that fan out to many parallel researchers can multiply token spend much higher.
• MCP server scoping as a true isolation mechanism. MCP sessions do provide tool-name isolation, but cross-session state (file system, network) is still shared unless explicitly sandboxed.
• "Sub-agent returns summary" best practice. It's standard advice but I couldn't find a published benchmark showing optimal summary length as a function of parent context size.

Summary¶

• Context engineering is what comes after prompt engineering. It's the discipline of curating what ends up in the model window across an entire agent run.
• Four operations: write (scratchpads, memory), select (RAG, tools), compress (summarization, trimming), isolate (sub-agents, scopes).
• Failure modes: lost-in-the-middle, tool result bloat, retrieval pollution, context thrashing, stale scratchpad, cascading sub-agents.
• Measure it. Token counts, retrieval stats, tool call counts, sub-agent spawns — without observability, you're tuning blind.
• Tooling in 2026: LangGraph 1.2.9 for graph state, LlamaIndex 0.14.23 for retrieval, Langfuse 3.218.0 for observability. Anthropic prompt caching for cost compression on repeated prefixes.

The shift from "writing the right prompt" to "managing the right context" is the difference between a demo and a production agent. The four operations are how you do it.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.