Context Engineering and Memory

From prompt engineering to context engineering

In the chatbot era, most of the craft was in the prompt: clear instructions, a few good examples, a request to think step by step. Those skills still matter. But an agent's context on turn 40 is mostly not something a person wrote. It is a pile of tool definitions, earlier messages, file contents, search results, error logs and its own past reasoning.

Anthropic's September 2025 essay "Effective context engineering for AI agents" named the shift. Context engineering is the job of curating the whole set of tokens the model sees on every turn, not just the instructions. Its stated goal is to find the smallest set of high-signal tokens that maximizes the chance of the outcome you want.

The attention budget and context rot

Context windows have grown enormously. Several 2026 models, including Claude Opus 5, Claude Sonnet 5.5 and Gemini 4 Argon, offer a context of around one million tokens. It is tempting to conclude that you can just put everything in. That conclusion is wrong, and understanding why is the foundation of this module.

Anthropic frames context as a finite attention budget. Every token you add draws down the model's ability to focus on the others. As context grows, models show context rot: their ability to accurately recall and use any particular piece of information declines. Fitting in the window is not the same as being used well.

• Attention is spread across pairs of tokens. In a transformer, every token can attend to every other one. As the context grows, the number of relationships grows much faster than the number of tokens, and each one gets a thinner slice of attention.
• Training data is mostly shorter. Models see far more short sequences than extremely long ones during training, so they are less practiced at using information buried deep in a long context.
• Noise looks like signal. An old, irrelevant tool result is still text the model may treat as instructions or evidence. Distractors hurt more than empty space.

System prompts, AGENTS.md and CLAUDE.md

The system prompt is the one part of the context you fully control, so it is worth getting right. Anthropic describes a "right altitude" between two failure modes. Too low, and you hard-code brittle if-then rules for every edge case. Too high, and you give vague guidance that assumes the model shares context it doesn't have. Aim for clear principles plus a few concrete examples, organized into labeled sections.

For coding agents, project-level instructions now live in files checked into the repository. AGENTS.md, contributed by OpenAI, is a README written for agents rather than humans, and it became one of the founding projects of the Agentic AI Foundation in December 2025. Claude Code reads CLAUDE.md files for the same purpose. The agent loads them automatically at the start of a session, so every run begins with the project's conventions.

# AGENTS.md

## Commands
- Install: `pnpm install`
- Test one package: `pnpm --filter <pkg> test`
- Lint before committing: `pnpm lint`

## Conventions
- TypeScript strict mode; no `any` in new code.
- Database access goes through `src/lib/db` only. Never write raw SQL.
- Prices are stored in USD as decimals, not cents.

## Gotchas
- The `legacy/` folder is frozen. Do not edit it.
- Integration tests need `DATABASE_URL`; skip them if it is unset.

Just-in-time retrieval

The traditional way to give a model knowledge is to retrieve likely-relevant documents up front (classic RAG) and paste them into the prompt. That works for single questions, but it has two problems for agents: you have to guess what will be relevant before the work starts, and everything you retrieve sits in context for the rest of the task.

Agents increasingly use just-in-time retrieval instead. The context holds lightweight handles, such as file paths, URLs, table names or search queries, and the agent loads the actual content with tools only when it needs it. Claude Code is the standard example: rather than indexing a whole codebase into the prompt, it uses tools like grep and glob to find files, then reads just the parts that matter.

Handles carry useful information on their own. A file called tests/billing/test_refunds.py tells the agent a lot before it opens it. Folder structure, naming, file sizes and timestamps all help an agent decide what to read, which is one reason well-organized projects are easier for agents to work in.

Keeping long tasks on track

Even with careful retrieval, an agent that runs for hours will eventually produce more history than it can usefully attend to. Four techniques handle this, and production agents usually combine several.

Anthropic reported that, on an internal long-horizon evaluation, context editing (clearing stale tool results) improved performance by 29%, and that adding its file-based memory tool cut token consumption by 84% over a 100-turn run. These are vendor-reported figures on Anthropic's own evaluation, but the direction matches what practitioners see: removing junk from context often helps more than adding more information.

Sub-agent isolation deserves special attention because it changes the shape of the system. In Anthropic's multi-agent Research system, each sub-agent explores one line of inquiry with its own context window, then returns a short distilled summary to the lead agent. The lead's context stays clean, even though the sub-agents together may read far more material than any single context could hold. Module 6 covers the cost side.

Agent Skills and progressive disclosure

Tool definitions and instructions are context too. An agent that knows how to fill in tax forms, edit spreadsheets, and follow your brand guidelines does not need all three sets of instructions in context while it fixes a CSS bug. Progressive disclosure means showing the model a short index first and loading the details only when a task needs them.

Agent Skills, introduced by Anthropic and published as an open standard at agentskills.io on December 18, 2025, apply this idea. A skill is a folder containing a SKILL.md file plus any scripts or reference files it needs. Only each skill's name and short description sit in context by default. When a task matches, the agent reads the full SKILL.md, and from there it can open further files or run the bundled scripts.

---
name: quarterly-report
description: Build the finance team's quarterly report from the
  ledger export. Use when asked for a quarterly or Q-report.
---

# Quarterly report

1. Run `scripts/load_ledger.py <export.csv>` to clean the data.
2. Follow the section order in `reference/template.md`.
3. Charts: use `scripts/charts.py`; never hand-draw numbers.
4. Check totals against `reference/checks.md` before finishing.

Secondary sources report that the format has been adopted well beyond Anthropic, including by OpenAI's Codex and ChatGPT, VS Code, Gemini CLI and goose. The same principle shows up in tool search or deferred tool loading: instead of putting every tool definition from every connected MCP server into the context, the agent gets a search tool and loads full definitions only for the tools it is about to use. Module 4 covers this in more detail.

Memory architectures

Everything so far is about managing context within a task. Memory is about carrying useful information across tasks and sessions: the user's preferences, facts about a project, lessons from past mistakes. Since the model itself is stateless, all agent memory is ultimately a system that decides what to write down, where to keep it, and what to put back into context later.

The tiered design from MemGPT (now Letta) borrows from operating systems: a small fast tier in context, like RAM, and larger slower tiers outside it, like disk, with the agent itself deciding what to page in and out. File-based memory is simpler and has become popular because it is transparent. You can open the file, see exactly what the agent remembers, and fix it.

Consolidation is the newest idea. Rather than letting memories pile up, a background process periodically reorganizes them, much as people consolidate memories during sleep. Secondary sources report that Claude Managed Agents shipped a "dreaming" feature along these lines as a research preview in May 2026. Treat this as an early, evolving pattern.

Measuring memory, with caveats

Several benchmarks try to measure how well an agent remembers over long interactions. LoCoMo and LongMemEval test recall and reasoning over long multi-session conversations. BEAM and PersonaMem are newer additions, the latter focused on tracking a user's evolving preferences.

Read published scores carefully. Mem0, for example, reports 92.5 on LoCoMo and 94.4 on LongMemEval on its own blog. These numbers are self-reported, and different vendors use different configurations, model backbones and scoring methods, so they aren't comparable across vendors. A memory system that tops one vendor's chart may do worse on your data.

• Test memory on conversations that look like your real ones, not only on public benchmarks.
• Measure what you care about: correct recall, but also not recalling outdated facts after they change.
• Track cost and latency. A memory layer that adds a model call to every turn changes the economics of the agent.