Agent Harness Handbook

Appearance

Case Study: Aider

At a glance Aider is the open-source CLI pair-programming agent best suited for studying harness internals — the trade-offs of its edit format, its tree-sitter repo map, its lint/test feedback loops, and its deep git integration, each backed by verifiable benchmark data.

Case Study: Aider ​

Aider is an open-source CLI pair-programming agent created by Paul Gauthier in 2023. It has no GUI, no subagent orchestration, and no elaborate planning module — nearly all of its "intelligence" comes from just four things: the edit format, the repo map, the lint/test feedback loop, and git integration.

This is exactly what makes aider uniquely valuable for harness research: it is simple enough that the effect of every design decision can be isolated and measured. And aider's author has done precisely that — the officially maintained LLM leaderboard and the series of benchmark blog posts are probably the most rigorous experimental record in public literature on "how harness details affect success rates."

Why read this one first

In Model vs. Harness we argued that "the same model in different harnesses differs dramatically in capability." Aider is the best evidence for that claim: it has run experiments with the same pool of models for years, and by changing nothing but the format the model uses to output edits, success rates can differ by 3x.

Overall Structure: A Loop Built Around Edits ​

┌──────────────────────────────────────────────────────────┐
│  aider (Python CLI, runs in your local git repo)         │
│                                                          │
│   System prompt (with the edit format spec)              │
│        +                                                 │
│   repo map (tree-sitter symbol map, default ≤1k tokens)  │
│        +                                                 │
│   Full text of the files the user /adds into the session │
│        +                                                 │
│   Conversation history / lint / test error output        │
│        │                                                 │
│        ▼                                                 │
│   LLM ──── emits edits in edit format ────► parse &      │
│        ▲                                    flexible     │
│        │                                    apply        │
│        │                                      │          │
│        │                                      ▼          │
│        │                                  write files    │
│        │                                      │          │
│        │                     ┌─────────────────┼────────┐│
│        │                     ▼                 ▼        ▼│
│        │                  auto-lint    auto-test     git │
│        │                (errors fed back) (same)    auto │
│        └────── feedback ────┘                    commit  │
└──────────────────────────────────────────────────────────┘

Note what's not in this diagram: no task decomposer, no multi-agent, no long-term memory. Aider bets everything on one thing — maximizing the success rate of the core action of "getting the model to change code correctly." Let's take it apart piece by piece.

The Make-or-Break Factor: The Edit Format ​

Problem Definition ​

There is a parsing layer between an LLM "writing code" in a conversation and that code "being reliably written to disk as correct files." The input format of this parsing layer is the edit format — it is the output protocol between the harness and the model. Get the protocol wrong and no amount of model cleverness helps: edits fail to apply, apply in the wrong place, or the model writes worse code to accommodate the format.

Aider maintains a family of formats, dispatched according to model capability (you can force one with --edit-format):

FormatContentsWhen it's used
wholeReturns the updated entire fileWeaker models; small files; most reliable, but the most expensive and slowest
diff<<<<<<< SEARCH / ======= / >>>>>>> REPLACE search-and-replace blocksThe default for today's mainstream strong models
diff-fencedSame as diff, but the file path goes inside the code fenceThe Gemini family (they often fail to honor diff's fencing conventions)
udiffA simplified variant of the unified diffOriginally created to cure GPT-4 Turbo's "lazy coding"
editor-diff / editor-wholeTrimmed-prompt versions of diff/wholeUsed for the "editor model" in architect mode

What the diff format looks like (the syntax deliberately imitates git merge-conflict markers, because models have seen them countless times in their training data):

mathweb/flask/app.py
```
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```

Experiment 1: Plain Text Beats Function Calling (2023-07) ​

Aider's earliest benchmark was built on 133 Exercism Python exercises: the model reads the problem, edits the implementation file, runs the unit tests, and the benchmark measures how often it "applies the edit to disk correctly end to end and passes the tests."

The July 2023 results contained one counterintuitive finding: OpenAI had just released the function calling API, and the author expected it to make structured output more reliable, so he implemented two function-call-based formats, whole-func and diff-func. The result — the function call formats underperformed the plain text formats on every model tested. GPT-3.5 even hallucinated function calls that didn't exist: it returned "name": "python" and stuffed an entire Python file into the arguments field.

The author's explanation is worth remembering (it later became aider's methodology):

Imagine you ask a colleague on Slack to change some code. Would they rather paste in a markdown code block, or hand-type a correctly escaped, syntactically valid JSON data structure?

The more complex the output format, the more attention the model has to divert from "thinking about code" to "thinking about format," and the more likely format errors become. Lower the format's cognitive overhead, and the model's coding quality and its format compliance improve at the same time.

Experiment 2: udiff Takes GPT-4 Turbo from 20% to 61% (2023-12) ​

When GPT-4 Turbo shipped, users widely complained that it had become "lazy" — ask it to modify a large file and it would paper over the job with comments like # ... existing code unchanged .... Aider's author built a dedicated "lazy benchmark": AST-scanning 9 popular open-source Python projects for 89 refactoring tasks (lift a class method into a top-level function), with files large enough to induce lazy behavior.

Results on gpt-4-1106-preview:

ConfigurationScore
SEARCH/REPLACE block format (baseline)20%
udiff (simplified unified diff) format61%
Baseline + an emotional-appeal prompt ("the user is blind, has no hands, and will tip you $2,000")Worse than baseline
udiff + the same emotional appealAlso worse

Same model, only the output format changed, and the success rate improved roughly 3x. Meanwhile, the folk remedies circulating online — "sound pitiful, promise a tip" — turned out to be a negative optimization under controlled measurement. This is the textbook case for "settle harness decisions with benchmarks, not folklore."

Why does the unified diff work? The author's four principles:

  • FAMILIAR: the unified diff is the default output of git diff, and models have seen mountains of examples in their training data.
  • SIMPLE: aider explicitly tells the model not to write line numbers (@@ ... @@ instead of @@ -2,4 +3,5 @@), because experiments repeatedly showed that LLMs are bad at handling line numbers. Each hunk degenerates into a single search-and-replace.
  • HIGH LEVEL: the system prompt encourages the model to output "two coherent versions of the entire function" rather than line-by-line, surgical minimal edits. Remove that instruction and the edit error rate rises 30–50%.
  • FLEXIBLE: model-emitted diffs are often imperfect (missing comments, forgotten + prefixes, uniformly wrong indentation). Aider's applier (the apply logic) has a whole progression of lenient fallback strategies: normalize hunks, match with relative indentation, split oversized hunks into small ones and retry each, and so on. Disable flexible application and the edit error rate rises 9x.

Engineer both ends of the output protocol

Note that last point: a good edit format isn't just "specifying the format in the prompt" — it also includes a tolerant parser/applier on the consuming end. Model output will always have blemishes; the harness's job is to treat a 95-point output as if it were 100, not to throw away an entire round of work over one blemish (or pay the model to retry).

Today's Leaderboard: Format-Model Matching Continues ​

Aider's official leaderboard keeps measuring new models with the polyglot benchmark (a multilingual, Exercism-style exercise set). Besides pass rate, every entry also publishes a "percent using the correct edit format" column — meaning "does the model comply with the output protocol" is tracked long-term as a first-class metric, independent of "is the code correct." In the leaderboard snapshot taken when this page was written (including 2025 models such as GPT-5 and Gemini 2.5 Pro): GPT-5 (high) leads at 88.0% using the diff format, with a 91.6% format-compliance rate; Gemini 2.5 Pro uses diff-fenced; and some weaker open models still use the whole format.

The leaderboard itself tells a story: strong models converge on the diff family, weak models fall back to whole — the edit format is a harness parameter tiered by model capability; there is no silver bullet.

The Repo Map: Compressing the Entire Codebase with tree-sitter ​

The Problem: The Context Window Can't Fit the Whole Repository ​

For a model to change code in a large repository, it needs three kinds of information: where to change, what the relevant code looks like, and how to change it. Stuffing the whole repo into the context isn't realistic, and manually /add-ing files costs too much human effort. Aider's answer is the repo map — a symbol-level map of the entire git repository, sent to the model with every request.

The Mechanism ​

  1. Extraction: tree-sitter (the incremental parser widely used by IDEs and LSP servers) parses each source file into an AST, finding every symbol definition (classes, functions, methods, and their full signatures) and every reference site. Aider originally used ctags for this and switched to tree-sitter in October 2023, gaining richer signature information and out-of-the-box multilingual support (via the py-tree-sitter-languages package).

  2. Ranking: the repository becomes a graph — source files are nodes, dependency/reference relationships are edges — and a graph ranking algorithm runs on it to surface the most-referenced, and therefore most important, symbols.

  3. Trimming: the top-ranked symbol definitions are packed into a token budget (--map-tokens, 1024 tokens by default). The budget adjusts dynamically with session state: when the user hasn't /add-ed any files yet, aider substantially enlarges the repo map, trying to let the model understand the whole repository first.

Here's what the map looks like (an excerpt from aider's own repository):

aider/coders/base_coder.py:
⋮...
│class Coder:
│    abs_fnames = None
⋮...
│    @classmethod
│    def create(
│        self,
│        main_model,
│        edit_format,
│        io,
⋮...
│    def run(self, with_message=None):
⋮...

Why This Is a Brilliant Design ​

It lands precisely on the boundary between "what the model needs" and "what it doesn't":

  • The model doesn't need the full implementation of the BarLog subsystem — the signatures alone tell it how to call things.
  • The model does need to know which abstractions exist in the repository, so that when it writes new code it reuses them instead of reinventing them.
  • When the map isn't enough, the model can use the map as an index and actively ask to see specific files — aider adds those files to the session. The repo map is therefore both "compressed context" and a "retrieval entry point."

This is the classic pattern of context engineering: retrieve structure, not document chunks. Vector embeddings slice code as if it were prose; tree-sitter slices it as code — for questions like "what calls what," the latter's information density is an order of magnitude higher.

Comparison: repo map vs. RAG

Many coding agents use vector databases for code retrieval (embed → nearest neighbor). The repo map takes a different tack: it exploits the code's static structure (the definition/reference graph) rather than semantic similarity. The cost is that it can't answer semantic questions like "where is user login handled"; the payoff is zero hallucination (signatures come verbatim from source), zero infrastructure (no embedding model or vector store needed), and results that are deterministic and reproducible. The two can complement each other — but aider proves the pure-structure approach already goes a long way.

The Lint / Test Feedback Loop ​

After every AI edit, aider automatically runs two checks and feeds the error output back to the model for self-repair:

  • Auto lint: built-in linters for mainstream languages (also configurable per language or via --lint-cmd), run by default on every edited file (--no-auto-lint to turn off). The contract is plain: the linter prints errors to stdout/stderr and exits non-zero.
  • Tests: run manually with /test; --test-cmd plus --auto-test runs them automatically after every edit. When tests fail, aider automatically attempts a fix.

This "edit → verify → feed the verification result back to the model" loop was built into aider's early benchmarks — every task in the Exercism benchmark gets a second chance: if the unit tests fail after the first attempt, the error output (first 50 lines) is sent back to the model for repair. In the leaderboard's bar charts, the gap between "first attempt" and "final result" is the net payoff of the feedback loop.

Two engineering details of the feedback loop

  • Truncation: only the first 50 lines of error output are sent, so it can't blow up the context. Feedback information needs a context budget too.
  • Don't use a formatter as a linter: many formatting tools "return non-zero whenever they modify a file," and aider would mistake that for lint errors and ask the model to "fix" them. The official advice is to wrap such tools in a shell script (run twice; the second run's exit code is the true status). The interface contract of external tools (exit-code semantics) is part of the harness contract too.

Deep Git Integration: Version Control as the Agent's "Undo System" ​

Aider assumes you work in a git repository (if you don't, it offers to run git init for you), and then puts git to work at three levels:

  1. Auto commit: every AI edit is committed the moment it lands, with the commit message generated by a weak model (--weak-model) from the diff and the conversation history, following Conventional Commits by default.
  2. Dirty-file protection: before editing a file that already has uncommitted changes, aider commits your changes on their own — guaranteeing that "your edits" and "the AI's edits" stay clearly separated in git history, so an AI botch can always be rolled back losslessly.
  3. In-session operations: /undo reverts the last AI change, /diff shows everything changed since the last round, plus /commit and /git for native commands.

There is also an attribution mechanism: commits authored by aider append (aider) after the author/committer name (configurable as an aider: prefix or a Co-authored-by trailer).

Seen from a harness perspective, the essence of this set of designs is grounding the agent's exploratory behavior on a reliable rollback foundation. Agents are bound to make mistakes; the harness's job is not to eliminate errors but to make them cheap, visible, and reversible. Git is a ready-made, thirty-years-proven "time machine for the filesystem" — aider didn't reinvent it; it just had the discipline to drop a checkpoint at every step.

Trade-offs ​

Aider's minimalism isn't free, and its boundaries are just as clear:

  • The human stays in the loop for file selection. The repo map solves "understanding," but "what to change" depends mostly on the user's /add. In the repo map article, the author explicitly lists "automatically finding all files that need modification" as unfinished future work. It's a trade that leaves planning with the human — autonomy given up, predictability gained.
  • Strongest on single-file/few-file tasks; strained by large cross-repo overhauls. No subagent parallelism, no task decomposition — a session is one linear stream of edits.
  • The cost cliff of the whole format. Weak models can only rewrite whole files; on large files, latency and token cost collapse the experience — a harness's capability ceiling is locked to the strongest format it can afford.
  • The scope of the benchmark. Aider's benchmarks center on small-to-medium exercises and refactoring tasks; "88% on Exercism" does not mean "equally excellent on your 500k-line monorepo." Its value lies in relative comparison (format A vs. format B under the same benchmark), not in absolute prediction.

What to Learn from Aider ​

  1. The output format is the harness's first battlefield. The same GPT-4 Turbo scores 20 with SEARCH/REPLACE and 61 with udiff. All the effort you invest in prompts, retrieval, and planning can be canceled out by one bad output protocol. When designing an output format, ask yourself: how often has the model seen it in its training data? Does it force the model to compute line numbers? What does a parse failure cost?
  2. Be lenient on the consuming end. Aider's flexible diff applier cuts the edit error rate by 9x — never assume the model will strictly honor the contract; the harness should absorb the jitter in the parsing layer.
  3. Use the code's structure as context, not the code's text. The tree-sitter + graph-ranking repo map proves that a ~1k-token symbol map can replace tens of thousands of tokens of stuffed-in full text.
  4. Every decision must be measurable. Nearly every post on the aider blog is a controlled experiment: baselines, ablations (disable flexible parsing, drop the high-level diff instruction), quantified conclusions. A finding like "emotional-appeal prompts are worse" can only come from a benchmark.
  5. Ride existing infrastructure instead of inventing new. Git for rollback, linter exit codes as the verification contract, tree-sitter for structural extraction — every layer of aider stands on the shoulders of mature tools.

Further Reading ​

References ​