Agent Harness Handbook

Appearance

Case Study: Devin

At a glance An anatomy of the harness behind Devin, the "fully autonomous software engineer" — the long-horizon architecture built from a shell/browser/editor/planner combination, the launch demos and the debunking controversy of 2024, the first-hand lessons on context engineering and model selection published on Cognition's engineering blog, and the boldest bet yet on the autonomy-vs-controllability spectrum of agent productization.

Case Study: Devin ​

On March 12, 2024, Cognition launched Devin — not under the banner of "yet another AI coding assistant," but as "the first AI software engineer." That narrative choice was itself a declaration of harness position: Devin did not cast itself as a tool that helps humans write code, but as a colleague who takes a task, completes it independently, and ships the result. Your relationship with it is handing off work and reviewing deliverables — not pairing and autocomplete.

Of all the coding agent case studies, Devin is the one that pushed autonomy to its extreme. That is also what makes it the best research specimen: why its demos went viral and then got debunked, what harness lessons its team later published on their engineering blog, and how the product found its balance between "fully autonomous" and "controllable." Taken together, these three threads map the exact boundary conditions of long-horizon agent design.

Timeline at a Glance ​

  • March 2024: Launch blog post and a series of demo videos (Introducing Devin), claiming to resolve 13.86% of SWE-bench issues end-to-end without assistance — far beyond the previous best of 1.96% — plus a demo of "Devin taking real freelance jobs on Upwork." The company simultaneously announced a $21 million Series A (led by Founders Fund).
  • April 2024: The YouTube channel Internet of Bugs published a frame-by-frame debunking video, and the demo controversy erupted (see below).
  • December 10, 2024: Devin became generally available (official announcement): from $500/month with 250 ACUs (Agent Compute Units) included, unlimited seats, and entry points spanning Slack, an IDE plugin, and an API.
  • April 2025: Devin 2.0 shipped with a $20-entry pay-as-you-go tier (TechCrunch coverage), pivoting from "expensive experiment" to mass-market pricing.
  • July 2025: After OpenAI's acquisition of Windsurf collapsed and Google hired away its CEO and research lead, Cognition stepped in and acquired the remaining Windsurf entity (TechCrunch coverage), and has since kept its own SWE model line going (e.g., SWE-1.5).
  • 2025–2026: Cognition's engineering blog entered a highly productive stretch, and the harness lessons it published (the context engineering and model routing covered below) became first-hand source material for the industry.

Harness Architecture: Give the Agent a Whole Computer ​

The launch post describes the system itself in only a few sentences, but every one of them is a harness decision:

"We've equipped Devin with common developer tools including the shell, code editor, and browser within a sandboxed compute environment—everything a human would need to do their work."

Unpacked, the essence of this architecture is: don't give the model a chat box — give it a computer. Four key components:

  • Shell: installing dependencies, running tests, executing arbitrary commands — the universal interface for touching the environment;
  • Code editor: reading and writing code in structured form, rather than spitting diffs into a chat;
  • Browser: reading docs, looking things up, verifying the pages it deploys — turning "learning an unfamiliar technology" into a runtime capability instead of training-time memory;
  • Planner: what the launch post calls "long-term reasoning and planning," supporting "complex engineering tasks requiring thousands of decisions," and able at every step to "recall relevant context, learn over time, and correct mistakes."
text
┌──────────────── Devin cloud sandbox (one isolated VM per task) ────────────────┐
│                                                                                │
│   ┌─────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────────────┐            │
│   │  Shell  │  │   Editor    │  │   Browser   │  │    Planner     │            │
│   │ deps /  │  │ read &      │  │ read docs   │  │ long-horizon   │            │
│   │ tests   │  │ write code  │  │ verify pages│  │ plan & context │            │
│   └────┬────┘  └──────┬──────┘  └──────┬──────┘  └────────┬─────┘            │
│        └──────────────┴────────┬───────┴──────────────────┘                    │
│                                ▼                                              │
│                          ┌────────────┐                                       │
│                          │    LLM     │  ← the model is just the operator     │
│                          └────────────┘      of this "computer"                │
└───────────────────────────────┬────────────────────────────────────────────────┘
                                │ deliverables: PR / deployment results / reports
        Entry points: Slack / Web / IDE / API ──► User role: assign work + review

This design is nearly isomorphic to the architecture of OpenHands — hardly a coincidence, since OpenHands began life as OpenDevin, the community's open-source clone of Devin. The two share one conviction: the toolset for long-horizon software tasks must cover the full working surface of a human engineer. An agent that only generates code inside a chat box won't get far. The difference is that OpenHands turned this architecture into a self-hostable, auditable open-source platform, while Devin productized the sandbox, the scheduling, and the entry points as a closed-source offering.

The contrast with Claude Code makes Devin's orientation even clearer: Claude Code lives in your local terminal — synchronous, present, interruptible at any moment — the "pair programmer" model. Devin lives in a cloud VM — asynchronous, remote, delivering via pull requests — the "outsourced colleague" model. The same shell + editor + browser combination, differing only in where it is deployed and how humans interface with it, grew into two completely different products.

2024: The Demo, the Virality, and the Debunking ​

Devin's launch was one of the most successful product reveals in agent history — and the one carrying the harshest lessons.

What Story the Demo Told ​

The launch post presented seven demos: learning from a blog post how to use ControlNet to generate images with hidden messages; building and deploying a Game of Life site from scratch to Netlify; locating and fixing bugs autonomously in an open-source repo; setting up an LLM fine-tuning environment given nothing but a GitHub repo link; resolving a real sympy issue from SWE-bench — and the most explosive item of all: "We even got Devin to take on real jobs on Upwork, and it completed them."

The Debunking: Three Problems Exposed by Frame-by-Frame Analysis ​

In April 2024, Carl Brown, an engineer with three decades of experience, published a video on his YouTube channel Internet of Bugs, Debunking Devin: "First AI Software Engineer" Upwork lie exposed!, re-examining the Upwork demo frame by frame for about half an hour. What he found:

  1. The task was swapped. The original Upwork order called for "deployment instructions on how to get this model running," but in the demo Devin was fed only the first sentence of the order description — what it actually completed was a different, more flattering programming task.
  2. It fixed a bug it had created itself. In the demo's most impressive segment, "discovering and fixing an error in the codebase," the file being fixed does not exist in the original repo — it was a file Devin itself had created along the way. Part of the debugging prowess on display was Devin cleaning up after itself.
  3. The efficiency narrative doesn't hold. The video's accounting showed that a human engineer would need far less time to do the same work than Devin actually took.

The analysis then spread widely on Hacker News and Reddit (e.g., TLDR's summary); Cognition never issued a substantive point-by-point rebuttal.

The Footnote on the Benchmark Number ​

The 13.86% on SWE-bench deserves its footnote read too: the launch post itself notes that the evaluation ran on a randomly sampled 25% subset, and that Devin was unassisted while the comparison systems were assisted (told which files to change). The number itself may not be fabricated, but the circulating version — "7x beyond the previous best" — dropped every qualifier.

What This Controversy Really Means for Harness Researchers

The debunking's firepower did not come from "Devin is weak." It came from the fact that outside observers had no way to check what was happening inside the harness. Closed-source harness + curated demos + benchmark numbers with footnotes — that combination was simply the industry norm at the time. Two years on, the evaluation community has answered the problem with a consensus: "no harness disclosure, no comparing scores" (see Model vs. Harness). The Devin controversy is the most famous paving stone on the road to that consensus.

Cognition's Engineering Blog: First-Hand Harness Material ​

What the Devin team contributed to the industry durably was not the demo videos but the engineering blog posts published from 2025 onward. These posts are a rare case of a top agent company writing down its harness design decisions.

"Don't Build Multi-Agents": Two Principles of Context Engineering ​

On June 12, 2025, Cognition's Walden Yan published Don't Build Multi-Agents — and the timing is worth savoring: Anthropic's famous multi-agent research system post went up on June 13. Two leading agent companies published opposite positions within a day of each other.

The post's thesis, stated up front: the reliability problem of long-horizon agents is, at bottom, a context engineering problem — "this is actually the first priority of engineers building AI agents." From that, two principles:

Principle 1: Share context — and share full agent traces, not single messages. The classic "main agent decomposes the task → subagents execute in parallel → results get merged" architecture is fragile because subtask descriptions inevitably lose the original context in transit. The post uses a famous example: ask the system to clone Flappy Bird, and subagent 1's idea of "game background" comes out looking like Super Mario, while the bird subagent 2 builds looks nothing like a game asset — every step is locally "reasonable," but assembled, it's all misunderstanding.

Principle 2: Actions carry implicit decisions, and conflicting decisions produce bad outcomes. Even if you copy the full task to every subagent, each one's actions still rest on assumptions the others can't see: the bird and the background that subagents 1 and 2 produce clash stylistically, because neither can see what the other is doing.

The post's conclusion is radical: by default, rule out any architecture that violates these two principles. The preferred design is a single-threaded, linear agent (context stays naturally continuous); when a task threatens to blow through the context window, use a specially trained compression model to distill the history into key decisions and events — rather than slicing the task among multiple agents whose contexts can't see each other.

Where This Principle Sits in This Site's Framework

Yan's "share full traces, not single messages" is the extreme-position version of the core subagent design decision — "what gets returned to the main agent" — in Subagents; his emphasis on compression models is the engineered-compression route within Context Engineering. The value of this post is that it isn't academic theorizing: it's the summary of lessons from a long-horizon product like Devin learning the hard way.

What deserves emphasis is how misleading the title is: what Cognition opposes is not "multiple executors" but multi-agent architectures that shred the context. The proof: a year later they shipped a "two-agent" system of their own —

Devin Fusion: Model Selection and the Multi-Model Harness ​

Devin Fusion, published June 29, 2026, is the most candid piece in the harness engineering literature on model routing. The problem is framed pragmatically: you can't run every task on the most expensive model, but the existing routing options "look good on benchmarks and write code you wouldn't merge."

Fusion's answer is two techniques:

Sidekick mode. Run two agents in parallel: the main agent on a frontier model, the sidekick on a cheap one — both independent agents with full toolsets, each maintaining a persistent, cacheable context. The key interaction discipline: the main agent should keep its hands off as much as possible and make only the decisions that genuinely matter — setting the plan, resolving ambiguity, final review — while delegating and monitoring by default. The post explicitly contrasts this with "having one model consult another via a tool" (Cognition's own "Smart Friend" experiment and Anthropic's "Advisor" tool): that approach smashes the KV cache on every consultation, which gets expensive. In sidekick mode the two contexts each cache continuously — that is the structural reason it's cheap.

Dynamic mid-session routing. Instead of picking one model at task start and living with it, a lightweight classifier decides during execution when to upgrade or downgrade the model. The most elegant engineering touch: model switches are scheduled at context compaction points — compaction triggers a cache miss anyway, so switching models is effectively "free."

The results: on its FrontierCode benchmark, Fusion holds frontier-level performance at roughly 35% lower cost (later figures: up to 60%); in internal trials, 88% of merged PRs were driven end-to-end by the automatic router.

text
┌──────────────────── Devin Fusion's two-agent structure ────────────────────────┐
│                                                                                │
│   ┌────────────────────┐  delegate / monitor / collect  ┌────────────────────┐  │
│   │    Main agent      │ ─────────────────────────────►  │      Sidekick      │  │
│   │  (frontier model)  │ ◄─────────────────────────────  │    (cheap model)   │  │
│   │ plan, clarify,     │       results & status          │ hands-on execution │  │
│   └─────────┬──────────┘                                 └─────────┬──────────┘  │
│             │  its own persistent context, cached independently    │             │
│             ▼                                                      ▼             │
│     context compaction point ──► switch models — the cache miss was due anyway  │
└─────────────────────────────────────────────────────────────────────────────────┘

Read Fusion side by side with "Don't Build Multi-Agents" and Cognition's actual position comes into focus: multiple agents are fine, as long as context stays continuous and decision authority stays centralized. Sidekick honors those two principles exactly — the main agent keeps every key decision and the full picture, while the sidekick is a strictly delegated execution arm. That's not a flip-flop; it's the principles carried through to the level of model selection.

Autonomy vs. Controllability: Devin's Product Trade-Offs ​

Put all the case studies on one spectrum and Devin's position is immediately obvious:

ProductWhere it runsHuman's roleAutonomyControllability mechanisms
DevinCloud sandbox, asynchronousHand off work, review PRsHighest: hours unattendedSandbox isolation, PR review gate, Slack course-correction
Claude CodeLocal terminal, synchronousPresent, pairing, interruptible anytimeHigh: autonomous loop with a human in the loopPermission approvals, plan mode, per-tool confirmation
OpenHandsSelf-hostable sandboxConfigurableHigh (architecturally isomorphic to Devin)Open source and auditable, customizable policies
CursorInside the IDEThe driverLow-to-medium: mostly suggestions/editsManual confirmation of every diff

Devin's trade-offs boil down to three sentences:

1. Replace process approval with a deliverable gate. Claude Code has you press y/n before every action; Devin has you review one PR at the end. The former distributes control throughout the process; the latter compresses it into the point of delivery. The cost is obvious: the price of drifting off course mid-way is entirely sunk, so Devin must lean harder than any peer on its planner, its progress reports, and mid-flight course-correction in Slack — the launch post's promise to "report progress in real time and accept feedback" is not a decorative feature; it's a structural necessity of this architecture.

2. Trade sandbox isolation for freedom of action. Giving an agent a whole computer means it can install any dependency, visit any website, run any script — an unacceptable risk on your local terminal, a controlled experiment in a disposable cloud VM. For the core trade-off discussed in Permissions, Safety & Human-in-the-Loop, Devin's answer is: "physically quarantine the blast radius, then delegate as much as possible inside the quarantine."

3. The autonomy narrative inflated — and then the debt came due. The 2024 demo controversy, and the mediocre task-completion rates in third-party evaluations after GA (The Register's early-2025 story was literally titled "'First AI software engineer' is bad at its job"), were the price of maxing out autonomy as the selling point. What's interesting is Cognition's subsequent pivot: the engineering blog increasingly stresses internally verifiable engineering metrics (merged-PR ratio, cost per task) rather than viral demos — itself a lesson learned from the controversy.

Three Takeaways from Devin

  1. A long-horizon agent's toolset must cover the full human working surface (shell + editor + browser); miss one surface and it will fail systematically on the corresponding tasks;
  2. "Multi-agent or single agent" is the fake question; "is the context continuous, is decision authority centralized" is the real one;
  3. The more autonomous the product, the more you must build verifiability (traces, deliverables, internal metrics) as a core feature — otherwise the first debunking will be enough to destroy the narrative.

Further Reading ​

References ​