Does Code Mode make a web research agent cheaper? We tested it on BrowseComp
We gave the same agent the same Exa search tools twice, once called directly and once through Code Mode, and asked it ten of the hardest questions in OpenAI's web research benchmark.
The short answer
Running an agent's web search through Code Mode cut the tokens it read by more than half on hard research questions, at about the same accuracy. On ten questions from BrowseComp, OpenAI's benchmark of questions that take many searches to answer, the agent used 11.7 million input tokens through Code Mode and 25.5 million with the same tools called directly, and the run cost 44 percent less. We measured it on the Agent Voyager Project (AVP), our open agent benchmark.
| BrowseComp, first ten questions | Exa tools called directly | Exa tools through Code Mode |
|---|---|---|
| Correct answers | 8 of 10 | 7 of 10 |
| Input tokens | 25.5 million | 11.7 million |
| Search text returned to the model | 8.4 million characters | 2.4 million characters |
| Total cost | $0.50 | $0.28 |
| Full run on AVP | See every step of the direct run | See every step of the Code Mode run |
Every question, answer, and step from both runs is public on AVP, in the direct run and the Code Mode run.
What we tested
Claude Code, Anthropic's coding agent, ran on Claude Haiku 5.5 and searched the web with Exa. In one run it called Exa's search and page tools one at a time, and every result landed in its conversation. In the other, the same tools sat behind Code Mode, which lets the agent write a short TypeScript program that runs its searches and returns only what it keeps. Claude Opus 4.8 graded each answer with BrowseComp's own grading instructions.
The saving came from the hard questions. On the question that took the most work, the direct agent read 16.8 million tokens and the Code Mode agent read 2.0 million. On questions that one or two searches settle, Code Mode cost a little more, since writing and running a program takes a turn of its own. Cost fell less than tokens because much of each request is text the model provider has already cached and bills at a lower rate.
Code Mode only works when tools describe their output
Our first attempt saved nothing. Exa's search server returns plain text and does not declare what its results look like, so the agent's programs had no fields to pick from and handed back the raw results whole. We served the same Exa search through a small wrapper that returns typed results, used it in both runs, and gave the agent those types in its instructions. In a smaller test of nine simpler questions, half of the agent's programs returned raw results while it had to discover the types itself, and nine of ten returned only the facts once the types were in its instructions. On BrowseComp, the typed programs cut the search text reaching the model by 71 percent.
What this test does not show
Ten questions, each run once, cannot separate 7 correct answers from 8. One of the Code Mode misses was a question it was still working on when the run hit our 15 minute limit, and the direct agent had missed that same question in an earlier trial. What the data supports is comparable accuracy at roughly half the tokens. A full BrowseComp run would be needed to claim more.
One part of a Context Layer Platform
Code Mode is one capability of Port of Context, a Context Layer Platform (CLP). Turning it on is a single setting for each server you connect, and an administrator can give any connected tool an output type, which this test shows is what makes Code Mode pay off. The same platform gives agents a map of your business, search across your documents, and access to connected systems under each person's own credentials. Our CLP benchmark measures what those pieces do together on customer support tasks.