Test19 November 20266 min read

I tested the idea behind Claude Code's new Code Mode. The AI read 11x fewer tokens and stopped miscounting

Bar chart. Tokens the model read: 140,809 for one tool call per message, 12,788 for writing a script. Model turns: 122 vs 20. Fully correct answers: 5 of 9 vs 8 of 9.
All 9 runs added up. Orange: one tool call per message. Blue: the model writes a script that calls the tools.

When an AI coding agent works on your project, it uses tools: read a file, search, edit. Normally it asks for one tool, waits for the result, reads it, and then asks for the next one. Every step is a new turn, and every turn re-reads the whole conversation so far.

Yesterday the Claude Code team posted on X about an experimental feature called Code Mode (about 12,000 likes in a day). The idea: instead of asking for tools one by one, "Claude writes scripts that chain your tools together". One script can read 15 files, filter the results and send back only the part that matters. The Code Mode docs suggest it for tasks with many related tool calls.

That sounds good, but how much does it really save? And does the AI make more or fewer mistakes when it writes code instead of reading every result? I wanted numbers, so I built a small test on my Mac.

What I tested (and what I didn't)

To be clear first: I did not test Claude Code's own Code Mode. It only runs in an interactive terminal session, so I couldn't run it many times from a script and measure it fairly. Instead, I tested the idea behind it, with an open model running locally, so I could count every token.

The three tasks

  1. Broken links. "Read every Markdown file in docs/ and find the links that point to local files that do not exist." This is the example the Code Mode docs use. There were 6 broken links, one of them an image in a subfolder.
  2. Count TODOs. "Count the TODO comments in src/ and name the 3 files with the most." There were 48 TODOs in 14 files.
  3. Same edit in many files. "In every JSON file in config/, change maxAttempts from 3 to 5. Leave other values alone." 8 files needed the change; 2 had other values and had to stay the same.

Each task ran 3 times in each mode: once with no randomness (temperature 0), and twice with normal randomness (temperature 0.7). I checked the answers with code, not by eye. For the edit task, I checked the files on disk after the run.

The results

Task (3 runs each)Classic: rightClassic: turnsClassic: tokens readScript: rightScript: turnsScript: tokens read
Broken links2 of 35987,7902 of 3*87,410
Count TODOs0 of 397,4913 of 362,952
Same edit in 10 files3 of 35445,5283 of 362,426
Total5 of 9122140,8098 of 92012,788

* In one run, the script found all 6 broken links but wrote the paths in a different form than I asked for ("docs/auth-old.md" instead of "auth-old.md"). My checker counted it as wrong. A person would probably accept it.

"Tokens read" is the total size of everything the model had to read, added up over all turns. This is what most AI APIs charge you for as input.

1. The token gap grows with every step

Line chart of tokens read, added up per turn, for the broken links task. Classic mode climbs for 20 turns to 30,789 tokens. Script mode stops after 3 turns at 4,005 tokens.
Broken links task, first run. Classic mode reads one file per turn, and each turn re-reads everything before it.

In classic mode, the model read the 15 docs one at a time. That is 15 turns, plus a few more to list files and answer. Each new turn carries every file it has already read, so the cost does not grow in a straight line. It curves up.

In script mode, the model wrote one loop: read every file, pull out the links, check each one, print only the broken ones. The 15 files never entered the conversation at all. Only the 6 results did.

The edit task shows the same pattern even more clearly: 45,528 tokens vs 2,426, a 19x difference, with the same correct result in both modes.

2. Counting is where classic mode fails

This one surprised me. In classic mode, the model was smart: it used the search tool once and got back a list of all 48 TODO lines. Then it had to count them by reading. It got it wrong in all 3 runs:

RunTotal TODOspayments.tsorders.ts
Right answer4897
Classic, run 149107
Classic, run 250118
Classic, run 349107
Script, all 3 runs4897

Language models are bad at counting long lists by eye. Code is not. In script mode the model simply wrote a loop that counts lines, and the computer did the counting:

for file_path in list_files("src"):
    count = 0
    for line in read_file(file_path).splitlines():
        if "// TODO" in line:
            count += 1
    if count > 0:
        todo_counts[file_path] = count

This is the part of Code Mode I find most useful. It's not only cheaper. For any job that needs exact numbers, it moves the work from the model's guess to real code.

3. Scripts can fix themselves

The first script for the links task had a bug. Its pattern grabbed a "(" as part of every link, so it reported every link as broken. The model saw the strange output, explained its own mistake, fixed the pattern and ran it again. The second run was correct. That cost one extra turn, and it was still 7x cheaper than classic mode.

Classic mode had its own slip on the same task: in one run it read all 15 files, then searched only for links ending in ".md", and missed the broken image link.

4. Writing scripts costs more output

It's not all savings. Script mode wrote more output: 5,211 tokens vs 4,170, because a script is longer than a short JSON tool call. Output tokens usually cost more than input tokens. But here the input savings (128,000 fewer tokens) are far bigger than the extra 1,041 output tokens.

On time, script mode took 229 seconds for all 9 runs and classic mode took 646 seconds. My Mac was busy with other work during the test, so treat those times as rough.

When to use Code Mode, and when not to

Based on this test, I would turn it on for:

I would not expect much from it for:

One more thing to know before you try it: when the AI runs a script, many actions happen in one step. The Code Mode docs say your permission rules still check each tool call inside the script, but in auto mode the safety check reviews the script as a whole, not each call. If you have hard rules (for example "never push to main"), keep them as deny rules or hooks, as the docs suggest.

If you want to try the real thing, the docs say you need Claude Code v2.1.323 or later, a supported model, and an interactive terminal session started with CLAUDE_CODE_EXPERIMENTAL_CODE_MODE=1. It is experimental and may change.

Limits of this test

Sources

Written by Tabish Hassan, a software developer. Found a mistake? Email hello@tabishapps.com.