Opus 5.5 at Medium or High Effort? I Tested It on My Own Repos

Why bother

Opus 5.5 shipped today, and the first thing I wanted to settle was which reasoning effort to run by default. There are five levels: low, medium, high, xhigh and max. My Claude Code config had xhigh globally and medium pinned for Opus 5.5, so the real question was medium versus high.

Most of what's out there is benchmark scores, and benchmarks aren't quite the work I hand to agents every day. I barely write code myself anymore, and I don't review diffs line by line. So I cared about two things:

1. On my repos, with the kind of tasks I actually delegate, how different is the completion rate?
2. Whatever the difference is, is it worth the extra time and tokens?

Easiest way to find out was to measure it.

The setup

The idea is replaying history, SWE-bench style, except every task comes from my own repos:

1. Pick tasks. Agents scanned recent history across 11 repos — TypeScript side products, Rust desktop and credential tools, a Python publishing pipeline, a Node agent runtime. A commit qualified if it changed both source and tests (tests in separate files), was a feature or bug fix rather than a refactor, and touched tens to a few hundred lines. 22 made the cut.
2. Verify tasks. A second agent independently reproduced each one: with the commit's tests overlaid on the parent, they must fail; on the commit itself, they must pass, twice, to rule out flakes. It also checked fairness — every name, signature and error string the hidden tests rely on has to be stated in the task description, without leaking the implementation. A few task prompts were reworded before they passed.
3. Run. Each task ran twice at medium and twice at high, 88 runs total, each in a fresh git worktree at the parent commit with dependencies preinstalled and an identical prompt. The medium and high runs of a pair started together so both saw the same machine load.
4. Score. After each run, the original commit's tests were overlaid on the agent's changes. That's the hard signal. The full test suite also ran to catch regressions.
5. Blind judging. Each pair of solutions went to a judge as A and B, with the task, the original commit's diff, both diffs, test results and each agent's closing report — but not which effort produced which. Every pair was judged twice with A and B swapped to cancel position bias. 88 judgments in all.

The replay pipeline: pick, verify, paired runs, hidden-test scoring, blind judging

Time and tokens came straight from each agent's transcript, not from self-reports. I also scanned every command the agents ran for peeking past HEAD or into the main checkout. None did.

One mishap worth mentioning: halfway through, my local login token got replaced, the old one was revoked on the spot, and 39 agents died with a 401 within 12 seconds. Workflow subagents don't retry on 401. The few that survived were sitting inside long cargo builds and made no API calls during that window. Any pair with a casualty was thrown out whole and rerun in rebuilt worktrees so both sides kept equal conditions. Every number below comes from 44 complete pairs.

Results

The headline: completion was identical. All 88 runs passed the hidden tests, and none introduced new failures. The full-suite failures I did see were either already failing on the original commit or load-induced timeouts that passed on an idle rerun.

The differences show up in time, tokens and quality:

The judging row is the interesting one. Most pairs where judges preferred high came down to real bugs the tests couldn't catch:

1. A dashboard bar chart where medium scaled bar heights against the uncompressed series while drawing compressed bars, so the 90-day and all-time charts overflowed past 100%. Rendering wasn't tested, so both went green.
2. A decay routine where medium updated the stored state but not the in-memory copy, so a later full write-back erased the marker it had just written.
3. A migration that should refuse to write when a check fails. Medium ran the check after two files had already been modified, leaving the disk half-changed on refusal. High checked first both times.
4. Two rasterizers for a geometric primitive. Both high runs made the float path, the integer path and the SVG describe the same region, holding the required area tolerance at every angle; medium's broke at small sizes and at 45 degrees.
5. Two cases where medium deviated from semantics the task spelled out: a condition evaluated in the wrong order, and a change that tripped a repo check limiting hard-coded CJK strings.

Of medium's 7 wins, only one was a real bug: while editing a logout request, high stripped the client-IP headers the platform requires, so logout would fail in production. The rest came down to high changing things nobody asked for, or forgetting docs the change made stale.

The per-dimension judge scores have the same shape. High scored a bit higher on completeness (8.90 vs 8.72) and verification (8.89 vs 8.62); medium scored higher on staying in scope (8.76 vs 8.28). High likes to do a little extra — mostly more tests and doc updates, occasionally something it shouldn't have touched.

To be clear, 16 to 7 isn't statistically significant: a sign test gives p ≈ 0.09, and the confidence interval on judge preference crosses zero. This is a lean, not a proof. What actually moved me were those specific bugs — all-green tests, nobody reading the diff, broken in production.

Cross-checking with Artificial Analysis

Artificial Analysis published its Opus 5.5 evaluation today across all five effort levels. Here are the Intelligence Index scores and the output tokens spent running the full suite:

Their numbers line up closely with mine. High versus medium costs them 1.39× the tokens and roughly 1.33× the money; I measured 1.35× and 1.29×. Medium to high buys 3 index points for about 1.3× the cost, the best step on the curve. After that, high to xhigh buys 2 points for nearly double, and xhigh to max another 2 points for 2.6× the tokens.

The effort ladder: medium to high is the cheap step; each step above roughly doubles cost

They also note that max, xhigh, high and medium all sit on the intelligence-versus-cost-per-task Pareto frontier: at each of those price points no other model is both cheaper and stronger. Prices dropped from Opus 5 too, and cache reads got much cheaper, which suits agents that re-read the same context constantly.

Both datasets say the same thing: medium is already strong, high buys a real if modest improvement for about a third more, and beyond that returns shrink fast.

What I picked

Opus 5.5 at high is now my default, because of those bugs. There is no human review step in my workflow; tests and the model's own judgment are the only safety net, and the class of error tests miss only gets caught if the model thinks one step further. A third more time and tokens is worth that to me.

One gotcha while changing the config. I set effortLevel to high in settings.json, opened a new session, and it was still running xhigh. The culprit was ultracode: with "ultracode": true, every session is forced to xhigh and the effort setting is ignored, both interactively and with claude -p. You can verify via the effort field recorded in the session transcript, or the "with high effort" line on the startup screen. So if you want high as the default, ultracode has to be off; turn it on per session when you want it. The --effort flag and the CLAUDE_CODE_EFFORT_LEVEL environment variable both override the setting for one-off switches.

Which effort for which user

This comes from my setup; a different workflow may land elsewhere. Roughly:

1. Agents run end to end with no human review (like me): high. Completion is the same, but high catches more of what tests can't — exactly the errors you fear most when nobody reads the diff.
2. Pair programming where you read the diffs: medium is enough. A third faster, stays in scope better, and the issues it misses are the kind you'll likely spot in review. The saved waiting time is very noticeable.
3. Tight on quota (Pro, regularly hitting subscription limits, or heavy API usage): default to medium and pass --effort high for hard tasks. A 35% token gap adds up over a month.
4. Hard problems (algorithms, geometry, concurrency, data integrity, anything touching money) or multi-hour autonomous runs: xhigh is worth it here. AA shows two more points, at roughly double the cost, so spend it where it counts. I'd reserve max for problems where you're genuinely stuck.
5. Bulk mechanical work (grep, renames, formatting, simple subagent chores): low or medium. Thinking harder doesn't help.

A few limits. First, my task descriptions were fairly detailed because the hidden tests needed matching interface names, so completion may have hit a ceiling set by the tasks themselves; vaguer requests could widen the gap. Second, the judge was also Opus 5.5 at high. Blind A/B with swapped order reduces that bias but can't eliminate it. Third, I tested workflow subagents rather than interactive sessions, and left xhigh out. 44 pairs is enough to see a direction, not to settle it.

When the next model ships, the whole pipeline — pick, verify, paired runs, blind judging — is reusable. Swap the model name and run it again.