I Tested GPT-6 Sol and Luna for My Codex Workflow

Why bother

Here's roughly how I use Codex now. One main agent acts as the coordinator: it figures out what I want, breaks the work down, and does final acceptance. Under it, subagents do the actual jobs, like digging through code, implementing an agreed plan, or reviewing one specific risk. Each subagent can be given its own model and reasoning effort. Effort controls how hard the model thinks at each step, from low and medium through high, xhigh and max.

Until now I took the lazy route and put every role on GPT-6 Astra at high. The coordinator was Astra, the code searcher was Astra, the agent changing one line of config was Astra. It saved me thinking about routing, but usage burned very fast. A lot of that work never needed Astra in the first place: listing every call site of a function, or editing a few files to follow an existing pattern, doesn't need the most expensive model.

This GPT-6 release added Sol and Luna to Codex, both lighter and cheaper than Astra. OpenAI's docs suggest Sol/medium and Luna/high as starting points for subagents. What I wanted was simple: faster turnaround, fewer unnecessary tokens, no drop in completion quality. Routing some work to smaller models is one way to cut unnecessary expensive inference, but a cheaper price per token doesn't by itself mean fewer tokens or faster end-to-end work. That still depends on output volume, retries and extra coordinator work. Nobody could promise me the third goal either, so I had to test it.

Before any of that, there was a small smoke test. Luna/medium and Sol/high each extended the same shell script, and both passed the same 20 predefined checks. That task was too small to say anything about harder work, which is why the three rounds below exist.

One thing up front: there are no time, token or cost numbers in this post. Codex's subagent tool accepts the model and effort you request, but it doesn't independently attest which model actually served the request, and it doesn't give reliable per-run billing. So the one thing I could measure was whether the same task, given to different models and efforts, came back correct. How much money or time this saves is something I'll only learn from using it for a while, and that estimate has to include the coordinator's review and rework.

The setup

Seven routes: Astra/high as the control, and Sol and Luna each at medium, high and xhigh. Every round followed the same pattern:

1. Each route got a brand-new agent with no conversation history, an explicit model and effort, and its own git worktree. A worktree is a separate working directory for the same repo, so the seven agents couldn't step on each other, and the whole thing gets deleted afterward.
2. All seven got the identical task packet: what to do, which files they could change, which they couldn't touch, and what report to hand in.
3. The grading tests were written and frozen before dispatch, and the agents never saw them. These are usually called hidden tests. Before submitting, agents could write their own tests, debug and retry as much as they liked. After submission, nothing from the hidden tests was fed back and there was no repair round. Whatever they submitted was what got graded.
4. An agent saying "all tests pass" counted for nothing. Grades came from the frozen tests, the repo's existing regression tests, and actual values the evaluation code executed.

The Codex coordinator ran the evaluation, read the diffs and did the grading. I didn't read the code line by line. My part was setting direction and pushing back: when the tests didn't seem tough enough I had it make them stronger and retest on real PRs, and I specifically asked whether it had looked at the actual output values.

Each route ran once per round. This is not a statistical reliability benchmark. Every cell in the tables below is a single sample: it tells me what went wrong on a given kind of task under a given configuration, not how often it goes wrong.

The three rounds answer three different questions.

Round 1: three tasks in a big monorepo

The first round used the monorepo of one of my own products, about seven thousand tracked files. The three tasks were related, all around one streaming chat endpoint:

1. Trace. Explain how a streamed reply recovers after a disconnect, from server writes and auth through parsing, replay, and the completion signal. Graded against 12 predefined facts.
2. Implement. Add cancellation to a function that parses an SSE stream (the server pushes events down one at a time). The spec was detailed: every line-ending variant, partial lines, Chinese characters and emoji split across two chunks, a clean stop no matter when the caller cancels, and released locks and listeners. 29 frozen tests, plus 42 existing regression tests and a type check.
3. Review. Review an unmerged patch with 3 planted bugs, each requiring a concrete triggering sequence and effect, not "this might be a problem".

On the frozen tests, all seven routes passed. The 42 regressions and the type check passed too, and nobody reported a false bug in the review. Results that uniform suggested the tasks were too easy.

Then the coordinator read the submitted source, one by one, and found an edge they handled differently. When a stream ends, the buffer may still hold one last event without a trailing newline, and the parser flushes it out to the caller. If the caller cancels after receiving that event, its next request for data should get an AbortError. The task spelled this out, including the final-unterminated-line path by name, but the 29 frozen tests happened not to cover it.

So after reading the code, the coordinator added one case and ran it against all seven routes, still with no chance to fix anything. Only Astra/high and Sol/high passed. The other five all missed the same spot: they checked for cancellation before handing over that last event, but not after control came back, so the next call just ended normally.

Two caveats. First, this case was added after the code had been read, not frozen beforehand, so it has its own column and isn't folded into the 29. Second, Sol/xhigh failing while Sol/high passed doesn't mean high beats xhigh. One sample can't separate an effort effect from luck. What it does show is that turning effort up didn't automatically close the gap.

Luna/high had two more problems. On the trace task it said the repo had no dedicated test for the stream consumer logic. There was one: a file with 26 tests. It also added new tests into an existing test file when the task said new test files only. The added tests didn't weaken any assertion, but it was still out of scope. "I didn't find it" and "it doesn't exist" are different claims, and that turned into one of the delegation rules later.

Round 2: cross-project tasks in three repos

All three round-one tasks lived in one subsystem, which is narrow. Round two moved to three repos from work: a frontend platform, a backend, and a database schema repo. Four tasks, some spanning repos:

1. Value clamping in a frontend chart, 39 checks, including keeping true values separate from plotted ones.
2. Passing an async task's cancellation reason through intact in the backend, 27 checks.
3. A date-filter contract between the TypeScript frontend and the Python backend, compatible across old and new version combinations, 64 checks.
4. A new typed field in the backend with a matching schema migration, 39 checks, including scope checks that unrelated source stayed untouched.

This round came back clean. Every route passed every functional check, all 204 existing regression tests passed on every route, and the scoped type checks passed. The only lost points were Luna/high and Luna/xhigh at 38/39 on task four, and that point was a scope check: both edited existing test files. The SQL itself was fine.

There's a confound on that scope result that needs disclosing. The individual task descriptions mentioned existing test files, while the main packet explicitly said new test files only and took precedence. So what these two results show is that in these runs, those agents didn't resolve a conflict between instructions. They don't show that Luna generally ignores scope, or that medium is better than high. Round three used consistent test-ownership wording.

I'd specifically asked whether actual output values were being checked, not just pass/fail, and the coordinator recorded and inspected every route's outputs. A true chart value of -90 was drawn at -10, a true 45 drawn at 30, with the 35 delta preserved. A decimal of 123456789.123456789 made the trip from TypeScript to Python and back without losing a digit. Database readback kept an old row's null distinct from a new row's zero. These values came from my evaluation code executing the submissions, not from anything an agent wrote in its report.

Two things went wrong that had nothing to do with the models. Under heavy machine load, one test hit a 5-second default timeout; with the same source and assertions, rerunning everything under a uniform 30-second budget passed. And the evaluation's output counter missed one ok line that an async diagnostic had split in half, even though the test framework's own summary already said everything passed. The counter got fixed. Neither counts against a model.

After round two I was a lot more comfortable giving implementation to Sol. With clear requirements and clear boundaries, all seven routes got the work right. The obvious limit is that these tasks were purpose-built, with very explicit requirements.

Round 3: blind replays of historical PRs

This is the round I cared about most. After round two I still didn't think the tests were tough enough, since purpose-built tasks have unusually clean requirements, so I asked for a retest on real historical PRs. Three PRs from work were picked, grouped into two tasks, and each repo was pinned at the commit before the fix. Agents only got the issue as it was described at the time. The tests added by the original fix were kept outside the agents' reach and brought in only after submission. No looking at later history, no looking at the original PRs. Each route had 32 minutes of active working time.

Task one was frontend. Navigation should preserve app state, but when a different account signs in, the previous account's profile and questionnaire data must not carry over. At the same time, an anonymous user's pre-signin draft and a same-account page refresh both have to survive. The original fix came with 9 tests. The evaluation first checked that those tests could actually tell fixes apart: the pre-fix code passes 5, applying only half of the original fix passes just 4 (persisting state makes the cross-account leak easier to see), and the full original fix passes all 9.

Task two spanned the database and the backend. A restricted database login could read sessions, but refreshing the session's last-activity time failed with a permission error, caused by an audit trigger. The fix needed to give the trigger bounded authority while keeping audit history unreadable and unwritable for normal logins. Then the backend repo's migration snapshot, the snapshot's integrity pin, and its integration-test runner all had to be updated. That runner has to actually create a database, apply the new migration, and run the backend's original tests. So this was a real cross-repo delivery, and it had to run end to end.

Everyone passed the frontend task. On the database task, six routes got the SQL right, but only three also got the real backend caller path passing in their own integration runner: Astra/high, Sol/high and Sol/xhigh.

What's most worth telling is how the incomplete ones were handed in.

Sol/medium wrote its own test for the new migration. That test called a SQL function directly, and it was green. The backend's real code doesn't call it that way; it goes through its repository layer to look up the session. When the coordinator ran the backend's original test against the database Sol/medium's own runner built, the lookup failed with a permission error. It had missed the grants that path needs.

Luna/xhigh's test was green too, but it updated the base table directly, under a special policy that only matched one hardcoded test wallet. The real backend code goes through a compatibility view and never touches the base table. So its green check proved that the path it built for itself worked, not that the bug was fixed. With the backend's original test, the update through the compatibility view returned nothing.

A self-test shortcut goes green while the real caller path through the compatibility view is blocked

The other two Luna submissions were more candid. Luna/high said in its own report that it was submitting a partial result: the compatibility view its runner set up was incomplete, and the update returned nothing. Luna/medium's migration self-check rejected a membership that PostgreSQL itself keeps in place, so the whole migration aborted. Its report said the work was incomplete, though it described that remaining membership slightly wrong. Honestly, a submission that tells me it isn't done is far easier to deal with than one that looks green and isn't fixed.

The grader also made a mistake this round. Its runner adapter was written around how the original PR did things: it assumed one fixed environment-variable name, and it assumed the backend tests run in a single phase. Astra/high and Sol/xhigh sensibly opened a separate connection for the restricted login, and the adapter misread that as the owner connection and failed them. Luna/high split the backend tests into two phases, which is also a valid layout, and the adapter didn't handle that either.

That was the grader's fault, not a problem with how they implemented it. So a companion adapter was added. It first confirms the connection the agent provided really is a non-privileged restricted login, then plugs it into the original test fixture. For split phases, it collects results across phases and requires all three original tests to pass. Not a character of any agent's SQL, source, or the original assertions changed. The same adapter was applied to all seven routes; the original fix still passes, and a control with repaired snapshots but the old runner, which never runs the new version's test phase, still fails even though it exits with status 0. The original misjudged results are kept too. This was compatibility handling added after submission, not part of the frozen grader, which is why it's called out separately.

A few smaller issues had nothing to do with code correctness. Sol/medium and Luna/medium wrote their reports to the repo root instead of the specified location. Luna/xhigh went a step further and wrote its report into the main checkout, outside its own worktree. These were files in the wrong place, not source regressions, so they were logged separately and kept out of the table.

Both xhigh submissions did one extra thing on the frontend task: they changed locally stored profile data to a per-account format and dropped old data at the next sign-in. Sol/xhigh disclosed this in its report. All 9 original tests still pass, so it isn't wrong, but it's a product decision the original fix never made, and someone has to review whether it's the right one. Both xhigh submissions I observed did this; two samples can't show that higher effort always does, but the extra still needs someone to look at it.

For the record, here's what was actually inspected, again by the coordinator: every report, the full frontend diffs, the database migrations, the backend runner diffs, every failure output, and the recorded values. For the runner misses, it also read the agents' self-test source to see exactly which path they'd gone down instead. It did not audit every line of every self-test or every file.

What I settled on

After three rounds, Codex routing looks like this:

If an existing script or tool can already do a purely mechanical job, just run the tool instead of spawning an agent.

xhigh and max are neither defaults nor an automatic "didn't work, bump it up a level" escalation. They're for one hard problem you can name. Across the three rounds, xhigh didn't consistently beat high, and it brought extra decisions that needed review.

Beyond the table, a few delegation rules I think matter a bit more than the table itself:

1. Write the full task for every subagent. What result you want, which sources to read, which files it may change, what evidence to return, and when to stop and come back. When the agent has no history, dropping constraints to save tokens is a false economy.
2. Mixing models needs a no-history fork. In the native Codex subagent tool I used, a full-history fork inherits the parent's model and effort and doesn't accept a model override. To explicitly pick Sol or Luna, fork with no history or a bounded number of turns. That's this tool's current behavior, not necessarily every Codex interface. Role config files can also pin their own model and effort, so changing only the global default isn't enough. And agents that are already running won't switch models because you edited the config.
3. Don't have Astra redo what a cheaper model already did. If the coordinator re-investigates and rewrites everything it delegated, you paid twice. Whether a route actually saves anything depends on review, rework and coordination too, not just unit price.
4. Not finding something doesn't prove it's absent. A claim like "the repo has no X" should come with the scope that was searched. The test file Luna/high said didn't exist in round one was right there.
5. Acceptance has to run the real caller path. A model's own green tests only prove the path it built for itself works. Both unfixed submissions in round three had green self-tests.

Which setup fits you

This comes out of my own usage, and a different workflow could land somewhere else:

1. One session doing the whole job, few subagents: keep the main agent on Astra/high. Most of this post doesn't apply to you.
2. Lots of subagents, usage burning fast (like me): start from the table. Move code search, inventories and mechanical edits to Sol/medium and Luna, move implementation to Sol/high, and keep coordination and acceptance on Astra.
3. Anything touching permissions, money or database migrations: keep the judgment on Astra. Sol/high can implement a plan that's already been decided, but don't hand off both the design and the final acceptance.
4. Turning everything up to xhigh to be safe: I wouldn't. In these rounds it wasn't more reliably correct, and both xhigh submissions this time made changes nobody asked for, which you still have to review.

Limits

To close, the limits:

1. Each route ran once per round, and the tasks across the three rounds aren't a random sample of engineering work. None of this estimates failure rates or separates effort effects from run-to-run variance.
2. With no independent model attestation and no per-run billing, there's no measured speedup, token reduction or cost saving here. Moving to cheaper models is meant to cut usage; that's the expectation, and the real effect will only show over time.
3. The round-one EOF-abort case was added after reading code; the round-three runner adapter was compatibility handling added after submission; the round-two scope result has an instruction conflict mixed in. Each is flagged above.
4. Every task had fairly explicit requirements. Triage from vague reports, architecture design, multi-hour autonomous tasks and live security were not tested.
5. The three rounds used 8, 27 and 27 worktrees, all deleted afterward, and every repo's main checkout matched its state before testing. None of the business code written during the experiment was committed or deployed; the change that shipped was Codex's routing configuration.

If a new model comes out, or I want to move some kind of work down another level, I can rerun this same process. I just have to check each time that the grader is exercising the real path.