Many Agents, One Plan: Consistent Quality from Non-Deterministic AI

🎲 Same Prompt, Different Result Link to heading

Give a coding agent the same task twice and you get two different results. Same model, same repo, same words. Different code.

In the IDE post I argued that we trusted compilers because they’re deterministic. The same input gives the same output. Agents don’t have that property, and better models won’t simply hand it to us.

So I stopped treating the variance as noise to suppress. I treat it as something to spend, and I spend it where it’s cheap. I’ve used this pattern in different ways over time, and it keeps showing me the same thing: several independent agents plus one adjudicator give more consistent results than one agent working alone.

💸 Where Variance Is Cheap, and Where It Isn’t Link to heading

A plan is a page. I can read one in a few minutes. Three plans take maybe fifteen. If they differ, the differences are easy to see side by side.

Code is the opposite. Every diff has to be reviewed, and review is already the bottleneck. I made that case in Review Early. Three different implementations of the same feature aren’t three options. They’re three reviews, and then I still have to pick one.

That gives me a simple rule: use the variance while it’s a page. Shut it down before it’s a diff.

In practice it looks like this:

For the hard early work, design and planning, I start several agents in parallel and give each one the same task.
A main agent compares the plans, picks one, and strengthens it with what the others caught.
Where good plans disagree, the decision comes to me.
Then one agent builds it inside tight constraints: interfaces and tests reviewed before any code is written.

Wide at the top, narrow at the bottom. The box at the bottom doesn’t make the builder deterministic. It makes the outcomes that matter checkable, which is the part I actually need.

🔭 What Several Plans Tell You That One Can’t Link to heading

One plan tells you what one run thought. Several independent plans tell you three more things:

Where they agree. That part is probably solid.
Where they differ. That’s a real decision. A single plan usually makes it silently, and you never find out there was a choice.
What only one noticed. A risk, an edge case, a dependency. That’s a miss you’d otherwise have shipped.

The second point is the one I value most. A single plan reads as if every choice were obvious. Three plans that disagree show you which choices weren’t. Silent decisions are one of the first things I look for in an AI-generated PR, and a disagreement between plans drags one into the open before any code exists.

🧷 Independence Is the Whole Trick Link to heading

The planners have to be independent. Separate contexts. No peeking at each other’s plans. No shared draft.

What they do share is the brief: the goal, the acceptance criteria, the constraints, and a fixed plan template with a length limit. The brief is the one thing that has to be good. A vague brief gets you three plans that are vague in different ways. The template makes the plans comparable and keeps them from padding.

An agent shown a hypothesis tends to agree with it. Hand planner two the output of planner one and you don’t get a second opinion. You get an edit.

Even real independence has limits. In 1986, Knight and Leveson published a study of 27 versions of one program, each written independently from the same spec. The versions failed together far more often than independence would predict. Separate authors, shared blind spots.

Agents from the same model on the same prompt correlate too. I’ve watched same-prompt runs land on the same idea again and again, sometimes as near twins. They differ less in the big idea and more in execution and in what they notice. So treat agreement between plans as evidence, not proof, and give the thing only one plan noticed more weight than its vote count suggests.

That gives you a choice to make up front:

To catch misses, give every planner the same prompt and let the variance in what they notice do the work.
To see real alternatives, give each planner a lens: “the simplest thing that works,” “assume the requirements will move,” “assume the downstream system is slow and flaky.” Same task, different pressure.

Know which one you’re doing. Same-prompt fan-out won’t hand you three different designs.

⚖️ Pick One Plan, Don’t Blend Them Link to heading

If you’ve used several agents to review code, adjudicating plans looks like the same job. It isn’t.

For review, I want recall. Collect every finding from every reviewer, then verify each one. Merging is easy because findings are independent. Fixing one doesn’t change another.

Design isn’t like that. A plan is a set of choices that depend on each other. The data shape implies the API, and the API implies the tests. Fred Brooks called this conceptual integrity in The Mythical Man-Month: a system should look like one mind designed it, even when many hands build it.

Blending plans breaks that. Take the storage approach from plan A and the error handling from plan C, and you’ve got a design nobody actually thought through end to end.

So my position is: pick one plan as the spine. Then graft in only the pieces from the others that are additions, not alternatives. A risk one planner noticed. A missing test. An edge case. If a piece from another plan would change the design rather than add to it, it isn’t a graft. It’s a disagreement, and it goes to the human.

This is also where fan-out parts ways with self-consistency. Wang and colleagues showed that sampling several reasoning paths and keeping the most common answer beats a single pass on reasoning benchmarks. That works when there’s one right answer to vote on. A design rarely has one. So I don’t vote. I use the samples to see the space, then pick one coherent plan.

🧑‍⚖️ The Judge Has Biases Too Link to heading

The main agent picking the plan is itself an LLM, and LLM judges have known biases. Zheng and colleagues found position bias (favouring an answer because of where it appears), verbosity bias (favouring the longer answer) and possibly a preference for their own answers.

All three show up in plan adjudication if you let them. So I make the judging fair:

Neutral labels. Plans come in as A, B, C. No hint of which agent or which run wrote what.
Shuffle the order. If the verdict flips when the order flips, it wasn’t a verdict.
Criteria first. The judge gets what a good plan must do, drawn from the task and its acceptance criteria, before it reads any plan.
Compare, then choose. The judge lists the agreements, the differences and the one-off catches before it picks. That forces it to read every plan properly, and the list is useful on its own.
Length isn’t quality. The longest plan isn’t the most thorough one. Ask what each plan decides, not how much it says.
The judge doesn’t compete. Ideally the main agent writes no plan of its own. If it does, its plan is a candidate like the others, labelled and shuffled.

And the judge doesn’t get the last word on real disagreements.

🙋 Disagreements Are Decisions for a Human Link to heading

When two good plans disagree, that’s not a tie for the judge to break. It’s a decision someone has to own.

Which one wins usually comes down to things that aren’t in the prompt: what we expect to change next year, how much operational complexity the team can carry, which tradeoff we can live with. That’s my call, not the agent’s.

This is the best thing fan-out gives me. A single planner makes those calls silently. Several planners surface them, and the judge’s job becomes writing a short list: “Plans A and C store this differently. Here’s the tradeoff. Which do you want?”

Then, once I’ve decided, the variance stops. One agent builds the chosen plan, inside reviewed interfaces and locked tests, the same box I described in Review Early.

🧪 What My Runs Showed Link to heading

Here’s one of my tests. It shows the two things this approach rests on: one agent working alone varies from run to run, and several agents with independent review narrow that.

I ran a small experiment on a visual reskin of a personal app. I compared two setups: one agent doing the whole task alone, and several agents, with one agent owning the result and others reviewing it independently. Same task, same starting commit, same model, several unattended runs of each, with my answers to any questions fixed in advance. Reviewers graded each result blind on neutral copies: a correctness review and a structure review, each out of 10.

One agent alone varied. Its three runs scored 8, 8.5 and 9 on correctness. Its tests scored 6, 6 and 7.
It also converged. All three runs picked the same visual concept, and two were near twins.
Several agents were steadier. Their runs scored 8.5 or 9 on correctness, almost always 8.5. Their tests scored 7 or 8.
Early reviews earned their place. Reviewing the plan and the tests before any code caught real problems, including tests that could never fail.
Several agents noticed more. One of those runs raised a performance question, how much more the new look costs to draw, that none of the one-agent runs noticed.

More agents isn’t the point, though. Several agents doing the same task independently, with one owner for the result, is a very different shape from many agents each doing a slice with no owner at all.

🎯 Consistent, Not Always Best Link to heading

Here’s the one thing I’ll hedge on. Fan-out gets you more consistent results. It doesn’t always get you the best possible result.

One agent working alone has a wide spread. Now and then it produces something sharp and inventive. Just as often it produces something you’d throw away: a confident plan built on a wrong assumption, a missed requirement, a design that falls apart at the first edge case. And you can’t tell in advance which run you’re going to get.

Fan-out cuts off the bottom of that spread without giving up much at the top. If one planner comes back with a great plan, the judge will most likely pick it. Instead of building the first plan you happened to get, you get several chances at a good one and a deliberate choice between them.

What it can’t promise is the best plan possible. You get the best of the plans you drew. If none of them found the great idea, adjudication won’t invent it.

For work I’m going to ship, that’s an easy trade. I’ll take a reliably good result over one that’s occasionally great and just as often terrible.

💰 When It’s Worth It, and When It Isn’t Link to heading

It costs more. Several planners instead of one, plus an adjudication step.

It costs less than it sounds. In my several-agent runs, about three quarters of the cost was the main agent’s own long conversation. The separate reviews were cheap. A planner in a fresh context, writing one page, is a short conversation, and so is a judge reading a few pages.

My rule of thumb follows the triage from Review Early.

Worth it:

Design work where a wrong early choice is expensive to undo.
Persisted data, security, or a contract other people depend on.
An unfamiliar area, where I don’t trust my own instinct to spot the miss.
Anything with several decisions that are mine.

Not worth it:

Small, easy-to-undo changes.
Work with one obvious approach. Three plans will just agree, and I’ll have paid for agreement.
Low-risk work like my reskin. One agent working alone was good enough there.

If the plan fits in a paragraph, one planner is fine.

✅ My Take Link to heading

Agents aren’t deterministic, and I don’t expect them to become deterministic soon. Compilers earned trust by giving the same output every time. Agents have to earn it some other way.

My way is to spend the variance where it’s cheap and informative, on a page of plan, and shut it down where it’s expensive, on code. Independent planners show me where the solid ground is, where the real decisions are, and what I’d otherwise have missed. One plan stays the spine. Real disagreements come to me. Then one agent builds inside a box I’ve already reviewed.

Diverge on the plan. Converge on the code.

A process can be consistent even when its parts aren’t. That’s not new either. It’s how good teams have always worked with people.

✅ Checklist Link to heading

Is this hard design work, or risky by the Review Early triage? If not, use one planner.
Run several planners on the same task, in separate contexts, with no shared drafts.
Same prompt to catch misses. Different lenses to see alternatives.
Treat agreement as likely, not certain. Same model, same blind spots.
Read the differences as decisions, and the one-off findings as possible misses.
Judge with neutral labels, shuffled order and criteria written first. Compare before choosing. Don’t reward length.
Pick one plan as the spine. Graft in additions only, never alternatives.
Send real disagreements to a human.
One agent implements, inside reviewed interfaces and locked tests.
For code review, do the opposite: collect every finding, then verify.

Sources

Related Posts