What Five AI Models Agreed On (And Why I Didn't Believe Them)

A few days ago I installed a Claude Code skill called llm-council. It's a nice adaptation of an Andrej Karpathy idea: instead of asking one AI a question and taking its word for it, you run the question past five advisors, have them peer-review each other anonymously, and let a chairman synthesize a verdict.

It worked. I liked it. And then I noticed the thing that bothered me.

Five advisors, one brain

The skill spawns its five advisors as Claude sub-agents. Same model, same weights, same session. The only thing that varies between them is the prompt — one is told to play the Contrarian, another the Expansionist, and so on.

Which means the disagreement is manufactured. The Contrarian objects because I told it to object, not because it sees something the others missed. And more importantly: whatever that model can't see, none of the five can see. You get one opinion wearing five costumes, and the peer-review round is a model grading four other outputs of itself.

Karpathy's original varied the model and held the prompt constant. That's the version where disagreement means something. So I set out to build that.

Five labs, one API bill of about two cents

The result is a skill called multi-council. It sends one identical, neutral question to five models from five different labs, with no assigned roles at all:

• GPT-5.6 (OpenAI) via the Codex CLI
• Gemini 3.1 Pro (Google) via the Antigravity CLI
• GLM-5.2 (Zhipu) via Claude Code pointed at Z.ai
• DeepSeek V4 Flash (DeepSeek) via OpenRouter
• Inkling Small (Thinking Machines) via OpenRouter

Three of those seats cost nothing extra, because Codex runs on a ChatGPT subscription, Antigravity runs on a Google account, and the GLM seat runs on a Z.ai coding plan I already pay $12.60 a month for. Only the two OpenRouter seats bill per token, and they come to roughly a penny each per run. Claude chairs the whole thing and deliberately never advises — keeping its priors out of the advisor pool is the entire point.

Deciding to vary the model instead of the prompt turned out to be the load-bearing design decision. If each advisor had a different model and a different role, then when two disagreed you'd have no way to tell whether that was GPT-versus-Gemini or contrarian-versus-expansionist. You'd be manufacturing conflict and then reporting it as genuine uncertainty.

The bugs were the interesting part

Orchestrating four different CLIs on Windows produced a better collection of failure modes than I expected. These are the ones worth passing on.

A flag that ate its own prompt

The Antigravity CLI takes the prompt as the value of its --print flag. So this:

agy --print --model gemini-3.1-pro-high "What is 17 times 23?"

silently sets the prompt to the literal string "--model", throws away my actual question, and returns a fluent, confident, well-organized answer about the --model flag. Exit code zero. No warning.

That one cost me four runs before I worked out what was happening, and it's the scariest bug in the collection, because nothing downstream can catch it. A well-written answer to the wrong question is indistinguishable from a correct answer unless you read it. The fix had to make the mistake impossible to construct rather than detectable after the fact — the prompt is now always the final element of the argument list, and a test asserts it.

Two of five seats were silently dead on Windows

Both codex and claude install as .CMD shims. Windows CreateProcess will helpfully append .exe to a bare command name, but it will not try .cmd. So both seats failed with "not found on PATH" while both CLIs were installed and authenticated and worked fine from a terminal.

Python's shutil.which() finds them, because it applies PATHEXT. CreateProcess doesn't. That asymmetry is the whole bug. The Antigravity seat survived purely by luck of being a real .exe.

My own config was sabotaging everything

A failed experiment with a local routing proxy had left an env block in my ~/.claude/settings.json pinning every Claude model alias to gpt-image-2 — an image model. That broke sub-agent dispatch and WebFetch across every session, and it made the GLM seat ask Z.ai for a model that doesn't exist.

The generalizable lesson, which took me embarrassingly long to find: settings.json's env block outranks the environment you pass to a spawned claude subprocess. If you shell out to Claude Code from a script, point CLAUDE_CONFIG_DIR at an isolated directory, or your own global config will quietly overrule you.

Prompts over 8KB broke two seats

Found on the first real production run, naturally. cmd.exe caps a command line near 8,191 characters, and any council question that includes a document blows straight past it. Both .CMD-shim seats now take their prompt on stdin instead of in the argument list.

Every one of these lived in the gap between "my mocks return what I assume" and "the binary does what it does." The test suite was green at 66 passing tests while two of five seats were dead.

Then the first real run proved my whole premise

I pointed it at Anthropic's new rules of context engineering for Claude 5 generation models, which argues that several established practices — explicit rules, few-shot examples, front-loading context, hand-curated memory — are now obsolete.

All five models came back saying essentially the same thing: directionally right, badly overstated, keep your guardrails, keep a curated CLAUDE.md. Five models. Five labs. Three countries. Near-total agreement.

And that agreement is the least trustworthy output of the entire run.

Look at what I asked. Five instruction-tuned assistants were asked whether developers should trust AI models more and constrain them less. "Keep the guardrails" is what alignment training produces regardless of the evidence. Their unanimity is equally consistent with the article being right and with it being wrong. It doesn't discriminate.

This is why the chairman leads with divergence and treats unanimity as a flag rather than as confidence. Correlated errors don't just survive synthesis, they get promoted by it, because agreement reads as certainty. A tool that reported "5 of 5 advisors agree, high confidence" would have laundered a shared prior into a fact.

The peer-review round earned its keep too. Four of the five advisors argued against a position the article doesn't actually hold — they read "trust the model's judgment" as "have no checks," and missed that the article proposes moving checks out of the system prompt and into verifier agents running rubrics. Not one advisor caught that. A reviewer did.

The tool also found a bug in itself

My favorite result. One advisor kept citing details about my local setup that weren't in the prompt. It turned out the GLM seat runs a real Claude Code instance, which reads my global CLAUDE.md — so four advisors were answering from the question, and one was answering from the question plus my workspace.

Not a peer. A privileged advisor, quietly better informed than the others, in a tool whose entire premise is that all five see the same thing. It's a correctness bug in the decorrelation guarantee rather than a crash, which is exactly the kind that survives a passing test suite.

Where it lives

It's on GitHub at michaelkpate/multi-council. Standard library Python only, no dependencies to install, and it ships three rosters — including an all-OpenRouter one that needs no subscriptions at all and runs about two or three cents per council.

The thing I'd tell anyone building something similar: the cheapest models are a trap. Diversity only helps if each source is individually competent. Five text opinions get read by a synthesizing model, and a fluent, confidently wrong advisor doesn't get averaged out the way a bad numeric prediction would. It gets quoted. Pay the extra tenth of a cent.