# On a well-built MCP server, Haiku matched Opus We ran the same 13 tasks against two MCP servers that share one issue-tracker backend. On the well-built server, Claude Haiku 5.5 solved 99.0% of its runs and Claude Opus 5.5 solved 100%, one run apart out of 104. On the second server, which copies flaws from real APIs, Opus still solved 99.0% and Haiku fell to 64.4%. If you're choosing a cheaper model for an agent, you want to know whether its misses come from the model or from the server it's talking to. We measured that with [M3](https://github.com/sineframe/m3), our open-source tool for [testing MCP servers](/mcp-testing) and the agents that use them. ## The results: a good server closes most of the gap On 12 October 2026 we ran six models at default settings: three Claude models through Claude Code 2.1.296 and three OpenAI models through Codex 0.162.1. Each model ran 13 tasks on each of the two servers, 8 trials per task. That comes to 104 runs per cell and 1,248 runs in all, 1,242 of them graded after 6 harness errors. | Model | Good server | Naive server | |---|---|---| | Opus 5.5 (Claude Code) | works 104/104 | works 103/104 | | Sonnet 5.5 (Claude Code) | works 104/104 | breaks 71/104 | | Haiku 5.5 (Claude Code) | works 103/104 | breaks 67/104 | | GPT-6 Astra (Codex) | works 100/104 | works 103/104 | | GPT-6.1 Sol (Codex) | works 101/102 | works 95/100 | | GPT-6 Luna (Codex) | breaks 89/104 | breaks 52/104 | The good server has 9 tools with flat arguments and enums. It accepts users and labels by name, filters on the server, returns a `total_count` and a cursor, and writes errors that say how to fix the call. The naive server exposes about 70 generated REST-style tools, including near-duplicates and a deprecated one. It paginates like GitHub (`page` and `per_page`, 20 per page, a bare array with no total and no "more" flag) and changes status through Jira-style workflow transitions. Priority is an integer from 0 to 4 that only a separate tool explains, users are referenced by id, and a bad call gets `400 Bad Request`. | Model | Good server | Naive server | |---|---|---| | Opus 5.5 | 100% [96.4, 100] | 99.0% [94.8, 99.8] | | Sonnet 5.5 | 100% [96.4, 100] | 68.3% [58.8, 76.4] | | Haiku 5.5 | 99.0% [94.8, 99.8] | 64.4% [54.9, 73.0] | | GPT-6 Astra | 96.2% [90.5, 98.5] | 99.0% [94.8, 99.8] | | GPT-6.1 Sol | 99.0% [94.7, 99.8] | 95.0% [88.8, 97.8] | | GPT-6 Luna | 85.6% [77.6, 91.1] | 50.0% [40.6, 59.4] | These are success rates with 95% Wilson intervals. To compare two models we bootstrapped over tasks with 2,000 resamples, and we call them equivalent when the 90% interval of the difference stays within 10 points. Haiku on the good server and Opus on the naive server both scored 99.0%, a difference of 0 with a 90% interval of [-1.9, +1.9]. Haiku and Luna have the same list price, and Haiku beat Luna by 13.5 points on the good server (95% interval [3, 28]) and by 14.4 on the naive one ([2, 29]). Even on the good server, Luna (85.6%) trails Astra on the naive server (99.0%) by 13.5 points ([-26, -3]). Put another way, the good server closes 97% of Haiku's gap to Opus and 77% of Luna's gap to Astra. Sonnet and Sol also cost the same, and they are equivalent on the good server. On the naive server Sol beats Sonnet by 27 points ([9, 48]): Sonnet went from 100% to 68% and Sol from 99% to 95%. We don't know why. Codex's code mode, where a loop over pages is easy to write, would explain it, except that Luna runs in code mode too and still fails. ## Where small models break Most of the damage is in four tasks that make the model page through a list and join the rows: H01 (a multi-filter list across two projects), H02 (which team has the most urgent issues among 520), H03 (a per-item triage rule) and H04 (a long chain of lookups and reassignments). The other nine tasks mostly hold up. On the naive server Sonnet solved 66 of their 72 runs, Haiku 63 and Luna 49, and the other three models 69 or more. | Anthropic | Good server | Naive server | |---|---|---| | Opus 5.5 | works 32/32 | works 32/32 | | Sonnet 5.5 | works 32/32 | breaks 5/32 | | Haiku 5.5 | works 32/32 | breaks 4/32 | | OpenAI | Good server | Naive server | |---|---|---| | GPT-6 Astra | works 32/32 | works 31/32 | | GPT-6.1 Sol | works 32/32 | works 26/31 | | GPT-6 Luna | breaks 22/32 | breaks 3/32 | Most of these failures look like success from the outside. The naive server had 129 failed runs, and 113 of them (88%) ended with `STATUS: DONE` and a wrong or partial answer. Only 13 said they couldn't finish. In a typical trace the model reads a page or two, sees no total and no "more" flag, and reports what it has as the complete list. On H01, Luna made 5 list calls, when one of the two projects alone takes 4 pages at the largest page size. ## Which flaw does the damage We switched one of the five design dimensions (descriptions, tool catalog, argument schema, errors, pagination) from good to naive at a time, ran each version 5 times per task, and looked at the mid and small models of both vendors on H01 to H04. | Server | Haiku 5.5 | Sonnet 5.5 | |---|---|---| | Good | works 32/32 | works 32/32 | | Only bad descriptions | works 20/20 | works 20/20 | | Only the bloated tool catalog | works 20/20 | works 20/20 | | Only Jira-style schema | works 18/20 | works 20/20 | | Only bare errors | works 17/20 | works 20/20 | | Only silent pagination | breaks 13/20 | works 17/19 | | All five (naive) | breaks 4/32 | breaks 5/32 | | Server | GPT-6.1 Sol | GPT-6 Luna | |---|---|---| | Good | works 32/32 | breaks 22/32 | | Only bad descriptions | works 19/20 | breaks 13/20 | | Only the bloated tool catalog | works 20/20 | breaks 13/20 | | Only Jira-style schema | works 20/20 | breaks 15/20 | | Only bare errors | works 20/20 | breaks 12/20 | | Only silent pagination | works 20/20 | breaks 9/20 | | All five (naive) | works 26/31 | breaks 3/32 | On its own, no flaw takes Haiku below 65% on these four tasks. All five together take it to 13%, where multiplying the single-flaw rates would predict about 50%, so the flaws make each other worse. Luna follows the same pattern from a lower start: 69% on the good server, 45% with only pagination naive, and 9% with all five, against about 35% predicted. Sol shrugged off every single flaw. Pagination does the most harm alone for both small models. Across all 13 tasks, Haiku drops from 99% to 86% when pagination is the only naive part of the server, and Luna from 86% to 65%. ## What this means for your MCP server - Say how many results there are and whether more exist. Pagination without a `total_count` or a `has_more` flag was the worst single flaw in our ablations, and it usually failed as a partial list reported as complete. - Filter on the server. When the model has to page through 520 issues and filter them itself, each page is another chance to stop early. - Accept names where people use names. Taking users and labels by name removed a lookup chain from several tasks. - Use plain status and priority fields where you can. Workflow transitions and unexplained integers add calls the model can get wrong. - Write errors that say how to fix the call. `400 Bad Request` gave the model nothing to retry with. - Keep the catalog small. With only the 70-tool catalog swapped in, Haiku and Sonnet still solved nearly everything, but each paid 20 to 30% more per success. ## What it costs Cost per successful task is an estimate. For Claude Code we used the cost it reports, and for Codex we multiplied token counts by list price. Haiku and Luna list at $0.10 in and $0.50 out per million tokens, Sonnet and Sol at $2 and $10, Opus at $4 and $20, and Astra at $10 and $50. | Model | Good server | Naive server | |---|---|---| | Haiku 5.5 | $0.0027 | $0.0110 | | GPT-6 Luna | $0.0036 | $0.0119 | | Sonnet 5.5 | $0.037 | $0.144 | | GPT-6.1 Sol | $0.061 | $0.130 | | Opus 5.5 | $0.081 | $0.275 | | GPT-6 Astra | $0.307 | $0.574 | Per successful task, Haiku on the good server costs about a hundredth of what Opus costs on the naive server, at the same success rate. The naive server raised every model's cost per success, by 3 to 4 times for the Claude models, and that includes the flagships whose success rate didn't move. It was slower too. Median latency went from 18 to 32 seconds for Opus, 28 to 41 for Astra and 38 to 55 for Sol. ## How M3 runs each cell Trackly, the tracker behind both servers, is a fake issue tracker with 6 projects, 40 users and 520 issues. Its backend is deterministic, and a random nonce per trial changes the answers, so a model can't remember them from an earlier run. Each cell is an ordinary pytest test that M3 runs against the real agent binary at a pinned version. Here is the whole test for H01 ("open, unassigned regression bugs in PAY or WEB in the Oct-W2 or Oct-W3 cycles, minus `wontfix-candidate`"): ```python pytestmark = pytest.mark.m3(suite_name="model-tiers") def test_h01(agent, variant) -> None: assert run_task(agent, Path(__file__).resolve().parent, variant) == "success" ``` M3 records every tool call on the wire. The benchmark replays those calls against the same backend, then checks the tracker's final state and the `STATUS:` line in the answer. There is no LLM judge. A scripted agent that sees only tool responses solves every task on every server version, so each failure we count is the model's, and we read a sample of the failed runs by hand. The whole run is one command, with each harness's models separated by commas and M3 options before `--`: ```sh MT_VARIANTS=good,naive m3 test --runtime=managed \ --harness claude_code@2.1.296=claude-opus-5-5,claude-sonnet-5-5,claude-haiku-5-5 \ --harness codex@0.162.1=gpt-6-astra,gpt-6.1-sol,gpt-6-luna \ --trials 8 --execution-timeout 600 -n 8 -- tasks _baseline ``` If you log in to Claude Code with a subscription, add `--credential-env claude_code:CLAUDE_CODE_OAUTH_TOKEN=CLAUDE_CODE_OAUTH_TOKEN`. The tasks, both servers and the grader are in the repository under [benchmarks/model-tiers](https://github.com/sineframe/m3/tree/main/benchmarks/model-tiers). Everything ran in one domain, an issue tracker. Each result is a model and harness pair at default settings, and a different harness could move it. The costs are estimates from reported costs and token counts. Codex shows OpenAI models their tools as TypeScript declarations rather than JSON Schema, which may account for part of the gap between vendors. The benchmark README lists the fixes we made during the pilot, and the records of the pilot and of every run are kept with the benchmark. ## What surprised us Opus caught a mistake of ours. Our rebalance task (H05) couldn't be done with the data we had generated. In three of its four pilot runs, Opus said so, explained why and left the tracker alone. We fixed the data and reran it. Haiku read "bugs" in H01 as the `bug` label and said so in its answer. Our data has both an issue type and a label called `bug`, so that was a fair reading. We changed the grader to accept either one, as long as an answer sticks to it. Sonnet surprised us most. In our smaller pilot it lost 17 points on the naive server, and in the full run it lost 32 and landed next to Haiku, while Sol, at the same price, lost 4. Opus didn't notice the naive server and still paid 3.4 times as much per success on it. ## Test your own MCP server on small models You can run the same comparison against your own server. Install the CLI and scaffold a project: ```sh uv tool install sf-m3-cli m3 init # a skipped starter test and an .env.example m3 setup ``` Write the task you care about as a pytest test, then run it across several models in one invocation, so a change on the provider's side hits every column at once: ```sh m3 test --runtime=managed \ --harness claude_code@2.1.296=claude-opus-5-5,claude-haiku-5-5 \ --harness codex@0.162.1=gpt-6-astra,gpt-6-luna \ --trials 8 -- tests/ ``` M3 is open source under Apache 2.0 and still in alpha: [github.com/sineframe/m3](https://github.com/sineframe/m3). Start with the [documentation](/docs/). If a flaw in your server breaks a model we didn't try, open an issue there.