Momentic QA Bench: 1,396 Bugs, 10 Agent Runs — Mo Wins at $2.67 Each

AI coding agents now write a large share of the code that ships, and someone has to check their work. Momentic, the YC Winter 2024 startup behind the QA agent Mo, published QA Bench on November 3, 2026: a benchmark that hands ten agent configurations a running web application, a fixed clock, and one job: find the real bugs. The reference set holds 1,396 human-reviewed, labeled bugs across four widely used open-source apps. Momentic's own agent Mo found 269 of them per 60-minute pass, at 19.3% recall. The next best configuration, Codex with GPT-6.1 Sol, found 86.

This is both a benchmark release and a pricing study. Every configuration is scored not just on recall (bugs found) and precision (reports that are real), but on cost per confirmed bug. Mo's headline figure is $2.67 per bug, which is 35% less than Claude Code with Opus 5.5 at $4.09, but more than triple Codex's $0.80. And the biggest caveat is printed on the benchmark itself: Momentic built it, ran it, and graded it. Here is what the numbers say, what they cost, and where to stay skeptical.

What QA Bench actually measures

Most agent benchmarks test whether an agent can write code. QA Bench tests the opposite: given a working application, how many of its real, labeled defects can the agent find in a fixed amount of time, and how many of its reports turn out to be real? That matters because the defects that survive static analysis and unit tests are increasingly the kind you only see by using the product: a button that does nothing, a filter that returns the wrong rows, a settings page that throws a stack trace.

The corpus is four open-source web applications, each pinned to a specific upstream commit and split into product areas (a Gitea "issues" area, a Metabase "dashboards" area, and so on):

Ten configurations ran the suite: Mo itself; Codex with GPT-6.1 Sol and GPT-6.1 Terra, each paired with two browser integrations (Playwright MCP and agent-browser); Claude Code with Opus 5.5 and Sonnet 5.5, each with both browser integrations; and Devin in its native browser. Each run gives the agent one product area in a fresh E2B sandbox, the same brief and entry points, no access to the application source, the labels, or earlier reports, and either 30 or 60 minutes. An LLM judge, GPT-6.1 Sol at high reasoning effort, matches each submitted finding against the area's labels and a list of claims human reviewers already rejected. Every configuration ran each area three times per budget, except Devin, which ran once.

One fairness detail is worth noting: Mo had more to do than the baselines. Mo had to produce a reproduction video and a written report for every bug it found; the other agents only had to identify the bug. Against that, Momentic also discloses two exceptions to its own no-web-access rule: Codex had web search on, and the agent-browser runs for Codex and Claude Code had open web access in the agent's shell.

The results: recall is the headline, and Mo wins it

In QA, recall is the metric that matters most: a bug the agent misses ships to production. Mo's 19.3% recall at 60 minutes is the best in the table by a wide margin, and still leaves about four in five labeled bugs unfound on an average pass. The coding agents were not built for this task, and the numbers show it.

All figures are per-pass averages over three runs per area (one for Devin), from Momentic's own write-up. Mo's lead held in all four applications at both budgets: 26% of Gitea's labeled bugs, 23% of Saleor's, 22% of Twenty's, and 9% of Metabase's. Metabase was the hardest app for everyone. It holds a third of the corpus in four dense areas, and Codex Sol with Playwright MCP matched just 3% of it. At the area level, Mo's three-run average beat every other configuration in 19 of 20 areas; the exception was Gitea notifications, the smallest area at 16 labeled bugs, where Claude Code Opus edged it 3.0 to 2.3.

The precision table tells a different story. Codex with GPT-6.1 Sol and Playwright MCP was the most precise configuration at both budgets (87.5% at 30 minutes, 86.0% at 60). Mo was next at 30 minutes (85.4%) and third at 60 (84.2%, just behind Devin's 84.4%). In other words: Codex reports fewer bugs, but a slightly higher share of what it reports is real. Mo reports a lot more, and carries about 51 known-invalid claims per pass at 60 minutes, against Codex Sol's 14.

The pricing math: what a found bug costs

This is the part of QA Bench that reads like a price card. Cost per matched bug is total run spend divided by total matches, so it blends the agent's model bill with how good it is at the job:

Currency conversions use November 3, 2026 market rates (1 USD = 0.746 GBP / 1.396 CAD), verified at publication. The cheapest way to find a reference bug in this benchmark is Codex with GPT-6.1 Terra and agent-browser at about $0.27, though it finds far fewer bugs than Mo. Per pass, Mo is by far the most expensive configuration at about $720 for the 60-minute run, against $241 for Claude Code Opus and $69 for Codex Sol. Against Claude Code, Mo spends 3.0x as much for 4.6x the bugs; against Codex, 3.1x of the 10.4x spend gap is bug count, and the remaining 3.3x is higher cost per bug.

Two honest numbers to keep beside the headline: Mo's cost per bug barely moved between budgets ($2.54 at 30 minutes, $2.67 at 60) because its matches rose 52% in the second half hour, with 19 of 20 areas improving. Claude Code Opus also improved 55%, but its spend more than doubled, so its cost per bug rose 42%. And GPT-6.1 Terra actually found fewer bugs at 60 minutes than at 30 in both browser integrations, which Momentic reads as a sign that longer runs need an agent that keeps exploring. For context on the models involved, see our reviews of GPT-6.1 Terra and the 7 best AI coding agents of 2026.

The browser tool matters as much as the model

One of the most useful findings has nothing to do with Mo. Codex and Claude Code each ran with two browser integrations (Playwright MCP and the agent-browser CLI, both driving hosted Browserbase browsers), and the integration moved the needle as much as swapping the model. Playwright MCP found 10% to 58% more bugs than agent-browser for Codex and for Claude Code with Opus 5.5 at both budgets. Which tool was more precise and cheaper per bug depended on the model: for GPT-6.1 Terra, agent-browser was the cheaper route per bug; for GPT-6.1 Sol, it was 5% to 20% more expensive.

If you are wiring a coding agent to a browser today, that is the most actionable line in the whole report: check the harness before you blame the model.

The caveats, stated plainly

Momentic deserves credit for publishing a limitations section most vendors skip. The caveats:

• First-party evaluation. Momentic builds Mo, and designed, ran, and graded the benchmark. Its own agent tops the table. Treat the figures as the company's best case, not an independent result.
• Agent-drafted labels. The candidate bugs came from bug bashes by Mo, Claude Code, and Codex, three of the agents being graded. If none of them can find a bug, it has no way into the reference set, so the labels may share the agents' blind spots. The report also notes that bugs discovered by the scored runs and then approved expand the set, which favors agents that report more candidates.
• An LLM judges the matches. GPT-6.1 Sol decides whether a report describes a labeled bug. Judging errors feed straight into the scores.
• Uneven corpus. Eleven of the 20 areas are Gitea, one is Saleor, and Metabase holds a third of the bugs, so its low recall drags everyone's totals down.
• Thin statistics. Three runs per area with no error bars; Devin ran once per area. Area-level differences of a bug or two are noise.
• Equal time is not equal work. The time budget does not cap spend, and Mo spends far more per run than the baselines, plus the repro video and written report the baselines skip.

None of this makes the benchmark useless; it makes it what it is: a vendor's instrumented comparison, openly described. The dataset is public on Hugging Face and the harness and labels are on GitHub, so anyone can rerun it.

What Mo is, and why this benchmark exists now

Mo is Momentic's scriptless QA agent: you give it a URL or a mobile build, and a swarm of explorer, bug-bash, and reproducer agents plans test cases, runs them in parallel in hosted browsers, simulators, or emulators, and returns a report with reproductions, no test scripts to write or maintain. Momentic launched Mo on September 28, 2026, and its product page claims 117,010 bugs caught before deploy, with Notion, Superpower, Iris, Committee for Children, and Boundless among the companies testing it, per third-party launch coverage. Those are company figures, not independent results.

The timing is the tell. As coding agents flood teams with machine-written code, the QA layer becomes the bottleneck. Our own review of the open-source Bug0 QA agent last week covered the same pressure from the other side. Momentic's bet is that recall is the scarce resource: a cheap, precise agent that finds 6% of bugs is less useful than a pricier one that finds 19%, because the missed bugs ship. At $2.67 a bug against a human QA hour, the arithmetic is not absurd. It is just unproven outside Momentic's own lab. Cognition's free Devin security plugin covers the fix side of the same loop for Devin users.

Bottom line

Momentic says no benchmark existed for this task. QA Bench is its answer: real apps, real bugs, a clock, and a bill. Mo's recall lead is large, its cost-per-bug story is worth a look, and the browser-integration finding is useful regardless of whose agent you run. But every number in it was produced by the company selling the winner. If you are evaluating QA agents for your team, use QA Bench as a starting point, then run your own app through two of them and see which bugs come back. The labels are open; the excuses for not checking are gone.

Is Momentic's QA Bench an independent benchmark?

No. Momentic built Mo, and designed, ran, and graded QA Bench itself, as the report says on page one. The dataset, harness, and labels are public on GitHub and Hugging Face, so third parties can rerun and challenge the results, but as published, the figures are the vendor's own.

How much does it cost to find a bug with each agent?

At the 60-minute budget, measured cost per confirmed bug was $2.67 for Mo, $0.80 for Codex with GPT-6.1 Sol and Playwright MCP, $0.27 for Codex with GPT-6.1 Terra and agent-browser, $4.09 for Claude Code with Opus 5.5 and Playwright MCP, and $3.89 for Devin. These are benchmark-measured run costs, not product list prices.

Which agent found the most bugs?

Mo, by a wide margin: 269.3 labeled bugs per 60-minute pass at 19.3% recall, versus 86.3 for the best Codex configuration and 59.0 for the best Claude Code configuration. But 19.3% still means about four in five labeled bugs went unfound on an average pass.

Does a better model or a better browser tool matter more?

Both moved the needle. Playwright MCP found 10% to 58% more bugs than the agent-browser CLI for Codex and Claude Code with Opus 5.5 at both budgets. The practical takeaway: the browser harness you give a coding agent matters roughly as much as the model behind it.

What is Mo and is it free?

Mo is Momentic's scriptless QA agent for web, iOS, and Android apps, launched September 28, 2026. Momentic has not published list pricing in the QA Bench report; the $2.67 figure is a measured per-bug run cost, not a price you can buy. Check Momentic's own site for current plans before budgeting around it.

Sources

• Momentic, "QA Bench: Evaluating QA Agents on Real Web Applications," November 3, 2026 (official benchmark report): momentic.ai/blog/qa-bench
• Momentic QA Bench results summary (official): momentic.ai/qa-bench
• Momentic, Mo product page (official): momentic.ai/mo
• QA Bench dataset, Hugging Face (official): huggingface.co/datasets/momentic/qa-bench
• QA Bench harness and labels, GitHub (official): github.com/momentic-ai/qa-bench
• AIToolsRecap, "AI News November 3 2026: LongCat 2.5, QA Bench, GDPx," November 3, 2026: aitoolsrecap.com
• AIWeekly, "Momentic Launches Mo, Catches 269 of 1,396 QA Bench Bugs," November 2026: aiweekly.co
• Hacker News, "Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app" (discussion): hn.jakechampion.name
• GenZ NewZ, "Momentic Launches Mo, an AI Agent for Scriptless Software Testing," September 28, 2026: genznewz.com
• Let's Data Science, "Momentic Launches Mo for Scriptless Software Testing," September 2026: letsdatascience.com
• Currency conversions at November 3, 2026 market rates (1 USD = 0.746 GBP / 1.396 CAD), verified at publication.