Before a model can reason for a customer, it runs Umbodi's frozen tasks: auditing a page, writing a fix, refusing to invent a figure, ignoring instructions planted in a site, keeping the customer's language. Each task runs three times through Umbodi's own provider code and is scored by machine. The table is updated with each new run, and a model below the bar can still be chosen with your own key.
7 models · 4 meet the bar · the bar (runs · errors · after one repair): 90 · 0 · ≥95% · newest run 2026-10-14
Measured
| Model | Run | Runs | Errors | Valid, no repair | Correct after one repair | Median latency | Output tokens, all runs | Verdict |
|---|---|---|---|---|---|---|---|---|
| claude-haiku-4-5 | 2026-10-13 | 90 | 0 | 100% | 100% | 1.9 s | 31,242 | Passed |
| myt/glm5.2 | 2026-10-13 | 90 | 0 | 100% | 100% | 2.3 s | 68,314 | Passed |
| gemini/gemini-3.5-flash | 2026-10-14 | 90 | 0 | 100% | 100% | 6.4 s | 21,448 | Passed |
| openai/gpt-5-mini | 2026-10-14 | 90 | 0 | 100% | 100% | 6.8 s | 114,364 | Passed |
| myt/gemma-4-26b-a4b-it | 2026-10-12 | 39 | 1 | 97.4% | 97.4% | 4.5 s | 11,234 | Below the bar |
| myt/qwen3.6-35b-a3b | 2026-10-12 | 39 | 6 | 61.5% | 74.4% | 6.5 s | 63,383 | Below the bar |
| myt/deepseek-v4-flash | 2026-10-12 | 39 | 2 | 94.9% | 94.9% | 8.6 s | 42,339 | Below the bar |
Models with fewer than 90 runs were benched on an earlier 13-task set. Output tokens drive a model's cost; prices change, so they are not shown here.
Thirty tasks are frozen in the repository, each with the evidence, the prompt and a rubric a machine can check. A run passes when the answer is valid and meets its rubric; one repair is allowed and reported separately.
A model meets the bar with a full run of 90, no errors, and at least 95% correct after one repair. Customers see the same verdict next to every model they can choose in Umbodi.
Scores are machine checks, not a blind human review, which is still to do. Latency is wall-clock time from the machine that ran the bench to the provider.
Every model's newest result, with the bench file it came from, free to reuse with credit under CC BY 4.0: model-benchmarks.json
Whether an AI model can do Umbodi's reasoning reliably: thirty frozen tasks such as auditing a page, writing a fix, refusing to invent figures and ignoring planted instructions, three runs each, scored by machine.
Yes, with your own key; it is your choice. Umbodi marks it as below the bar, because some of its answers may be unusable, and never switches you to another provider without telling you.
Whenever a model is added or changes. Each run is committed with its date, and this page is rebuilt from those files, so a figure here is always the newest measured one.
Bring your own key for myt AI Cloud, OpenAI, Gemini, Mistral or Kimi, and see the verdict next to each model before you choose.