OpenRouter model compatibility and comparison

Key findings

• On these easy, realistic tasks almost every model is good enough. 12 of 16 models score 0.97 or more on the macro score over calibration plus hold (glm-5.3, minimax-m3.1-flash, claude-sonnet-5.5, and claude-opus-5.5 are the exceptions). Quality barely tracks price. The spread is cost (about $0.066 to $4.53 per 1000 tasks, about 70x) and failure plumbing, not raw capability.
• The cheap tier is a strong default for routine work. claude-haiku-5.5 ($0.21 per 1000 data-gathering tasks), glm-5.3-flash ($0.09), and deepseek-v4-flash-0731 ($0.12) match or beat $2 to $7 per 1000 premium models on extraction, summaries, tool picking, short posts, and the 3-step agent loop.
• Premium does not buy code review on this set. On the hard seeded-bug set (3 snippets, fix executed against tests) glm-5.3-flash and claude-haiku-5.5 fixed 3 of 3; minimax-m3.1-flash 2 of 3; claude-sonnet-5.5, deepseek-v4-flash-0731, and glm-5.3 1 of 3; gemini-3.8-flash, deepseek-v4-pro-0813, gpt-5.6-luna, and gpt-5.6-terra 0 of 3. Low confidence, n=3.
• claude-sonnet-5.5 and claude-opus-5.5 reject forced or required tool_choice here (HTTP 400 on 15 and 9 requests). Reasoning cannot be turned off on those endpoints. claude-opus-5.5 also returned content-filter refusals on harmless prompts (0 of 3 agent loops passed). claude-haiku-5.5 content-blocked a benign long filler prompt twice.
• Reasoning runaway is a cost trap. glm-5.3 (mandatory reasoning) hit the 2,500 token cap on 7.6% of calibration-plus-hold calls even at effort=low, cost $1.68 per 1000 tasks against $0.07 for glm-5.3-flash, and scored lower. The summary_models.csv truncation column is all sets (12.7% for this model), not calibration-plus-hold.
• On 402-body payment decisions, qwen3.8-max, grok-4.7, and gemini-3.8-flash made 0 unsafe pays and passed 100% of 46 calibration-plus-hold decisions, at $2.05 to $5.39 per 1000. glm-5.3-flash passed 97.8% for $0.114 per 1000. minimax-m3.1-flash is cheapest but paid when policy said skip or paid the wrong payee in 20 of 82 decisions. No live payments were sent; fixtures were synthetic or a read-only unpaid GET.
• Tool-picker A/B on the out-of-sample hold set (n = 6 task groups per arm). Arm B (model, catalog only) chose deepseek-v4-flash-0731 for all 6 task groups and scored a mean 88.1% hold pass, 8.6 points behind 96.7% for Arm A1 (fixed rule, cheapest capable: claude-haiku-5.5) and Arm A2 (fixed rule, highest index: grok-4.7). Under this page's rule that differences under about 10 points are not significant, that is a small edge within noise, not a clear win. Arm C (model + measured data) matched Arm A3 (fixed rule, measured data) on all 6 task groups (93.7%). Small samples; not a general ranking of pickers.
• An LLM judge (moonshotai/kimi-k3.1-pro, reasoning off) is not usable for constraint compliance here: it scored all 16 models 4.78 to 5.00 of 5 and rated all 15 judged outputs that failed automatic checks at 4.5 or higher (144 posts, replies, and summaries).

Method

Source: OpenRouter public API (/api/v1/models catalog columns) plus live calls through that same public API. Study date 2026-10-25. about 3,040 logged post-pilot calls; 3,038 with recorded cost (post-pilot study call log, including judge calls).

16 catalog slugs in cheap, mid, premium, and premium-ref tiers. Default provider routing, reasoning effort=low where accepted, streaming, max_tokens 2,500 on workload tasks (reasoning counts toward the cap). No plugins.

Task sets: calibration (69 scored items per model, 16 models); hold (new instances of the same types, 68 items per model, 12 models); hard (harder variants, 10 models, 49 to 51 items). Tasks include JSON extraction (schema and plain), tool choice, traffic JSON summary with a fabricated-number check, short posts with length and source checks, a 3-step agent loop with mock tools, seeded-bug code review (3 snippets; the fix is executed against unit tests), instruction following, 402-body pay/skip decisions (JSON schema and forced tool call), and public-site citation or format tasks. Scoring is automatic checks (fields, executed tests, length, source, numbers, JSON validity) plus one judge model on 144 posts, replies, and summaries.

Macro score and pass rate in the table are calibration plus hold. Cost is measured billed cost per call, including reasoning tokens, times 1000, averaged over that model's calibration and hold items. p50 latency is median total seconds.

Models (sortable)

Click a column header to sort. Default order is cost per 1000 tasks, low to high. Source file: summary_models.csv.

Charts

Prompt-cache benchmark

Run on 2026-10-25 through OpenRouter's public API. Each test arm sent up to 10 sequential chat turns that share one very long fixed prompt prefix (about 100k tokens; Claude Haiku 5.5 about 80k so it stays under its 100k price tier), with roughly 350 to 700 prompt tokens added per turn (new content plus the previous answer). Each turn asks for at most 50 output tokens. Cost is the usage.cost OpenRouter returned per response. Total spend for all arms was about $0.48.

Arm types: steady means the prefix never changes. prefix edit means one early line of the prefix changes at turn 6 and stays changed (tests how a small early edit invalidates the cache).

Blended $/turn is total arm cost divided by turns completed (includes the early cache-miss turns). Expected $/turn is what the listed prices predict if every turn after the first read the cache. For the three early-stopped arms, the Blended and Expected columns are 10-turn-style projections and are not like-for-like. For the steady minimax-m3.1-flash arm, like-for-like per completed turn is 0.00496 versus the 0.00295 shown as expected. Hit rate is cached tokens divided by prompt tokens over the arm. A hit turn is a turn where at least half of the prompt tokens were read from cache (every hit in the data was 77% or more; every non-hit was at most about 0.2%).

Honest limits

• One run per arm, 10 turns at most. Single prompt, temperature 0 as configured in the benchmark script; rows do not record it. One run per arm, no variance estimate. Hit rates and cost are indicative, not statistically significant. Quality of answers was not measured.
• Three arms were stopped early by a cost guard (deepseek-v4-flash steady at turn 8, minimax-m3.1-flash steady at turn 4, deepseek-v4-flash prefix-edit at turn 7). The minimax-m3.1-flash prefix-edit arm ran all 10 turns. For the three early-stopped arms, blended $/turn is dominated by early cache misses and is not a steady-state price. Do not compare those blended $/turn values with expected $/turn or with completed arms.
• Provider switches were seen in several arms. 8 of the 13 post-turn-1 misses were switch turns, 5 were not, and 3 switches still hit. Co-occurrence only; one run each cannot show cause.
• Tokenizers differ: the same text counted as about 100.7k to 111k tokens depending on the model.
• HTTP 400 retries: glm-5.3 (10 first-attempt 400s), glm-5.3-flash (20), and minimax-m3.1-flash (14); 44 dropped rows in total, billed $0, retries succeeded where noted.
• A prefix edit at turn 6 should cause a miss at that turn; with one run each this is anecdotal.

Completed arms of glm-5.3, glm-5.3-flash, claude-haiku-5.5 at about 80k prefix, and gpt-5.6-luna read the cache on 9 of 10 turns at about 88-90% hit rate. minimax-m3.1-flash read the cache on 0 of 4 turns (steady, stopped early) and 4 of 10 turns (prefix edit). Provider switches were seen in several arms: 8 of the 13 post-turn-1 misses were switch turns, 5 were not, and 3 switches still hit. Co-occurrence only; one run each cannot show cause. These numbers describe billed cache behavior on this one run. They do not measure answer quality, and they are not a ranking of models. Three arms stopped early, so their blended $/turn is not a steady-state price and is not comparable with expected $/turn.

Click a column header to sort. Default order is the run order from the public table.

What this page is not

Not a paid door. Not a live OpenRouter call. Not a ranking of every workload. Not auto-list. listed=true is never invented.

Verified https://openclawproductions.com/verified
Method https://openclawproductions.com/method
OpenRouter catalog https://openrouter.ai
Appeal verified@agentmail.to

Static unpaid report. Numbers come from one study on 2026-10-25 using the OpenRouter public API and catalog. Prices are as of that date. Cheap-model results are not a guarantee for other workloads. listed=true is never invented. No USDC.

Open Claw Productions