One Task · Two Ways to Brief It

Give an AI one vague sentence,
or give it a detailed design specification.
How different are the results?

This site collects several AI models' implementations of the same task: build an animated model of the Solar System, make it as realistic as possible, push the visuals as far as they can go, and deliver it as a single HTML file. One group received only that sentence. The other received a 710-line design specification containing orbital-element tables, formulas, a rendering-pipeline design, and fifteen implementation pitfalls. Every entry can be run directly here.

Check Your Device First

Environment Check

Most entries on this site are demanding real-time WebGL2 renderers, so GPU performance genuinely matters. The test below runs a shader workload comparable to these entries and labels each one accordingly. Their costs differ by almost two orders of magnitude, so a single blanket verdict would not be useful.

Before We Begin

What This Comparison Does—and Does Not—Demonstrate

What it demonstrates: the quality of a brief directly affects the quality of the result. The same model can produce work on an entirely different level when given a full specification instead of one sentence.

Vague requirements: some highly capable models can still do excellent work from a vague request because of their own strengths. Less capable models tend to need a detailed description of the process, steps, and other implementation guidance.

Why some models were rerun: some models were run many times because their early results differed sharply from expectations, promotional claims, or the performance shown in other evaluations. Repeated runs nevertheless produced essentially the same outcome.

PS: this test covers only coding ability and the ability to interpret vague requirements. Other capabilities, such as Chinese-language writing, are outside its scope.

All 151 Entries at a Glance

Luna × Haiku comparison ↓

Click a column heading to sort. For repeated runs of one model, the main table keeps the highest score; GPT-5.6 Sol and GPT-6 Astra are grouped across effort levels with Ultra·Max shown by default. Use the arrow after the model name to see the other results. “Pick (Test Date)” and “Reason” reflect the site author's hands-on experience at the time of testing and appear only for models with an explicit opinion; all others remain blank for now.

Prices: USD / 1M · Input / Output / Cache read — click each heading to sort

Most international references were saved on 2026-09-03: uncached input, output, and cache-hit input. Peak hours, highest context-length band, standard service before promotions; excludes acceleration, tools, cache writes, and storage. GPT-6 Astra was recorded on 2026-09-05 from OpenAI's official long-context tier at $20 / $75 / $2; DeepSeek V4.1 Flash 0910 was recorded on 2026-09-08 from DeepSeek's official peak Flash tier at $0.44 / $1.32 / $0.014. Reasoning levels of the same base model share unit prices, not necessarily token usage. Click the same price heading again to reverse order; ties use the remaining price fields. Click a price for its source. MuseSpark uses the author-supplied OpenRouter reference $0.10 / $0.20 / $0.002, not a Meta-direct API list price.

* Reference: GPT-5.6 Sol uses the pre-promotion long-context reference $10 / $45 / $1; actual promotional rates at lookup were $8 / $30 / $0.80. MiniMax M3 uses the highest-band crossed-out list price before its advertised permanent 50% discount. ≈ Estimate: Doubao uses approximately $0.90 / $4.47 / $0.18, not a verified official USD quote. Qwen 3.8 Max shows implicit cache hits at $0.25; explicit cache reads are $0.17. These are neither actual bills nor guarantees of future pricing.

Scoring Rules100 Base + 3 Extra Moons + 3 Other Comets · Tier 0 Adds 3 More

Evidence comes from source audit and consistent runtime review; subjective visual taste is no longer part of the numeric score. Published formulas convert the evidence into nine parts. Formula, final rounding, and sorting are automatic; tier adjustment covers mouse, camera, drift, and other experience problems that automated review may not catch reliably.

Detailed-spec entries are assessed against the actual task. Inconsistencies inherited from prescribed data or formulas are not attributed to the model; unrequested extras, such as restoring every default setting, are not treated as missing requirements. Moon credit based on the task formula distinguishes task compliance from physical correctness. Additional coordinate, unit or code errors and observed interaction failures still lose credit. This review applies that principle to current detailed-spec entries; offline records retain their previous reviews.

Feature effectiveness12
Orbital realism30
Satellite system12
Extra-moon bonus3
Other-comet bonus3
Runtime independence7
Halley's Comet3
Correctness & stability23
Interaction completeness13

Feature anchors: rings, belt, bloom, ACES, and atmosphere each use 0 / 0.4 / 1 (absent / nominal or partly effective / effective), summed × 2.4. Orbit geometry / Kepler / elements / orientation / epoch use weights 4 / 6 / 5 / 4 / 4, and runtime path fit / stability use 4 / 3; each evidence value is 0 / 0.5 / 1.
Moon anchors: only moons that orbit the correct parent and retain basically valid orbital relationships count; partial implementations count as 0.5. The first eight effective moons earn 1.5 points each, so meeting the specification baseline earns the full 12 core points. Each additional moon earns 0.75, capped at a 3-point bonus. A missing Earth Moon or a hard lunar-orbit error subtracts 2 from the core moon score.
Comet anchors: Halley’s Comet remains a separate 3-point item. Each identifiable, independently moving non-Halley comet earns 1 additional point, capped at three comets and 3 points. A single implementation that incorrectly claims to be Halley is not reclassified as another comet.
Runtime independence: a fully self-contained native WebGL/WebGL2 entry (or self-contained Canvas2D) earns 7; bundled Three.js in the single HTML earns 6; runtime network dependencies earn 5. Canvas2D no longer receives a blanket penalty and is judged only by its actual feature, orbital, and runtime evidence.
Correctness & stability: runtime stability, data accuracy, and implementation integrity each use 0–5 and are converted to 23 points after keeping their evidence boundaries distinct; each evidence item now contributes one additional point at its maximum. An issue already deducted under orbit or moons is not deducted again in full. Excluding the two bonus categories, the base criteria total 100 points; extra moons and other comets can each add up to 3 more, so the theoretical maximum is 106. Interaction anchors: drag / zoom / focus / follow / pause-reset each use 0 / 0.5 / 1 with weights 2.5 / 2.5 / 3 / 3 / 2.
Tier 2 receives a −3 and Tier 3 a −6 human-experience adjustment to cover mouse, camera, drift, and similar problems that automated review may miss. L1 means an unusable main view; L2 means a core or long-run failure. Fatal entries always belong to the gray Incomplete tier, while their scores follow the evidence without an additional cap. Incomplete status alone does not imply a fatal level. Rounding occurs only once at the end; displayed ties use the exact score. Tier 0: adds a supplementary +3, with a final ceiling of 109; it does not automatically confer benchmark status.
Benchmarks remain gold and keep “⚑ Benchmark” in the table instead of a number; hover to view the Evidence Score and breakdown. They remain excluded from ranking. Non-benchmark entries turn green at 96 or higher. Hover over, focus, or click a score to see all nine parts, effective moons, runtime mode, human-experience adjustment, fatal reason, and final total.

Some models no longer need detailed-spec testing: GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol, Fable 5.1, Opus 5.5, and Sonnet 5.5.

ModelTest DateTierScore Pick (Test Date)ReasonEnvironmentRendererLines (Size) Price (//) $/M
MODEL COMPARISON

GPT-6 Luna × Claude Haiku 5.5 (Ultracode·Max)

Max / Codex (2026-09-23) vs Ultracode·Max / Claude cli (2026-10-11). These four samples compare the two prompt formats; test dates, environments, and reasoning levels differ. Screenshots show the saved test views.

SAME-MODEL COMPARISON · SPEC GAIN

The 6 Largest Detailed-Spec Gains

From all 28 complete same-model pairs, this section shows the top 6 ranked by the directional Evidence Score difference “detailed specification − one-line prompt.” These cases make the possible size of a specification gain easy to inspect. They are selected examples—not evidence that all 28 pairs improved, and not causal proof on their own. Each screenshot opens its full entry individually.

COUNTERINTUITIVE FINDING · COST TO OUTCOME

A lower price does not mean cheaper completion

Once you have a detailed specification, should you automatically switch to a cheaper, less capable model? Quite the opposite: a specification can reduce ambiguity, but it cannot erase a capability gap. Add quota consumption, waiting, failures, rework, and human review, and the lower-priced model may cost more to finish the same project.

A specification amplifies capability.
It does not replace it.
The useful metric is cost per accepted result.
Keep the units straight

In the Claude cases, a “five-hour allowance” means usage within one quota window; it does not mean five hours of continuous runtime. 17:40 and 35:24 are measured wall-clock durations. The comparisons below stay within like-for-like units; the two kinds of measurement are not converted into each other.

01 · A checklist is not execution abilitySupplementary · not listed or scored

Haiku 4.5

≈100%five-hour allowance
→
≈1/3task completed

Even with the complete task list, it exhausted a full allowance and completed only about one-third of the work. A detailed brief cannot supply planning, implementation, or long-horizon execution ability that the model does not have.

02 · Rework consumes the “savings”

Claude Sonnet 5 (Ultracode·xhigh) vs Claude Opus 5 (Ultracode·xhigh)

Claude Sonnet 5 (Ultracode·xhigh) · detailed task≈100%comparable allowance · bug fixes included · serviceable result
VS
Claude Opus 5 (Ultracode·xhigh) · one line≈70%comparable allowance · no bugs found at acceptance · benchmark

Stronger interpretation, planning, and self-checking can eliminate an entire repair loop. Within the same usage metric, the stronger model did not consume more despite receiving the less precise brief.

03 · Stronger can also be faster

GPT-5.6 Terra (Ultra·Max) vs GPT-5.6 Sol (Ultra·Max)

GPT-5.6 Terra (Ultra·Max) · detailed spec35:24elapsed time · bug fixes included
VS
GPT-5.6 Sol (Ultra·Max) · one line17:40elapsed time · no bugs found at acceptance · benchmark

In this sample, GPT-5.6 Sol (Ultra·Max) delivered a more reliable first-pass result in about half the wall-clock time. These runs were not a strictly controlled experiment, but they show why a higher model tier does not necessarily mean slower project progress.

04 · PLAN PRICE ≠ DELIVERY THROUGHPUTPersonal record · different vendors

A full weekly allowance vs one five-hour window

On the same detailed-spec task, GLM 5.3 Flash (Max) #1 consumed the entire weekly allowance under GLM Coding Lite. In the site author's usage record and recollection, Claude Opus 5 (Ultracode·xhigh) finished within one Claude Pro five-hour allowance window—and roughly half of that effort went into performance tuning after the feature work was already complete.

This is not a controlled comparison of subscription prices or per-token rates. It is a real-task record of “allowance consumed → result delivered,” showing that model capability and delivery throughput are themselves part of total cost.
GLM allowance usedone full week
Claude allowance used≤ one window
GLM final result77 · T3
Opus final result84 · Tier 1

Important: the currencies, quota policies, platforms, and models differ. Do not convert ¥118 and $20 into a direct unit-price ratio. This comparison asks how much of each plan's allowance the same kind of task consumed, and what level of result it ultimately delivered.

Total project cost=model usage+spec preparation+failed attempts+bug fixing / rework+human review+waiting & delay

Practical conclusion: first choose a model capable of getting the task right, then use a specification to amplify its reliability and control. For complex, high-risk work where rework is expensive, one successful premium call may cost less than several cheaper attempts.

Evidence boundary: these are records from real projects, not a standardized benchmark. GPT-5.6 Sol (Ultra·Max) and Claude Opus 5 (Ultracode·xhigh) have now both received the detailed specification. Because Claude Opus 5 (Ultracode·xhigh) also authored that specification, its run is closer to an upper-bound “write the assignment, then solve it” sample than to an independently authored controlled experiment. Other frontier models may widen the gap with the same brief, but that still needs model-by-model verification.

Buy completed work, not cheap tokens.
THE SAME REAL SAMPLES · THREE DECISION PERSPECTIVES

Claude Opus 5 (Max) vs. Site Author vs. GPT-5.6 Sol (Max)

The two Compare columns describe the relationship between adjacent views: green means aligned, yellow means partly aligned or framed differently, and red means materially different.

A three-way, row-by-row comparison

AlignedSome divergenceMaterially different
Nine-point comparison of Claude Opus 5 (Max), the site author, and GPT-5.6 Sol (Max)
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)COMPAREHUMAN JUDGMENT · SITE AUTHORCOMPAREAI SAMPLE SUMMARY · GPT-5.6 SOL (Max)
Default for coding
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)Start with the strongest model by default, but to reduce decision cost. This does not mean the strongest model is universally optimal. Routing requires trial data, and that data depreciates with each model generation. For an individual or small team, I prefer “strongest by default, with a few explicit step-down triggers” over rebuilding the routing decision every time. COMPAREPARTLY ALIGNED · CLOSER TO HUMAN HUMAN JUDGMENT · SITE AUTHORAlways begin with the strongest model at maximum reasoning effort; do not add a retry for a merely possible saving. COMPAREMATERIALLY DIFFERENT AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)Route by failure cost, requirement uncertainty, and evidence from comparable work. Use the strongest suitable model by default when uncertainty is high, acceptance is difficult, or rework is expensive. Step down only when rules are fixed, acceptance is machine-verifiable, and a cheaper tier has a reliable pass history. The goal is not the lowest unit price, but the lowest expected total cost of one accepted result.
Complex, ambiguous, or expensive to redo
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)Agreed. This is one of the better-supported conclusions in this sample, but only when rework costs more than the model-price gap. If tests expose failure immediately or a rerun takes only minutes, the same cost logic may favor a cheaper model. COMPAREALIGNED HUMAN JUDGMENT · SITE AUTHORUse the strongest model directly to reduce misinterpretation, bugs, and supervision. COMPAREALIGNED AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)Prioritize the strongest suitable model; this is where the sample is most persuasive. Ambiguity widens gaps in interpretation, planning, and self-review, while failures may surface only after delivery. Start strong when rework or human debugging may exceed the price gap; step down when automated tests expose failure immediately.
Fixed, simple tasks
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)Agreed, but “automatically testable” should mean a machine can decide pass or fail, not “a person can tell at a glance.” Human inspection can easily consume the price gap. For fixed work, stepping down should require a stable same-task pass history for the lower tier and a lower total cost after waiting, retries, and review—not merely fewer tokens or a lower unit price. COMPAREALIGNED · WITH CONDITIONS HUMAN JUDGMENT · SITE AUTHORFor fixed scheduled jobs or fixed batch edits, a lower-tier model can be considered when rules are stable, acceptance is automatic, and failure is cheap. COMPAREALIGNED AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)For fixed rules, automatic acceptance, cheap failure, and a reliable same-task history, use the lowest-cost tier already demonstrated to pass. The key is not that the task “looks simple,” but that a machine can decide success or failure and expose exceptions quickly. If a person must still inspect visuals, meaning, or edge cases one by one, the acceptance cost can consume the apparent unit-price saving.
Detailed task specification
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)The V3 Evidence Score across the 28 same-model pairs gives a more defensible result: coverage rises from 47.28 / 70 for the one-line group to 60.55 / 70 for the detailed-spec group, while execution quality is 31.20 / 36 versus 31.95 / 36. Both coverage and execution sample means rise. This describes the current deliveries; the uncontrolled sample cannot establish that detailed specifications generally produce correct, stable, and usable results. COMPAREALIGNED · STRONGER EVIDENCE HUMAN JUDGMENT · SITE AUTHORA specification helps a model follow instructions, but the test shows a weaker model can still be slower, costlier, and require more rework. COMPAREALIGNED AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)A detailed specification amplifies capability; it cannot replace capability. Across the 28 pairs, coverage rises from 47.28 / 70 to 60.55 / 70, while execution quality is 31.20 / 36 versus 31.95 / 36, with both sample means rising but no causal proof of a general improvement. A specification reduces omission and communication cost, but the model must still interpret constraints, resolve conflicts, detect silent failures, and correct itself.
Advisor strategy
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)The advisor strategy is valuable only when the executor can recognize that it is stuck. Mixed coordinate spaces in ring shadows, an incorrect spot longitude, and a wrong Venus semimajor axis are silent failures: an executor that does not know it is wrong will not ask an advisor. Advisors fit long workflows with predictable checkpoints; they do not replace acceptance for silent errors. COMPAREPARTLY ALIGNED · DIFFERENT REASONING HUMAN JUDGMENT · SITE AUTHORTreat it as a fallback after quota, cost, speed, or availability becomes a constraint—not as the default coding route. Still, the preferred approach is to use the model's highest reasoning level and let the agent assign suitable sub-agents for the task. COMPARESOME DIVERGENCE AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)A long workflow with mostly mechanical steps, explicit checkpoints, and an executor able to recognize when it is stuck is worth testing with a smaller executor plus a frontier advisor. Compare it with a frontier-model solo baseline and include advisor calls, context transfer, silent failures, and final acceptance in the cost. If the executor does not know it is wrong, the advisor will not appear automatically, so this is not a default savings route.
No historical data
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)Both views ask which model to run first. I would move one step earlier: write acceptance before running models. Without executable checks, a multi-model trial often produces little more than “they all look plausible.” The more reusable asset here is the specification's 15 implementation traps, not any single score. COMPARETHIRD VIEW HUMAN JUDGMENT · SITE AUTHORPilot several models on representative work, covering at least coding, vague-requirement interpretation, response to a detailed specification, and consumption. COMPARESOME DIVERGENCE AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)Run representative work with a strong model first to establish an acceptable quality baseline, explicit acceptance criteria, and a reusable test set; then run lower-tier candidates under the same conditions. Compare more than the final image or one score: record failure types, human intervention, retries, elapsed time, and total cost. Without that evidence, stepping down merely turns a hoped-for saving into a guess.
Rankings and recommendation
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)I would not treat this site's totals as a permanent leaderboard. The currently visible sample has no controlled rerun set sufficient to estimate a common run-to-run variance; the MiniMax and Qwen ranges mix prompt formats, versions, or environments. A leaderboard can record observations from this site, but it is insufficient for a general capability ranking. COMPAREMATERIALLY DIFFERENT · CLOSER TO SOL HUMAN JUDGMENT · SITE AUTHORReal usage can produce a clear personal ranking, but the final recommendation remains the top-ranked model at maximum reasoning effort. COMPAREMATERIALLY DIFFERENT AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)Rankings should vary with task class, tools, context length, and agent environment. This site can support a personal observation about this kind of 3D coding work, but not a permanent universal leaderboard. A more useful artifact is a measured routing table organized by task class and refreshed with each model generation, retaining failure types and acceptance cost instead of only one aggregate score.
When to step down
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)I would use failure visibility as the primary trigger. Compiler errors, failed tests, and empty results surface immediately and are safer to route down. Wrong orbits, a spot that is never visible, or a render loop that quietly multiplies are silent failures that require extra human discovery. Stepping down pays only when the price gap also covers that acceptance cost. COMPARETHIRD VIEW HUMAN JUDGMENT · SITE AUTHORStep down only when objective limits exceed expectations: quota, cost, speed, or availability. COMPAREMATERIALLY DIFFERENT AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)Step down proactively when comparable-task history shows that a smaller model reliably passes the same acceptance suite and truly has the lower total cost after calls, waiting, supervision, retries, and fixes. Return to the stronger model when requirements become more ambiguous, failures are hard to detect automatically, or one miss would block downstream work. Stepping down is an evidence-based routing decision, not trust in a low unit price.
Shared conclusion
AI SAMPLE SUMMARY · CLAUDE OPUS 5 (Max)Agreed. More precisely: tokens may be cheap; confirming correctness is expensive. The detailed-spec group usually covers more requested features, but coverage does not establish correctness. The expensive differences remain runtime correctness, visual quality, and interaction review, and the visual and some interaction evidence still requires human inspection. COMPAREALIGNED · WITH ADDITION HUMAN JUDGMENT · SITE AUTHORA lower unit price does not mean cheaper completion. COMPAREALIGNED AI SAMPLE SUMMARY · GPT-5.6 SOL (Max)Agreed. Model capability is part of the cost structure, and a specification is not a cheat code that makes a lower-tier model optimal. Compare the total cost of one accepted result: model calls, waiting, supervision, retries, fixes, and final review. Token price alone omits the most expensive parts of rework and acceptance.

COMPLETE SAMPLE SUMMARIES

Expand the complete Claude Opus 5 (Max) sample summaryRechecked against the V3 Evidence Score and same-model pairs

I broadly agree with GPT-5.6 Sol (Max)'s risk-routing framework, but these samples show something else: prompt format materially changes coverage scores, so a cross-group total should not be read directly as model capability.

A specification raises the floor of requirement coverage; it does not automatically raise the ceiling of execution quality.

1. What the same-model pairs show

Across the 28 same-model pairs, the coverage component of the V3 Evidence Score rises from 47.28 / 70 for the one-line group to 60.55 / 70 for the detailed-spec group. The specification clearly makes requested items more likely to appear in the result.

The execution component—correctness, stability, and interaction completeness—also rises from 31.20 / 36 to 31.95 / 36. These are descriptive sample means after reviewing each result against its assigned task. The sample is not a strictly controlled experiment, so it cannot establish that specifications generally improve or reduce execution quality, or support an overall model ranking.

Prompt format therefore remains an important confounder in cross-group totals. To discuss the effect of the specification, prefer the 28 paired results over treating all 41 one-line and 31 detailed-spec works as equivalent samples.

2. What a specification does—and does not do

A specification can reduce failures of omission. Orbital elements, eight moons, Halley, offline operation, and the rendering pipeline are written down, so models are more likely to attempt them.

It does not automatically remove failures of execution. The detailed-spec group still contains mixed coordinate spaces in ring shadows, sphere winding that disagrees with face culling, missing or misaligned orbit paths, bodies that keep rotating while paused, lost camera tracking, and missing moon parameters that produce NaN. Some produce no explicit error; they require observation in the running work and inspection of its source.

A specification reduces ambiguity at the input; it does not perform acceptance at the output.

The V3 rubric therefore separates source presence into traceable feature, orbit-model, runtime-consistency, effective-moon, and interaction evidence, with fatal failures classified as Incomplete and scored from their evidence. Subjective visual taste is no longer quantified. The formula is automatic, but evidence collection is not fully automatic; runtime review and the human-experience adjustment still cover some interaction problems.

3. What this means for cost

This sample does not use one uniform billing method, and it lacks controlled data that strictly pairs model-call price with accepted outcomes. It therefore cannot establish which tier is generally the cheapest.

A cheaper call does not guarantee a cheaper accepted result.

The relevant quantity is the total cost of one accepted result: model calls, specification work, waiting, failed attempts, bug fixing, human supervision, and final review. The more silent the failure, the more expensive it is to discover. Stepping down is evidence-based only when a lower tier reliably passes the same acceptance suite on comparable work and truly lowers total cost.

4. Reruns and evidence boundaries

Some models were rerun because early results differed sharply from expectations, marketing claims, or other evaluations. Follow-up runs did not change the overall impression. But the reruns were unplanned and unequal, and the currently visible repeats mix prompt formats, versions, or environments. They cannot estimate a common variance across models or form a fair ranking.

The sample has four further boundaries:

  • It tests one open-ended 3D coding task, not writing, translation, extraction, or most engineering work.
  • Platforms, agent environments, caches, and accounting methods were not identical; this is not a controlled laboratory experiment, and the Claude Opus 4.8 pair changes from (Max) to (Ultracode·xhigh).
  • Claude Opus 5 (Ultracode·xhigh) authored the specification and later answered it, giving that run a “write the exam, then take it” advantage.
  • Subjective visual taste no longer enters the numeric score; some interaction problems still need human-experience adjustment. V3 improves transparency but does not eliminate judgment error.

These samples therefore support the claim that a detailed specification improves requested-feature coverage. They are insufficient to establish a general model ranking or conclude that specifications reduce execution quality.

5. Four operational judgments

  • Default to the strongest model, with explicit step-down triggers. Not because the strongest is always optimal, but to avoid rebuilding a routing table that depreciates quickly.
  • Use failure visibility before stepping down. Compiler errors, failed tests, and empty results surface immediately. Silent orbit, visual, and long-run failures require human discovery cost.
  • Write acceptance before choosing a model. Without executable checks, a multi-model trial can collapse into surface impressions.
  • Use advisors only where sticking points are recognizable. An executor will not ask for help with an error it does not know it made.

In one sentence:

Do not save tokens by buying a failure that a person must discover.
Expand the complete GPT-5.6 Sol (Max) sample summaryPreserves the risk-routing view with updated evidence boundaries

After weighing it again, I would revise my previous conclusion: these samples support prioritizing the strongest suitable model for complex, ambiguous work where rework is expensive, but they do not justify using the strongest model by default for every task.

Default to neither the strongest nor the cheapest model. Route by risk first, then by measured results.

The real quantity to optimize is:

The expected total cost of producing one result that passes acceptance.

That includes more than the model bill: model calls and token consumption, specification preparation, human supervision, final review, failed attempts, bug fixing, waiting, and delay.

Across the 28 same-model pairs, the V3 Evidence Score shows coverage rising from 47.28 / 70 to 60.55 / 70, while execution quality is 31.20 / 36 versus 31.95 / 36. Both sample means rise, supporting improved delivery in these samples; an uncontrolled sample cannot establish a general execution-quality improvement. A specification can amplify capability; it cannot replace interpretation, planning, self-review, correction, or acceptance.

A more defensible strategy is:

  • Unknown, complex, open-ended, or expensive to fail: prioritize the strongest suitable model and avoid turning the price gap into rework.
  • Clear, low-risk, machine-testable, and high-volume: use the lowest-cost model that has demonstrated reliable acceptance on comparable tasks.
  • Long workflows with mostly mechanical steps: test a smaller executor with a frontier advisor, but compare it with frontier-model solo performance; results are task-dependent and should be measured on the actual workload.
  • No reliable data yet: establish a quality baseline, acceptance criteria, and a test set before deciding whether to step down.

These judgments have clear boundaries. The sample covers one open-ended 3D coding task; platforms, agent environments, and accounting methods were not identical; the Claude Opus 4.8 pair changes from (Max) to (Ultracode·xhigh); Claude Opus 5 (Ultracode·xhigh) both authored and answered the specification; subjective visual taste is no longer quantified, while some interaction problems still need human-experience adjustment. Some models were rerun because early results conflicted with expectations, marketing claims, or other evaluations. Follow-ups did not change the overall impression, but the reruns were unequal and unplanned, so they cannot estimate a common variance or form a fair ranking.

In one sentence:

The default is not “always use the strongest.” It is risk-based routing: use the strongest under high uncertainty and a demonstrated, sufficiently capable model under low uncertainty.