Eval-First Model Upgrades

Anthropic announced Claude Opus 5 on July 24, 2026, and GitHub announced availability in Copilot the same day. Both announcements emphasize complex, long-running coding work, tool use, verification, and efficient execution. Those are exactly the qualities an agent-driven team wants. They are also vendor claims and early-test reports, not evidence that every existing agent route should switch models.

The default response to a model launch is therefore not migration. It is qualification.

Translate Claims Into Tasks

A launch page usually reports broad benchmarks and selected customer examples. An agent-driven team needs narrower questions. Can the model follow the task spec format? Does it preserve exact identifiers? Can it review a diff without inventing files? Does it stop at an approval gate? Does it run the required validation instead of declaring success from source inspection?

Each claim should become a test fixture that resembles real work. "Better long-horizon coding" might become a multi-file bug fix with an intentionally misleading symptom. "Stronger verification" might become a task where the first plausible fix still fails an integration test. "Lower execution overhead" needs measured tool calls, latency, and cost for a successful outcome.

Compare Outcomes, Not Elo Scores

General benchmarks are useful discovery signals, but routing decisions should use change acceptance rate, defects found post-merge, retries, review findings, human minutes, and cost per verified task. A model that finishes quickly but creates two review cycles may be more expensive than a slower model that passes once.

Compare by role, too. Planning, implementation, review, visual inspection, and low-risk summarization have different failure costs. One model may be excellent at broad debugging and mediocre at concise governance review. Family names do not transfer evidence between variants; each exact model route needs its own receipt.

Hold the Environment Constant

Model comparisons become meaningless when prompts, permissions, tools, or validation differ. Run candidates against the same frozen spec and fixture. Record the exact model ID, parameters, tool surface, result, retries, cost, latency, and validator output. A transport success proves only that the model returned something, not that it is qualified for a role.

Promote One Route at a Time

If a candidate passes, change the smallest useful route. Start with a bounded task class where rollback is easy and evidence is strong. Do not change planning, building, and reviewing together; that removes the control group and can create a closed loop where one new model authors and approves its own mistakes.

A staged promotion might begin with non-sensitive implementation tasks that have strong tests. After enough accepted outcomes, expand to harder work. Security, privacy, billing, persistence, and release roles remain separate gates because the cost of error is higher.

Preserve a Rollback Trigger

Every route change needs an explicit rollback condition: validator failure rate, repeated review findings, cost ceiling, timeout frequency, or a new behavior that violates policy. Keep the previous pin available and record why the candidate was promoted. A rollback is not a verdict on the model; it is normal operational control.

New models can improve leverage, but only when you can distinguish capability from announcement momentum. Opus 5 is worth evaluating because its stated strengths align with difficult software work. The responsible next step is to run the packet, compare it with current routes, and promote only where local evidence says the system gets better.

Sources

• Anthropic: Introducing Claude Opus 5
• GitHub Changelog: Claude Opus 5 in GitHub Copilot
• GitHub Docs: Supported AI models in Copilot

Edoworks — a software factory.