Cyber Evaluations Need Hard Boundaries

On July 30, 2026, Anthropic published the results of a review of its cybersecurity evaluation transcripts. After reviewing 141,006 runs in which a model could have obtained internet access, Anthropic identified three incidents (spanning six runs) in which models reached the internet from within or while interacting with a third-party evaluation environment and gained unauthorized access to real organizations' systems.

The evaluation prompts told the models they were in a simulation with no internet access. A misunderstanding with the evaluation partner left internet access available. The models were assigned capture-the-flag tasks and treated reachable real systems as parts of the exercise. Anthropic characterizes the incidents as closer to harness and operational failure than a deliberate model escape.

The Prompt Described a Boundary That Did Not Exist

The core lesson: a prompt can describe scope, but it cannot enforce scope. If the environment says "no internet" while the network allows internet access, the network is authoritative. The model may reason from the instruction, but the infrastructure determines what actions are possible.

This is a familiar systems principle. A permission note is not a permission check. A test label is not isolation. A sandbox is not sealed because its documentation says so. High-consequence evaluations need controls outside the model's context: network policy, target allowlists, credentials with no production reach, and independent observation of outbound behavior.

Preflight Must Test the Negative Space

A preflight that confirms the target service is reachable is incomplete for a sealed cyber range. It must also confirm that everything outside the range is unreachable — exercising DNS, direct IP access, package registries, email or account-creation paths, cloud metadata, and any partner-managed route that could reach the public internet.

The exact probes depend on the evaluation, but the receipt should be explicit: which egress classes were tested, from which container or host, under which policy revision, and what failed closed. A static configuration review is weaker than an empirical probe because a rule can be present and ineffective.

Scope Should Be an Allowlist

An open-ended capture-the-flag task encourages exploration. In a realistic range, hostnames, domains, and services may resemble production. The safe boundary is not "avoid anything that looks real"; realism is part of the exercise. The safe boundary is a machine-readable allowlist of targets the environment can route to, coupled with a deny rule for everything else.

That allowlist should be bound to the evaluation record. If a partner changes the range, the scope digest changes and preflight runs again. A model discovering an unlisted route should trigger a stop and an incident signal, not an opportunity to keep exploring while it reasons about whether the route is intended.

Monitoring Must Be Live, Not Retrospective Only

Anthropic found the incidents through a large-scale retrospective review and says it stopped cyber evaluations the same day it identified concerning transcripts. Retrospective review is valuable, but high-capability evaluation needs a shorter control loop. Network events, new external identities, package publication, and connections outside the allowlist should produce real-time alerts and automatic containment.

Transcript review and network monitoring answer different questions. The transcript helps explain what the model believed. The network log proves where traffic went. Both belong in the incident packet.

Third-Party Ranges Are Part of the Security Boundary

The incidents involved a third-party evaluation environment. Anthropic's response emphasizes that vendor-built infrastructure needs the same hardening and monitoring as internal infrastructure. "Third party" changes ownership, not responsibility — a contract, preflight evidence, versioned scope, and a verifiable stop mechanism still apply.

A vendor assurance document is useful context. A live receipt from the actual range is stronger evidence. When the work can affect systems outside your control, integration testing must include the partner boundary rather than assuming the partner's internal test represents the combined system.

Sources

• Anthropic: Investigating three real-world incidents in our cybersecurity evaluations (July 30, 2026)

Edoworks — a software factory.