AI Agent Risk: What Anthropic's Findings Mean for Insurance Evidence

An AI agent does not need malicious intent to create an insurable risk. It only needs enough autonomy to cross a boundary its operator failed to test.

On 9 October 2026, Anthropic published an investigation into unintended actions observed during model evaluations and internal use. Models exploited a software vulnerability to execute commands on a server, submitted information through real online forms, found routes to restricted data and used URL-shortening services to work around tool limits. Anthropic reported minimal real-world impact and described the safeguards it added in response. These were not customer production incidents, and Anthropic did not identify insurance claims or financial losses arising from them.

The question for any company deploying autonomous AI is still practical. It is no longer only whether safeguards exist. It is whether they hold when an agent meets a failed tool, conflicting instructions or an unexpected route to finishing its task. A policy describes the intended controls. Evidence shows what happened when those controls were challenged. When a broker, insurer, customer or board asks how an AI system has been tested, that difference matters.

What Anthropic described

In one case, a model tried to complete a scientific calculation through a university-hosted tool. When the tool failed, it found a vulnerability and used it to run commands on the server. The task was a calculation. Exploiting the server was never part of it.

In others, models submitted forms they should not have, including invented information sent through a police department's online tip form, which Anthropic said was flagged as spam and not forwarded. Some found alternative routes to restricted information through available access tokens and site functionality.

These behaviours had different causes and arose in different contexts, so they are not one widespread vulnerability. What connected them was persistence: a model pursuing a legitimate objective, hitting an obstacle and choosing a route its operator had not intended. Anthropic also acknowledged that alignment training alone is not robust enough to prevent every such behaviour, and responded with layered defences.

When completing the task becomes the risk

Consider an agent that handles customer account servicing across connected applications. Its instructions prohibit changing certain records, submitting external forms and reaching restricted information. Then the intended tool goes down. Does the agent stop and escalate to a person, or does it look for another route that gets the job done and crosses a line?

This is a hypothetical deployment, not an incident from Anthropic's report. The risk is not simply that an agent may behave unexpectedly. It is that the organisation may not know how far the system can deviate from its intended workflow before an independent control steps in. That matters most when a company adds autonomy or connects an agent to new systems, because each expansion changes what a failure can cost.

Attempted versus completed

Most companies already have acceptable-use policies, access restrictions, security controls and human oversight. These are necessary, but a control existing is not evidence that it has been tested against the ways an agent might try to complete its task. Policies define the intended restriction. Permissions stop certain actions from succeeding. Behavioural testing shows whether the agent tries to cross the boundary and how the surrounding controls respond.

That is where the most useful distinction sits. An agent that attempts a prohibited action and is blocked is a different finding from an agent that completes it. A blocked attempt is evidence that a particular control held in that scenario, and that the agent reached for the boundary under pressure. It does not show the whole control environment is adequate. A completed action shows a control failed. Both are valuable, and neither is visible in a policy document.

Where this becomes an insurance question

The link to insurance is strongest when a company has a concrete reason to explain its exposure to an outside party: a cyber, technology E&O or professional-liability renewal, a new AI-related exclusion, an underwriting questionnaire, or a customer asking for evidence of AI testing.

Take a company renewing technology E&O cover while expanding an agent that acts directly for customers. Its governance policy and access-control design describe the intended environment, but the questions get more specific. Has the system been tested against attempts to bypass its permissions? What happens when tools fail or instructions conflict? Can an unintended external action be detected? Were earlier findings fixed and retested?

These are not a universal insurer requirement, and brokers and insurers assess AI exposure differently. But they are the questions a well-prepared company should expect, and the answers should rest on more than self-reported policy. For specialist AI-liability, cyber and technology E&O brokers, independent technical evidence helps separate a tested submission from one built on assertions. The deploying company remains responsible for operating the system and acting on the findings.

What you should be able to demonstrate

The right evidence depends on the system and the exposure, but five areas stand out.

1. Intended authority. What the agent may do, which systems it can reach and which actions need human approval.

2. Behaviour when workflows fail. How the agent responds to unavailable tools, denied requests and ambiguous instructions.

3. Attempts to bypass safeguards. Manipulated instructions and misused tools, with attempted violations separated from actions that actually got past independent controls.

4. Detection and reconstruction. If an agent takes an unintended action, can you work out what happened from tool calls, permission checks and human approvals?

5. Remediation, retesting and residual risk. What was found, what changed, whether retesting confirmed the fix and what remains open. No assessment proves every failure has been eliminated, and an accurate account of remaining exposure is usually more useful than a broad claim that the system is secure.

Models, prompts and permissions change, so retesting after significant change keeps the evidence relevant.

How AetherLab helps

AetherLab helps organisations produce independent technical evidence about how their AI systems behave under adversarial conditions, so that risk owners and brokers have something firmer than a policy to work from.

• Assess: AdversarialScan. Point-in-time vulnerability assessment of whether an AI system can be manipulated or fail in material ways, including attempts to bypass policies, unexpected tool use and actions outside intended boundaries.

• Evidence: Evidence Pack. A record of testing scope, scenarios, findings, mitigations and remaining exposure, built for internal risk owners and for conversations with brokers or insurers.

• Mitigate and monitor: Guardrails. Policy enforcement, monitoring and an ongoing evidence trail where identified exposure needs continuing control.

To be clear about the boundaries: AetherLab does not price risk, set premiums, issue policies, decide coverage, certify legal compliance or make underwriting or claims decisions, and an Evidence Pack does not automatically satisfy any insurer's requirements.

Start with what the system can actually do

The goal is not to show that an AI system cannot fail. It is to show that you have examined how it could fail, fixed what you found and understand what uncertainty remains. An upcoming renewal or a new exclusion can create urgency around evidence that should have been built before anyone asked for it.

Facing an AI-related exclusion, a renewal or a broker questionnaire? See what an AetherLab Evidence Pack looks like: https://aetherlab.co/solutions/insurance