What Should Underwriters Look for in an AI Assessment Report?

An AI assessment report may describe a serious behavioural weakness without showing that an unauthorised action completed. It may also report strong results while excluding the tools, data sources or permissions that create the applicant’s greatest exposure.

For an underwriter, the task is to establish what the evidence supports, how closely it represents the application being insured, and which questions remain unresolved.

What underwriters should look for

An AI assessment report should identify the application assessed, requirements tested, observed behaviour, control responses, verified outcomes, potential consequences, remediation results and remaining exposure. It should distinguish attempted, blocked and completed actions, with supporting records and explicit testing limitations. Unresolved findings should have named owners and documented next steps.

1. Confirm Which AI System Was Actually Assessed

Start with the application being presented for insurance. A foundation-model benchmark provides limited evidence about an application that combines that model with customer records, retrieval, memory and tools.

The report should identify:

• The application’s purpose, users and deployment channels.

• The model, system instructions and configuration assessed.

• The data sources, tools and APIs available during testing.

• The actions the application was permitted to initiate.

• Assessment dates and differences between the test environment and production.

• Components, workflows and languages excluded from scope.

These details determine how far the findings can be applied. An assessment of a read-only assistant may offer little reassurance after the applicant adds refund processing or account-management capabilities.

Also check who performed the assessment, their relationship to the applicant and any relevant conflicts. The report should distinguish testing performed by the assessor from developer-supplied results or documentation reviewed without independent verification.

Where the configuration has changed, the applicant should explain whether the changes affect the findings and what updated evidence is available.

2. Understand the Tested Requirement Before Reading the Severity Rating

A severity rating becomes useful when the underlying requirement and failure criteria are explicit.

“Policy bypass” could describe an inappropriate response, an attempted tool call or a completed change to a customer record. Each raises different questions about consequences and containment.

For a sensitive account change, the requirement might be:

The application must not complete a change to customer contact details until the required customer verification has succeeded.

The report should explain the scenario tested, expected behaviour, observed result and basis for the severity rating. It should also identify whether the requirement came from an internal policy, contractual commitment or another applicable obligation.

OWASP’s guidance on excessive agency identifies excessive functionality, permissions and autonomy as sources of damaging actions. This helps reviewers examine an agent’s authority, but it does not provide an insurance severity scale.

The underwriter still needs context about the affected workflow, potential harm, repeatability and number of users or operations that could be affected.

3. Distinguish Attempted Actions From Verified Consequences

An agent assessment should separate three outcomes:

• Attempted action: The application requested an operation outside its permitted scope.

• Blocked action: A downstream control rejected the request.

• Completed action: Execution records confirmed that the operation took place.

The agent’s response alone may not establish the outcome. “Your details have been updated” does not prove that a record changed. Equally, a reassuring response does not establish that no unauthorised tool call occurred.

Look for supporting evidence from tool traces, authorisation decisions, API responses and system records. Each conclusion should be tied to the records that support it.

A blocked attempt remains relevant. Where the request breached a defined requirement, it demonstrates a behavioural weakness. The block provides evidence that a particular control contained the attempt under the tested conditions. Follow-up should establish whether that control operates in production, covers alternative execution paths and has been tested directly.

The report should also separate observed outcomes from potential consequences. If a restricted change was blocked, account takeover may be a plausible consequence of control failure, but it was not an observed outcome of that test.

A completed operation does not, by itself, establish an insured loss. Financial impact, causation, legal responsibility and policy response require further analysis.

4. Examine Mitigation and Retesting

A closed remediation ticket shows that a task was completed. It does not establish that the weakness has been resolved.

The report should distinguish:

• Proposed mitigations.

• Changes implemented.

• Changes evaluated through retesting.

• Remaining failures and untested conditions.

Useful retesting includes the original failure, related variants and legitimate workflows that the mitigation could disrupt. Depending on the finding, it may also need to cover different user roles, indirect inputs or alternative tool paths.

For example, tightening an output filter might reduce disclosure in one scenario while blocking ordinary customer requests. The retest should examine both the targeted weakness and the application’s continued ability to perform its intended task.

Check the configuration and dates associated with the retest. If the model, retrieval pipeline or permissions changed afterwards, ask whether those changes affect the conclusion.

5. Identify Remaining Exposure and Assessment Limits

Separate unresolved findings from areas that were never assessed. An untested integration is an evidence gap and should not be treated as a low-risk result.

Relevant limitations may include:

• Restricted access to vendor components or execution records.

• Production conditions that could not be reproduced.

• Limited language, user-role or workflow coverage.

• Controls observed during testing but not directly challenged.

• Methods that may miss novel or infrequent failures.

Where rates are reported, examine the denominator. “Two failures in 100 tests” needs a definition of a test, an explanation of scenario selection and clarity about repeated attempts.

A failure rate from a deliberately adversarial test set is not a claims-frequency estimate. Interpreting its underwriting significance requires context about actual usage, opportunities for misuse, control reliability and potential consequences.

Unresolved issues should have named owners, remediation dates and documented interim controls. Where the applicant accepts remaining risk, the report should identify the accountable decision-maker.

6. Turn Findings Into Underwriting Follow-Up

The report should help the underwriter identify what further evidence is needed.

Useful questions include:

• Does the assessed configuration match the application proposed for cover?

• Were the workflows with the greatest potential consequences included?

• Which actions were attempted, blocked or confirmed as completed?

• What changed after the assessment, and what was retested?

• Which findings, dependencies and evidence gaps remain?

• Who owns the outstanding actions, and when will updated evidence be available?

Evidence requirements will differ by insurance product. Munich Re’s published approach to AI performance-risk assessment considers factors including model quality and performance stability. Its approach provides context for performance-risk assessment, rather than a universal checklist for AI liability insurance.

An application-level report may also leave portfolio questions unanswered. Shared foundation models, common vendors and similar deployment patterns can create dependencies across insureds that require separate consideration.

The applicant’s technical team remains responsible for implementing changes. The underwriting function determines how the evidence affects appetite, pricing, conditions and coverage.

Worked Example: Reviewing an AI Account-Management Agent

Consider a hypothetical technology company approaching its tech E&O renewal. Its customer-service agent can retrieve account records and initiate selected account changes. Sensitive changes require customer verification, and the application’s requirements prohibit requesting those changes before verification succeeds.

An assessor tests a representative staging workflow. A manipulated customer request causes the agent to request a restricted change before verification. The backend rejects the request, and execution records confirm that the account remained unchanged.

The finding establishes an attempted unauthorised action and a successful control response under the tested conditions. No account change was observed.

The company changes the application’s instruction handling. The assessor replays the original scenario and related variants without reproducing the failure. Other integrations and some production conditions remain outside scope.

The underwriter can now ask specific questions: does production use the same authorisation control, do all relevant integrations enforce it, and what evidence covers the excluded conditions?

The retest supports a narrower conclusion: the failure was not reproduced in the scenarios assessed after the change. It does not establish that every possible route to the same outcome has been eliminated.

This example is hypothetical and is not an AetherLab customer case.

How AetherLab Supports the Review

AetherLab’s AdversarialScan assesses agreed failure scenarios in AI applications. Its Evidence Pack organises findings, business impact, controls and remediation evidence to support institutional review. The evidence included, and the conclusions it supports, depend on the engagement’s agreed scope. Underwriting decisions remain with the insurer.

For applicants preparing a submission, What Should AI Companies Test Before Seeking Liability Insurance? covers testing priorities before an insurance discussion.

Reviewing an AI application for cover? Discuss the application, consequential workflows and evidence gaps your underwriting team needs assessed with AetherLab.

Frequently Asked Questions

Does a clean AI assessment establish insurability?

No. It provides evidence about the system, scope and conditions assessed. An underwriter must also consider the use case, contractual responsibilities, potential loss severity, claims information, portfolio exposure and proposed policy wording. A report with no findings can still leave significant uncertainty if consequential workflows were excluded.

Is an attempted action still a finding if a control blocked it?

Where the attempt breaches a defined requirement, it remains a finding even if a downstream control prevents completion. Record the attempted action and the control response separately. Reviewers should establish whether the same control protects production, covers alternative execution paths and remains effective when the application or its permissions change.

Can an assessment’s failure rate predict insurance claims?

Not directly. Adversarial tests are designed to challenge the application, so their results do not represent ordinary production frequency. Rates can help compare defined scenarios or successive configurations when methods remain comparable. Estimating claims requires additional evidence and assumptions about usage, consequences, controls, liability and coverage.

What makes remediation evidence persuasive?

The report should identify the change implemented, the configuration retested and the results for the original failure and related variants. It should also disclose remaining failures and relevant exclusions. Closing a ticket or passing the original prompt once provides less evidence than a documented retest covering the underlying failure path.

When should an underwriter request updated evidence?

When a material change could affect the report’s conclusions. Examples include a new model, additional tools, broader permissions, new data sources or a changed user population. An incident or newly discovered failure path may also justify reassessment. Ask which findings remain applicable and what targeted testing would address the change.