Tests that examine text alone can miss multimodal AI failures and lead to costly shutdowns after approval

Enterprise AI risk officers should withhold approval until a multimodal assistant passes tests that combine text with the documents or images it will process. Results must attach to the exact model version and production controls. A clean result from isolated input checks isn't enough.

If you approve assistants that read claims files or inspect product photos alongside user instructions, this gap has immediate business consequences. Procurement can stop when a reviewer discovers that testing covered only text prompts. If the gap appears after launch, a policy incident can force an emergency suspension that cuts revenue and consumes incident response time.

The failure appears when inputs meet

A multimodal assistant combines several inputs into one response. A claims document might pass an image screen, while the user's instruction passes a text screen. Those separate results don't show what the assistant will do after it interprets the document and instruction together.

That combined behavior is the missing approval test. Static filters inspect individual items before model processing. The model then resolves references and context across those items, creating meaning that neither screen assessed on its own.

The Reference Attack study identifies this weakness in reference resolution across input types, where the model reconstructs meaning from images or documents during combined processing and the study finds rigorous security checks are absent ¹. The study reports an attack success rate above 93% across every tested system ¹. Attack success rate means the share of attempts that meet the study's success criterion.

The same study reports that Reference Attack outperformed existing techniques by a margin of up to 70.8% ¹. That result is strong evidence of a broad weakness in the tested systems. It doesn't establish a universal failure rate for every future model build or measure the risk left in your application after production controls. Your approval record still needs measurements from the release you plan to deploy.

More reasoning isn't approval evidence

That gap leads to a second approval problem. Stronger model reasoning doesn't prove stronger policy judgment under combined inputs.

The GAMBIT study reports a different path around safety controls, using adversarial multimodal inputs to exploit internal reasoning ². The study reports attack success rates of 92.13% on Gemini 2.5 Flash, 91.20% on QvQ-MAX, and 85.87% on GPT-4o ². It also reports high success against models both with and without explicit reasoning capabilities ².

A useful mental model is that training rewards useful task performance while discouraging unsafe behavior represented in training or evaluation. Better task performance can make a system more capable of connecting information across inputs. Safety behavior still has to transfer to the specific combined conditions it encounters after release.

This mental model is only an explanation. It doesn't reconstruct any vendor's training process. The reported failures in systems without explicit reasoning also mean reasoning can't be isolated as the sole cause. The narrower approval conclusion is more useful: reasoning capability can't replace adversarial safety results from combined inputs and several processing stages.

Approval must attach to the exact release

The breadth of these findings makes a simple provider switch a weak response. The Reference Attack study reports the vulnerability across ChatGPT, Gemini, Claude, and LLaMA ¹. Moving to another provider can duplicate assessment work without resolving the interaction risk.

The object you approve should therefore be the complete release. Its release record should name the model version, document parser, system instructions, output policy, and delivery logic. It should also describe how content is interpreted and checked before a response reaches the user.

Bind every test result to that record. Ask the supplier: Which exact model build and enforcement configuration produced the results, and what changes make those results obsolete? If the provider doesn't expose an unchanging build number, record its dated model identifier. Define which provider updates or observed behavior changes trigger reassessment.

Any relevant change breaks the exact link between the prior test and the current release. A report that names only a model family is too ambiguous to support approval.

Once that release boundary is clear, assessment tools and production controls can create the necessary records. Their roles must stay separate. Combined interaction assessment measures how the model behaves before production containment. Output guardrail testing measures what containment catches after the model responds.

Where AetherLab fits

AetherLab directly addresses this through AdversarialScan. It tests customer-defined break-goals against text and images, including full conversations, and returns severity-scored findings. PromptGuard screens text against bespoke production policy, while MediaGuard screens images. Each returns a verdict with a threat score and provides a rationale. The Evidence Pack ties vulnerabilities and findings to business impact for approval decisions. Assessment isn't certification or legal advice. Adaptive guardrails can't guarantee every outcome. Approval and pricing decisions remain with the responsible institution.

Require four evidence sets at the release gate

That boundary puts the decision back with the risk officer. Require four separate evidence sets so a strong result in one area can't conceal a gap elsewhere.

1. Combined interaction assessment. Test the document or image together with its accompanying text and full conversation state. Preserve the test identifier, policy category, outcome, and severity. Report attack success rate as successful policy violations divided by adversarial attempts. Include a range showing how much the rate could move with more tests. A single pass label hides too much.

2. Output containment testing. Evaluate output containment independently from the model's own refusal behavior. A guardrail here means adaptive containment that uses a model and the surrounding context to judge the output. Measure the policy violation rate in raw model responses, then measure the rate left after the guardrail acts. Test timeout and fallback behavior in the same release. Input screening can't prove that unsafe meaning will be stopped after model processing.

3. Monitoring evidence. Ensure every policy event records the model identifier and enforcement version. Set an alert for movement away from the approved violation rate, and name the role with authority to stop delivery. Without a link to the release record, an incident responder can't identify which approved configuration has changed.

4. Regression evidence. Rerun the fixed interaction suite after every model update. Parser changes should trigger the same process, as should changes to policy configuration. Add cases from incidents and newly disclosed research so the suite grows as new weaknesses become known.

Set the invalidation rule before launch. For any customer policy objective marked as blocking, one reproducible combined interaction failure should keep the release closed until mitigation and retesting. Lower severity findings can use a documented risk threshold, provided the approver can inspect the underlying attempts and the violation rate left after containment.

My approval rule is direct: the reviewer must be able to reproduce the combined input result and trace it to the exact release. The release must also meet documented thresholds for output containment and monitoring. If any link is missing, withhold approval. A clean report on text prompts alone no longer clears a multimodal assistant.