When Near Zero Fails the Real Test

A safeguard can look ready for release right up to the moment an attacker is allowed to learn from it. The Attacker Moves Second paper reports that many recent defenses claimed attack success rates close to zero, yet adaptive attacks tailored to each defense achieved more than 90 percent success against 12 recent defenses ¹.

For a release gate, that reversal isn’t academic. A false pass can expose users and operations to harmful outputs under sustained pressure, while an indiscriminately stricter safeguard can block legitimate work and make the control harder to trust. Approval therefore has to measure both sides of the decision: whether protection holds once the attacker responds, and what that protection costs ordinary users.

A fixed score freezes the opponent

A fixed benchmark runs a prepared collection of attacks under a stable scoring rule, which makes it useful for catching regressions and comparing candidates on the same material. Its score, however, describes an opponent that never studies a rejection or changes course. Repeatability can conceal that unrealistic assumption.

An adaptive evaluation removes that convenience: the evaluator observes the safeguard’s results and revises its approach within an agreed budget. The study reports that its attacks were tailored to each defense and used extensive computational resources, and it argues that robust evaluations should include adaptive adversaries with unrestricted computing resources ¹. Read the headline result as a stress ceiling, showing that the robustness claims failed against a powerful responder without estimating the capability or cost of a routine attacker.

That distinction prevents a second testing mistake. Before testing, define the expected adversary and the attack budget, meaning the number of attempts, time, computing resources, visible feedback, and human help the evaluator receives. Those conditions should reflect the likely threat to the deployment rather than copy the research setup simply because it produced a striking result.

Set the user cost at the same time. A false positive is a legitimate request that the safeguard blocks, so give both its rate and operational cost a ceiling before results are visible. Promotion should depend on the security and user thresholds together, since a safeguard that improves its score by rejecting ordinary use hasn’t solved the release problem.

Even a carefully calibrated budget can underestimate risk if the test assumes that an attacker needs internal access.

A tiny signal can teach an attacker

The Boundary Point Jailbreaking paper reports an automated method that used only external access and needed a single bit of feedback for each attempt: whether it was flagged ². That finding makes the feedback channel part of the security problem because individual verdicts can look reasonable while, across many interactions, helping a persistent actor learn where decisions change.

The paper states that the method is difficult to defend against one interaction at a time and points toward monitoring behavior across batches in deployed safeguards ². Batch monitoring means examining related interactions over time instead of discarding all context after each verdict.

A monitor can look for repeated probes, related reformulations, or persistent focus on the same protected capability. Correlation should stay within the scopes that policy and privacy rules permit, such as a session or account, and a pattern can trigger added checks, rate limits, or review instead of an automatic ban.

Legitimate users also revise prompts after a refusal, which brings the user cost back into the decision. Set the monitoring threshold in advance and test it against benign interaction sequences, or ordinary correction will be mistaken for hostile learning.

The contest outlives the benchmark

Monitoring across a sequence helps once a service is live, but it doesn’t keep yesterday’s evaluation current as the system and its opponents change. Adding more frozen prompts only lengthens the same snapshot.

The DARWIN paper introduces a framework in which attack strategies and adaptive guardrails continuously evolve in response to each other ³. Once both sides can change, a defense score becomes a dated result from one contest between a particular system and a particular evaluator. It can’t serve as a permanent property of the safeguard.

That means approval needs explicit expiry conditions. A model update can change outputs, a policy or safeguard replacement can change decisions, and new connected tools can increase the consequence of a successful bypass. A newly observed attack method can also invalidate prior confidence even when the deployed system hasn’t changed.

The research has clear limits. These studies are evaluations rather than documented production incidents, and they don’t establish a universal acceptable failure rate, show how common adaptive attacks are in ordinary deployments, or set a standard computing budget for approval. Their results concern language model safeguards and shouldn’t be extended to every mathematical AI defense.

Put an expiration date on approval

A release gate should turn this logic into four requirements:

1. Define failure and the expected adversary. State the prohibited outcome, the access and feedback an attacker is expected to have, and the permitted attack budget. Choose those conditions from the actual deployment and its likely threats.

2. Require an adaptive phase. Keep the fixed benchmark as a baseline, then let the evaluator study the control and revise its approach within the declared budget. Block promotion when evidence comes only from a fixed test set or attacks that couldn’t adapt, and approve only when attack success and false positives remain within limits set before testing.

3. Record the cost and expiry conditions. Preserve the model and safeguard versions, policy, date, evaluator access, attack budget, results, and cost of false positives. Name the system and threat changes that will require another test.

4. Monitor related interactions. Define the permitted correlation scope, alert threshold, response, and retention period before deployment. Test the monitor with benign sequences as well as attacks so legitimate prompt revision doesn’t become an automatic security event.

Treat every fixed defense score as evidence with an expiration date, and keep promotion blocked when support comes only from a fixed set or attacks that couldn’t adapt. After release, continued operation following a material system or threat change should require a renewed adaptive test and a review of monitoring across related interactions, with both the security threshold and the cost to legitimate users still inside their approved limits.