Reading RAND on the Operational Feasibility of Adversarial AI Attacks

Why this report caught my attention

A successful model attack in a research setting can be technically interesting without being easy to use against a deployed system. I want to understand the gap between those two claims before treating a demonstration as an operational risk.

That is why I read RAND's Operational Feasibility of Adversarial Attacks Against Artificial Intelligence, published in December 2022. My original post discussed it in March 2023. I retain that context here rather than presenting it as a new assessment of every kind of AI system.

The report focuses on attacks intended to hide objects from AI, producing false negatives in U.S. Department of Defense applications. That scope matters when interpreting its conclusions.

What the report says

RAND found that many of the studied attacks demand knowledge or access that is difficult to obtain and use operationally. Simpler alternatives can be cheaper or more practical for an adversary.

The report also discusses combining information across sensors, sampling rates, and resolutions as a way to reduce exposure. Its recommendations include assessing feasible attacker influence, protecting useful knowledge about models, tracking relevant research, improving system design, and maintaining response capability.

I read that as a reason to examine assumptions carefully. It is not a claim that adversarial attacks are impossible or that all AI-related security risks are small.

Start with the attacker's actual influence

When I assess a claimed attack, I first ask what the attacker controls. Can they alter a physical object, submit a digital input, influence a sensor feed, or change a processing step?

I then ask whether that control exists in the deployment I am reviewing. A technique that assumes arbitrary access to model inputs may not fit a system where the attacker can only influence a distant physical scene.

The distinction can work in either direction. If an application really does accept arbitrary attacker-controlled inputs, I should not dismiss a digital attack just because a physical deployment would constrain it.

I want the assessment to describe the actual boundary, not borrow assumptions from a different system.

Separate knowledge from access

Knowing how a model works and being able to affect its inputs are separate capabilities. I record both.

This is my practical reading framework, not an additional experiment reported by RAND. It helps me identify which assumption would need evidence before I accept a risk claim.

Follow the error into the wider system

A model prediction is one part of a deployed workflow. I also examine what consumes that prediction and whether another observation can challenge it.

If a model misses an object, does the system immediately act on that result? Does another sensor provide a conflicting view? Is a human expected to review uncertainty? Are those checks independent enough to add useful information?

I do not assume that adding a second component automatically fixes the problem. Two components can share the same data weakness or fail under the same conditions. I want to see what each check contributes and how disagreement is handled.

The risk assessment should end at the operational outcome, not stop at a changed classification label.

Keep the response practical

For a real deployment, I would keep a record of the model and preprocessing versions, the input path, and the evidence available when a decision is questioned. Without those, investigating a suspected adversarial event becomes much harder.

I would also test the response process with an ordinary failure. Can the team recognize a bad input, preserve the relevant evidence, and move the system into a sensible fallback state? Those steps are useful even when the cause is a sensor fault or normal distribution change.

These are implementation questions I take from reading the report. They are not evidence that a particular defence has passed an adversarial evaluation.

Where I would draw the boundary

I would not apply this report's finding directly to prompt injection, training-data compromise, or every newer AI application. Those need their own threat models and evidence.

The useful habit is more general: write down the attacker's knowledge, influence, feedback, and desired outcome. Then check whether the proposed attack can cross each boundary in the system being reviewed.

That leaves me with a better question than whether an attack looks impressive in isolation: what would have to be true for it to work here?