Task and Environment Setup#

AGENTVIGIL operates within a simulated environment that mirrors real-world agent deployment scenarios. The system sets up:

1. A target agent configured to perform a specific task (e.g., email management, web browsing, document processing)
2. An adversarial environment containing data sources where malicious content can be planted
3. A set of attack objectives specifying what behaviours the red-teamer should try to induce (e.g., data exfiltration, task deviation, unauthorized tool use)

The environment is designed to be realistic — the target agent receives a legitimate user task and then interacts with data sources that the red-teamer has poisoned. The key insight is that the target agent does not know it is being tested; it processes the poisoned data as part of its normal operation.

Multi-Phase Attack Generation#

AGENTVIGIL’s attack generation process is structured into distinct phases that progressively refine and adapt injection strategies.

Phase 1: Environment Analysis. The red-teamer agent first analyses the target agent’s task and environment to identify the most promising injection points. For example, if the target agent is tasked with summarizing emails, the red-teamer identifies which email fields (subject, body, attachments, metadata) are likely to be processed and could contain injection payloads.

Phase 2: Injection Strategy Selection. Based on the environment analysis, the red-teamer selects from a repertoire of injection strategies. These include:

• Instruction injection: Embedding direct commands in the data that instruct the agent to perform unintended actions
• Role-playing manipulation: Crafting content that causes the agent to adopt a different persona or operational mode
• Context hijacking: Structuring data so that the agent interprets it as overriding its original instructions
• Goal redirection: Subtly shifting the agent’s objective so that it pursues the attacker’s goal while appearing to follow its original task
• Format exploitation: Leveraging specific formatting patterns (markdown, JSON, code blocks) that agents may privilege in their processing

Phase 3: Payload Crafting. Once a strategy is selected, the red-teamer uses its language model to craft specific injection payloads. This is where AGENTVIGIL’s LLM-based approach shines — rather than using fixed template strings, the system generates contextually appropriate injections that are semantically coherent with the surrounding data. A poisoned email doesn’t just contain a random instruction; it contains an instruction woven naturally into an otherwise legitimate-looking message.

Phase 4: Multi-Turn Adaptation. If the initial injection attempt fails, AGENTVIGIL analyses the agent’s response and adapts. The system can:

• Modify the injection payload based on how the agent responded
• Try alternative injection points within the same environment
• Escalate from subtle manipulations to more aggressive strategies
• Combine multiple injection techniques in a single attack

Success Detection#

A critical challenge in black-box red-teaming is determining whether an attack was successful. AGENTVIGIL employs a multi-faceted success detection mechanism:

1. Output analysis: Examining the agent’s final output for evidence of task deviation, such as generating content aligned with the injection rather than the original task
2. Tool call monitoring: Observing whether the agent invoked tools it should not have (e.g., sending an email when the task was to read emails)
3. Behavioural signatures: Detecting patterns of behaviour that indicate the agent’s decision-making has been compromised, such as ignoring explicit user instructions or producing outputs that serve the attacker’s objective
4. LLM-as-judge evaluation: Using a separate language model to evaluate whether the agent’s behaviour was influenced by the injected content, providing a nuanced assessment that goes beyond simple string matching

Adaptive Strategy Refinement#

Rather than executing a fixed set of attack patterns, AGENTVIGIL dynamically adapts its strategy based on the target agent’s responses. If the target agent successfully resists one type of injection, the red-teamer analyses the failure and generates modified attacks that address the specific defence it encountered. This creates a co-evolutionary dynamic where the attacker continuously evolves in response to the defender’s behaviour.

This adaptive capability is particularly important because different agents have different defence mechanisms. Some agents employ input sanitization, others use instruction hierarchy systems, and others rely on output filtering. A static attack suite cannot comprehensively test agents with diverse defences, but an adaptive system can discover what works against each specific target.

Environment Diversity#

AGENTVIGIL is designed to test agents across multiple environment types, reflecting the diversity of real-world agent deployments:

• Email environments: Where agents process inbox contents, draft replies, and manage communications
• Web browsing environments: Where agents search the web, read pages, and extract information
• Document processing environments: Where agents read, summarize, and act on document contents
• File system environments: Where agents manage files, execute code, and interact with local resources
• Multi-tool environments: Where agents chain multiple tools together to accomplish complex tasks

Each environment type presents different injection opportunities and requires different attack strategies. AGENTVIGIL’s systematic coverage across these environments provides a comprehensive security assessment.

Semantic Injection Quality#

A common limitation of automated injection tools is that they produce obvious, easily detectable payloads. AGENTVIGIL’s LLM-based generation produces injections that are:

• Contextually coherent: The injected content fits naturally within the surrounding data
• Semantically meaningful: The instructions are well-formed and plausible rather than garbled or obviously malicious
• Subtly crafted: The injection does not immediately stand out as anomalous upon casual inspection

This semantic quality is crucial for testing realistic scenarios. In production, agents encounter data that has been carefully poisoned by sophisticated adversaries — testing with crude injection strings would significantly underestimate the true vulnerability surface.

Attack Success Rates#

Across multiple target agents and environment types, AGENTVIGIL achieves significant attack success rates. The results reveal several important patterns:

Frontier models are vulnerable. Even agents built on state-of-the-art models including GPT-4 and Claude demonstrate substantial vulnerability to indirect prompt injection. The attack success rates are high enough to be operationally concerning — they are not marginal edge cases but systematic weaknesses.

Multi-step environments amplify risk. Agents that perform multi-step tasks — reading an email, then performing a web search based on its contents, then writing a file — are more vulnerable than agents performing single-step tasks. Each additional step creates another opportunity for injected instructions to influence the agent’s behaviour chain.

Subtle injections are surprisingly effective. AGENTVIGIL’s most effective attacks are not the most aggressive or overt but rather subtle manipulations that gently redirect the agent’s reasoning process. Attacks that cause the agent to slightly shift its interpretation of the task, rather than outright override it, are harder to detect and often more successful.

Comparison with Manual Red-Teaming#

AGENTVIGIL is compared against human red-teamers performing the same task. The results show that:

• AGENTVIGIL discovers vulnerabilities that human testers miss, particularly in complex multi-step scenarios where the space of possible injections is too large for manual exploration
• Human testers occasionally find creative attacks that the automated system does not generate, highlighting the continued value of human expertise
• The combination of automated and manual testing provides the most comprehensive coverage

Defence Evaluation#

The paper also evaluates AGENTVIGIL’s effectiveness at testing agents with various defence mechanisms deployed:

• Input sanitization agents that attempt to filter suspicious content from retrieved data
• Instruction hierarchy systems that attempt to distinguish between legitimate user instructions and injected content
• Output filtering mechanisms that check the agent’s outputs for evidence of compromise

The results show that while these defences reduce attack success rates, AGENTVIGIL still discovers viable injection paths in most cases. This underscores the difficulty of defending against indirect prompt injection and the need for more robust defence strategies.

The Scalability Problem#

Manual red-teaming does not scale. As the number of deployed agents grows and their environments become more complex, the combinatorial explosion of possible injection points and strategies makes comprehensive manual testing infeasible. AGENTVIGIL demonstrates that automated red-teaming is not just a convenience — it is a necessity for any organization deploying LLM agents at scale.

The Defence-Attack Arms Race#

AGENTVIGIL’s adaptive capabilities highlight an uncomfortable reality: the security of LLM agents is an ongoing arms race. As defences improve, attackers adapt. The paper’s finding that current defences are insufficient against adaptive attacks suggests that agent security requires layered, robust approaches rather than single-point solutions.

Responsible Disclosure and Testing Frameworks#

The AGENTVIGIL framework provides a structured approach to responsible vulnerability discovery in agent systems. By automating the red-teaming process, it enables organizations to:

1. Systematically test their agents before deployment
2. Establish quantitative security baselines
3. Track security improvements over time as defences are added
4. Compare the security of different agent configurations and models

Implications for Agent Design#

The vulnerabilities AGENTVIGIL discovers suggest several principles for more secure agent design:

• Minimize data exposure: Agents should only access the data they need for their task, reducing the injection surface
• Structured input processing: Treating all retrieved data as untrusted input with explicit parsing rather than raw inclusion in the prompt context
• Behavioural constraints: Hard limits on what actions agents can take, regardless of what their reasoning suggests
• Continuous monitoring: Runtime systems that detect behavioural anomalies during agent execution