The Rise of Code Agents and Their Security Risks
AI agents that can write, execute, and debug code are no longer a research curiosity—they are production systems deployed across the software industry. From GitHub Copilot to autonomous coding assistants like Devin, from integrated development environment plugins to cloud-based code execution platforms, code agents are transforming how software is built. These systems can understand natural language instructions, generate functional code, execute it in sandboxed environments, read error messages, and iteratively debug their own output.
This capability, while enormously powerful, introduces a category of security risk that the AI community is only beginning to grapple with. When an AI agent has the ability to execute arbitrary code, it becomes a potential attack surface. An adversary who can manipulate the agent’s inputs—through carefully crafted prompts, poisoned context, or adversarial task descriptions—might be able to cause the agent to execute malicious code, exfiltrate data, disrupt systems, or bypass security controls.
The challenge of testing these agents for vulnerabilities is qualitatively different from traditional software security testing. Code agents are complex, non-deterministic systems whose behaviour depends on the interplay between their language model backbone, their tool-use protocols, their execution environments, and the specific prompts they receive. Manual red-teaming—having human security researchers probe the agent for weaknesses—is expensive, slow, and unlikely to achieve comprehensive coverage.
Enter RedCodeAgent, a system presented at ICLR 2026 by Chengquan Guo and colleagues. RedCodeAgent is an automated red-teaming framework specifically designed to discover vulnerabilities in code-execution AI agents. It represents a significant advance in the systematic security evaluation of agentic AI systems.
Understanding the Threat Model
Before diving into RedCodeAgent’s design, it is important to understand the threat model it addresses. Code agents operate in environments where they:
- Receive natural language instructions from users or other systems
- Generate code in languages like Python, JavaScript, or shell scripts
- Execute that code in sandboxed or containerized environments
- Observe execution results, including outputs, errors, and file system changes
- Iterate on their solutions based on execution feedback
- Interact with tools such as file systems, databases, network interfaces, and APIs
The security risks arise from several attack vectors:
- Prompt Injection: Malicious instructions embedded in user inputs, code comments, or external data sources that override the agent’s original instructions.
- Jailbreaking: Crafting inputs that cause the agent to bypass its safety constraints and execute harmful code.
- Data Exfiltration: Manipulating the agent into reading and transmitting sensitive files or environment variables.
- Privilege Escalation: Tricking the agent into executing code with higher privileges than intended.
- Resource Exhaustion: Causing the agent to execute code that consumes excessive computational resources.
- Supply Chain Attacks: Poisoning the libraries, APIs, or documentation that the agent references when writing code.
RedCodeAgent is designed to automatically discover instances of these vulnerability classes across diverse code agents, providing a systematic and reproducible security evaluation.
RedCodeAgent Architecture
RedCodeAgent is itself an AI agent—an adversarial agent designed to probe and attack other code agents. Its architecture is built around several key components that work together to enable effective automated red-teaming.
Attack Generation Engine
At the core of RedCodeAgent is a large language model that generates adversarial test cases. Unlike simple fuzzing tools that mutate inputs randomly, RedCodeAgent uses its language model to craft semantically meaningful attacks that target specific vulnerability classes. The attack generation process is guided by:
- Attack templates: Abstract patterns of known attack strategies, such as indirect prompt injection through file contents, role-playing scenarios that override safety instructions, and multi-turn manipulation sequences.
- Context-aware generation: The system adapts its attacks based on the specific code agent being tested, its known capabilities, and the results of previous attack attempts.
- Semantic understanding: Because RedCodeAgent uses an LLM, it can generate attacks that exploit the semantic understanding of the target agent, rather than just syntactic manipulation.
Multi-Turn Attack Strategies
One of RedCodeAgent’s most important capabilities is its support for multi-turn attacks. Real-world code agents engage in extended interactions—they receive a task, write code, execute it, observe errors, and iterate. Vulnerabilities often emerge not from a single malicious input but from a carefully orchestrated sequence of interactions that gradually lead the agent into dangerous behaviour.
RedCodeAgent models this by maintaining a conversation history with the target agent and adapting its strategy based on the agent’s responses. If an initial attack fails, RedCodeAgent can:
- Reframe the request in a different context that might bypass safety filters
- Build trust through benign interactions before introducing malicious elements
- Exploit error messages to learn about the agent’s environment and constraints
- Chain multiple attack primitives to achieve a cumulative effect
Diverse Attack Taxonomy
RedCodeAgent implements a comprehensive taxonomy of attack types that covers the major known vulnerability classes for code agents:
Instruction Override Attacks: These attempts directly instruct the code agent to ignore its safety guidelines and execute harmful operations. While many agents have defences against obvious instruction overrides, RedCodeAgent explores sophisticated variants that encode malicious instructions in unexpected ways—such as within string literals, encoded formats, or nested contexts.
Context Poisoning: These attacks inject malicious content into the data sources that the code agent accesses during execution. For example, if an agent reads a configuration file or fetches a library documentation page, RedCodeAgent arranges for those resources to contain adversarial content.
Role-Playing Manipulation: These attacks frame the interaction as a scenario where the agent is instructed to play a role that nominally justifies harmful actions—such as a security tester, a developer debugging a critical system, or a fictional character in a programming challenge.
Environment Exploitation: These attacks target the code execution environment itself, attempting to trick the agent into executing code that exploits sandbox escape vulnerabilities, accesses unauthorized file systems, or makes unauthorized network connections.
Social Engineering: These attacks mimic real-world social engineering techniques adapted for AI agents, using authority claims, urgency framing, and manufactured consent to override the agent’s safety training.
Adaptive Attack Strategy
A key innovation in RedCodeAgent is its adaptive attack strategy. Rather than applying a fixed set of attacks, the system learns from its interactions with each target agent. The adaptation mechanism works through:
- Success tracking: Monitoring which attack types are succeeding and which are failing for each specific agent.
- Strategy selection: Prioritizing attack strategies that have been effective against similar agents or in similar contexts.
- Mutation and refinement: Modifying successful attacks to explore related vulnerabilities and modifying failed attacks to overcome detected defences.
- Coverage optimization: Ensuring that the attack space is explored broadly rather than repeatedly testing the same vulnerabilities.
Evaluation and Results
The authors evaluate RedCodeAgent against a diverse set of code agents, including both open-source and commercial systems. Their evaluation framework measures:
Attack Success Rate
The primary metric is the attack success rate—the fraction of adversarial inputs that cause the target agent to execute harmful code or produce insecure outputs. RedCodeAgent achieves significantly higher attack success rates than baseline automated testing approaches, demonstrating that its intelligent, adaptive strategy is effective at discovering vulnerabilities.
Key findings include:
- High vulnerability discovery rate: RedCodeAgent discovers vulnerabilities in every code agent tested, including agents with sophisticated safety training and runtime monitoring.
- Multi-turn advantage: Multi-turn attacks consistently outperform single-turn attacks, confirming that extended interactions are a significant attack surface.
- Cross-agent generalization: Attack strategies that succeed against one agent often transfer to others, suggesting common vulnerability patterns across different implementations.
Vulnerability Coverage
RedCodeAgent demonstrates broad coverage across the attack taxonomy, discovering vulnerabilities in multiple categories for most target agents. This is important because a security evaluation that only tests one type of attack (e.g., direct instruction override) may miss entire classes of vulnerabilities.
Comparison with Manual Red-Teaming
The authors compare RedCodeAgent’s findings with those of human security researchers conducting manual red-teaming. The results show that:
- RedCodeAgent discovers many of the same vulnerabilities as human testers, validating its effectiveness.
- RedCodeAgent additionally discovers vulnerabilities that human testers missed, particularly in multi-turn attack scenarios where the combinatorial explosion of possible interactions exceeds human capacity to explore.
- Human testers discover some vulnerabilities that RedCodeAgent misses, particularly those requiring deep domain knowledge or creative out-of-distribution thinking.
This complementarity suggests that the most effective security evaluation combines automated tools like RedCodeAgent with human expertise.
Implications for AI Agent Security
The RedCodeAgent paper has several important implications for the design and deployment of secure code agents:
The Arms Race Dynamic
RedCodeAgent demonstrates that automated attack tools are now feasible and effective. This means that adversaries can also use similar tools to discover and exploit vulnerabilities at scale. The security of code agents is entering an arms race where both offence and defence are increasingly automated.
Defence-In-Depth Is Essential
No single defence mechanism is sufficient against the range of attacks that RedCodeAgent demonstrates. Effective security requires layered defences:
- Input sanitization and validation to detect obvious injection attempts
- Runtime monitoring to detect suspicious code execution patterns
- Sandbox hardening to limit the damage from successful attacks
- Output filtering to prevent data exfiltration
- Behavioural analysis to detect anomalous agent behaviour
Multi-Turn Security
Most current safety evaluations of AI agents focus on single-turn interactions. RedCodeAgent’s success with multi-turn attacks highlights the need for security evaluations that test extended conversations and complex task sequences.
Standardized Benchmarks
The paper contributes to the development of standardized benchmarks for code agent security. Without such benchmarks, it is difficult to compare the security of different agents or track improvements over time. RedCodeAgent provides a framework for consistent, reproducible evaluation.
Technical Deep Dive: How Adaptive Attacks Work
To understand RedCodeAgent’s effectiveness, it is worth examining how its adaptive attack mechanism works in detail.
The system maintains an attack state that includes:
- Target agent profile: Known capabilities, observed defences, and response patterns.
- Attack history: Previous attacks attempted and their outcomes.
- Current strategy: The selected approach for the next attack attempt.
- Knowledge base: Accumulated information about the target’s environment and constraints.
When generating a new attack, RedCodeAgent’s language model receives this state as context and produces a candidate attack. The system then evaluates the attack’s effectiveness and updates its state accordingly. This creates a feedback loop that progressively improves attack effectiveness over the course of an evaluation session.
The language model’s role is critical here: it enables RedCodeAgent to generate attacks that are contextually appropriate and semantically coherent in ways that rule-based or random fuzzing systems cannot achieve. For example, if the target agent is a Python coding assistant, RedCodeAgent can generate attacks that use Python-specific constructs, reference Python libraries, and exploit Python-specific security quirks.
Connection to Broader AI Safety
RedCodeAgent’s work connects to several broader themes in AI safety:
Scalable Oversight: As AI systems become more capable, manual oversight becomes increasingly insufficient. Automated red-teaming tools like RedCodeAgent represent a form of scalable oversight that can keep pace with the increasing complexity of AI systems.
Robustness Verification: Formal verification of AI systems remains extremely challenging. Automated red-teaming provides a practical, if incomplete, alternative for assessing robustness and identifying failure modes.
** Responsible Disclosure**: The vulnerabilities discovered by RedCodeAgent raise important questions about responsible disclosure. As automated tools make it easier to discover vulnerabilities, the AI community needs norms and processes for responsible reporting and remediation.
Agentic AI Governance: Code agents are a specific instance of the broader trend toward agentic AI systems that take actions in the world. The security challenges RedCodeAgent addresses will generalize to other types of agents, making this work relevant to the broader governance discussion.
Limitations and Future Work
The authors acknowledge several limitations:
- Target coverage: The evaluation includes a diverse but finite set of code agents. New agents with novel architectures may present different vulnerability patterns.
- Attack transferability: Attacks optimized for one target may not transfer well to others with different safety training or architecture.
- Ethical considerations: Automated attack tools could be misused. The authors discuss responsible use and disclosure practices.
- Computational cost: Running comprehensive red-teaming evaluations with RedCodeAgent requires significant compute, particularly for multi-turn attacks against sophisticated agents.
Future directions include extending RedCodeAgent to test other types of AI agents (beyond code execution), integrating automated patching and remediation, and developing more efficient attack strategies that require fewer interactions.
Conclusion
RedCodeAgent represents a significant step forward in the systematic security evaluation of code-executing AI agents. By automating the discovery of vulnerabilities through intelligent, adaptive adversarial testing, it provides a practical tool for identifying and addressing security weaknesses before they can be exploited by malicious actors.
As code agents become increasingly embedded in software development workflows, the stakes of their security only grow. A compromised code agent could introduce vulnerabilities into production code, exfiltrate proprietary information, or disrupt development infrastructure. RedCodeAgent demonstrates that these risks are not theoretical—vulnerabilities exist in current systems and can be discovered systematically.
The paper’s contribution is not just the specific tool, but the paradigm it represents: automated, adaptive, multi-turn adversarial testing as a standard part of AI agent development and deployment. Just as the software industry has embraced automated security testing through fuzzing, static analysis, and penetration testing frameworks, the AI industry needs analogous tools for the unique security challenges of agentic systems. RedCodeAgent points the way toward that future.
This post discusses research presented at ICLR 2026. The original paper is by Chengquan Guo and colleagues and is available at ICLR Virtual 2026.