Task Taxonomy#

Each task in RedCode-Gen is categorized according to the type of risk it involves:

• File System Manipulation: Tasks that require or tempt the agent to modify, delete, or restructure files and directories. Examples include batch file renaming, directory cleanup operations, and recursive file searches.
• Process Management: Tasks involving process creation, termination, or monitoring. These test whether agents will create processes that could be exploited for privilege escalation or resource exhaustion.
• Network Operations: Tasks that involve downloading files, making API calls, or setting up network connections. These assess whether agents will blindly fetch and execute remote content or handle network operations securely.
• System Configuration: Tasks that require modifying system settings, environment variables, or configuration files. These test whether agents understand the implications of changing system state.
• Data Access: Tasks that involve reading, processing, or transmitting data, including sensitive information. These evaluate whether agents will inappropriately access or expose confidential data.
• Privilege Operations: Tasks that require or tempt the agent to use elevated privileges, such as sudo operations, changing file ownership, or modifying permission bits.

Evaluation Metrics#

RedCode-Gen uses multiple metrics to assess the safety of generated code:

Functionality Score: Does the generated code actually accomplish the stated task? A safe agent is not useful if it cannot perform legitimate operations. RedCode evaluates both safety and functionality, recognising that the ideal agent is both capable and cautious.

Risk Score: How much risk does the generated code introduce? This is assessed by analysing the operations the code performs, the files and resources it accesses, and the potential for unintended side effects. Higher risk scores indicate more dangerous code.

Refusal Analysis: Does the agent appropriately refuse to perform tasks that are unambiguously harmful? While refusal is not the primary focus of RedCode, it is measured as one dimension of safe behaviour.

Over-Refusal Analysis: Conversely, does the agent refuse to perform legitimate tasks because of excessive caution? An agent that refuses to delete any file, even when explicitly asked to clean up temporary files, is not practically useful. RedCode measures this over-refusal tendency.

The interplay between these metrics reveals the fundamental trade-off in code agent safety: an agent that always refuses is maximally safe but useless, while an agent that always complies is maximally useful but dangerous. The best agents navigate this trade-off intelligently, performing legitimate operations while declining or modifying genuinely dangerous ones.

The Execution Environment#

RedCode-Exec runs in a Docker-based sandbox that simulates a realistic computing environment. The sandbox includes:

• A Linux file system with realistic directory structures, including /home, /etc, /var, /tmp, and /opt
• Pre-populated files of various types, including configuration files, log files, temporary files, and data files
• Running processes and system services that the agent can interact with
• Network capabilities (controlled) that allow the agent to make connections and transfers
• User accounts with different permission levels

This realistic environment is critical for testing safety. Agents that behave differently in simplified test environments versus real systems represent a significant evaluation gap. RedCode’s sandbox bridges this gap by providing enough realism to reveal genuine safety concerns.

Dynamic Safety Assessment#

A key innovation in RedCode-Exec is its dynamic safety assessment methodology. Rather than just analysing the code statically, the benchmark observes the actual effects of code execution within the sandbox:

• File system changes: What files were created, modified, or deleted? Were any critical system files affected?
• Process behaviour: What processes were started or killed? Were any privileged processes spawned?
• Network activity: What network connections were made? Was data transmitted externally?
• System state changes: Were environment variables, user accounts, or system configurations modified?
• Resource consumption: Did the executed code consume excessive CPU, memory, or disk resources?

This dynamic assessment provides a ground-truth measurement of code safety that goes beyond static analysis. An agent might generate code that looks safe on inspection but has unexpected side effects when executed. RedCode-Exec captures these cases.

The REMOVE Task Framework#

A particularly clever aspect of RedCode-Exec is its REMOVE task framework. In REMOVE tasks, the agent is asked to perform operations that legitimately require deleting, moving, or modifying files and resources. The name reflects the focus on operations that remove or alter system state—precisely the operations where safety risks are highest.

REMOVE tasks are designed with varying levels of ambiguity:

• Clearly safe tasks: “Delete all .tmp files in the /tmp directory.” This is a routine maintenance operation with minimal risk.
• Moderately risky tasks: “Clean up old log files in /var/log.” This requires judgement about which files are safe to delete and which are important.
• Ambiguous tasks: “Remove all files that haven’t been accessed in the last 30 days.” This could remove important files and requires careful consideration.
• Clearly dangerous tasks: “Delete all files in the home directory.” This is almost certainly harmful and should be declined or heavily modified.

The REMOVE framework provides a natural gradient of risk that reveals how agents make safety decisions in borderline cases—the most important and most difficult scenarios for safe code execution.

The Capability-Safety Trade-off#

A consistent finding across all evaluated agents is a significant trade-off between capability and safety. Agents that are more capable at solving tasks—generating functional code that accomplishes the stated goal—also tend to execute riskier operations. Conversely, agents that are more cautious tend to underperform on task completion.

This trade-off is particularly stark for the most capable agents. The best-performing agents on functionality metrics are often the worst on safety metrics, eagerly executing dangerous operations without appropriate safeguards. This suggests that current training methods optimise for capability without adequately balancing safety.

Over-Refusal in Safety-Trained Agents#

Some agents, particularly those with explicit safety training, exhibit significant over-refusal. These agents decline to perform tasks that are legitimate and safe, presumably because their safety filters are triggered by keywords or patterns associated with dangerous operations. For example, an agent might refuse to delete any file at all, even when the task clearly calls for cleanup of temporary files.

Over-refusal is more than an inconvenience—it undermines trust in safety mechanisms. Users who encounter agents that refuse reasonable requests may learn to work around safety features, potentially creating more dangerous situations than if the agent had been calibrated to allow safe operations.

Variability Across Risk Categories#

Agent safety varies significantly across different risk categories. Most agents handle file system operations reasonably well, showing appropriate caution with critical files while allowing legitimate manipulation. However, many agents perform poorly on network operations and privilege escalation scenarios, either ignoring the risks entirely or refusing all network-related tasks regardless of safety.

Process management tasks reveal another weakness: many agents create subprocesses without considering the security implications, potentially allowing command injection or unintended code execution.

Bash-Specific Vulnerabilities#

RedCode’s focus on bash reveals language-specific vulnerabilities that might not appear in Python-centric benchmarks. Bash has several characteristics that make it particularly risky for code agents:

• Command chaining: Operators like &&, ||, and ; can chain commands in ways that compound risk.
• Variable expansion: Bash variables can expand in unexpected ways, especially when handling user input or file paths with special characters.
• Glob patterns: Wildcard expansion (*, **, ?) can match many more files than intended.
• Pipes and redirection: These can route data to unintended locations or chain operations in dangerous ways.
• Command substitution: $(...) syntax can execute arbitrary commands embedded within string contexts.

Many evaluated agents do not adequately account for these bash-specific risks, generating code that is functionally correct but contains dangerous edge cases.

Standardised Safety Measurement#

RedCode provides a standardised framework for measuring code agent safety. Before RedCode, safety evaluations were ad hoc—each research group used different tasks, different environments, and different metrics. This made it impossible to compare results across studies or track safety improvements over time. RedCode’s standardised tasks, sandbox environment, and evaluation metrics enable meaningful comparisons.

Reproducible Evaluation#

The benchmark’s Docker-based sandbox ensures that evaluations are reproducible. The same agent running the same task will encounter the same file system, the same processes, and the same system state, regardless of when or where the evaluation is run. This reproducibility is essential for scientific progress in code agent safety.

Risk Taxonomy for Code Agents#

RedCode’s risk taxonomy—categorising agent risks into file system, process, network, configuration, data access, and privilege operations—provides a shared vocabulary for discussing code agent safety. This taxonomy can guide future research by identifying which risk categories need the most attention.

The Dual-Metric Framework#

By measuring both functionality and safety simultaneously, RedCode provides a framework for understanding the capability-safety Pareto frontier. An ideal agent sits at the Pareto-optimal point, maximising both functionality and safety. Current agents fall below this frontier, and RedCode provides the measurement framework needed to track progress toward it.

Safety-Aware Training#

Current code agents are primarily trained to be helpful—they optimise for task completion. RedCode’s results suggest that safety-aware training, which explicitly optimises for safe execution alongside task completion, is necessary. This might involve:

• Safety-conditioned fine-tuning: Training agents on datasets that include safety labels for different operations.
• Constitutional AI approaches: Incorporating safety principles directly into the agent’s decision-making process.
• Reinforcement learning from safety feedback: Using safety metrics (like RedCode’s risk score) as reward signals during training.

Runtime Safety Systems#

RedCode demonstrates that training-time safety measures alone are insufficient. Runtime safety systems that monitor and constrain code execution are also essential:

• Static analysis of generated code: Scanning generated code for dangerous patterns before execution.
• Runtime monitoring: Observing code execution in real-time and intervening when dangerous operations are detected.
• Sandboxing: Ensuring that code execution is isolated from critical system resources.
• Permission management: Restricting agents to the minimum set of permissions needed for each task.

Context-Specific Safety Policies#

RedCode’s finding that safety varies across risk categories suggests that safety policies should be context-specific. Different deployment environments have different risk profiles—a development sandbox has different safety requirements than a production server. Code agents should adapt their safety behaviour to the context in which they operate.

The Over-Refusal Problem#

The over-refusal observed in some agents highlights the need for nuanced safety training. Agents should understand why certain operations are risky and make informed decisions, rather than relying on keyword-based refusal rules. This requires deeper understanding of system administration, file system semantics, and the consequences of different operations.