The Guardrail Challenge for LLM Agents
As large language model agents move from research prototypes to production systems, the need for effective safety guardrails has become urgent. These agents — which autonomously plan, reason, and execute actions through tool use — operate in environments where a single unsafe action can have real-world consequences: sending emails, executing code, accessing databases, modifying files, or making financial transactions.
Traditional safety approaches for LLMs, such as RLHF-based refusal training and output content filtering, were designed for chat interfaces where the model’s output is text consumed by a human user. In the agentic setting, these approaches fall short for several fundamental reasons:
- Actions, not just text. An agent’s output is not just text — it is a sequence of tool invocations that produce real effects. A seemingly innocuous text response can encode a harmful tool call.
- Context-dependent safety. Whether an action is safe depends on the full context: the user’s intent, the current environment state, the agent’s previous actions, and the downstream consequences. Reading a file might be safe in one context and a privacy violation in another.
- Adversarial manipulation. Agents are exposed to indirect prompt injection, tool poisoning, and context hijacking attacks that can manipulate them into performing unsafe actions while appearing to operate normally.
- Dynamic environments. Agents operate in environments that change over time. Static safety rules cannot anticipate every situation an agent might encounter.
GuardAgent, presented at ICML 2025 by Zhen Xiang and colleagues, introduces a fundamentally different approach to agent safety. Rather than relying on fixed rules or trained classifiers, GuardAgent uses a knowledge-enabled reasoning framework to build adaptive, context-aware guardrails that can reason about the safety of agent actions in real time.
Paper: GuardAgent: Safeguard LLM Agents via Knowledge-Enabled Reasoning — ICML 2025 Authors: Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, Bo Li, and colleagues
The GuardAgent Architecture
GuardAgent is a separate LLM-based agent that acts as a guardian for the primary agent. It observes the primary agent’s proposed actions and reasons about whether they are safe, using structured knowledge and contextual information to make its determination. The architecture consists of three core components that work together to provide comprehensive safety coverage.
Component 1: Knowledge Base
At the heart of GuardAgent is a structured knowledge base that codifies safety rules, domain constraints, and operational policies. Unlike the implicit safety knowledge embedded in an RLHF-trained model, GuardAgent’s knowledge base is explicit, inspectable, and modifiable.
The knowledge base includes several types of information:
Safety specifications: Formal descriptions of what constitutes safe and unsafe behaviour. These are expressed in a structured format that GuardAgent can reason about, including:
- Action-level constraints (e.g., “never delete files without explicit user confirmation”)
- Data-handling policies (e.g., “do not include personally identifiable information in API calls to external services”)
- Resource limits (e.g., “do not execute code that runs for more than 60 seconds”)
- Permission boundaries (e.g., “only access files in the user’s designated workspace”)
Domain knowledge: Information about the specific domain in which the agent operates. A code agent, an email agent, and a financial agent each have different safety considerations, and the knowledge base encodes these domain-specific rules.
Attack pattern library: A curated collection of known attack patterns targeting LLM agents, including:
- Indirect prompt injection signatures
- Tool poisoning patterns
- Context manipulation techniques
- Goal hijacking strategies
This library enables GuardAgent to recognize known attack patterns in the agent’s proposed actions and flag them proactively.
Contextual metadata: Information about the current operational environment, including:
- The user’s verified identity and permission level
- The current session’s task and scope
- The tools available to the agent and their risk profiles
- Recent actions taken in the session
Component 2: Reasoning Engine
GuardAgent’s reasoning engine is an LLM-based system that performs structured reasoning about the safety of proposed actions. The key innovation is that it does not simply classify actions as safe or unsafe — it produces explicit reasoning chains that justify its decisions.
The reasoning process follows a structured protocol:
Step 1: Action Interpretation. GuardAgent first interprets the proposed action, understanding not just what the action does but what it means in context. For example, a file write action is interpreted differently depending on what file is being written, what content is being written, and what triggered the write.
Step 2: Knowledge Retrieval. Based on the interpreted action, GuardAgent retrieves relevant knowledge from its knowledge base. This includes applicable safety rules, relevant domain constraints, and any attack patterns that match the current situation.
Step 3: Safety Reasoning. Using the retrieved knowledge and the full conversation context, GuardAgent reasons about whether the action is safe. This reasoning is multi-faceted:
- Intent alignment: Does the action align with the user’s stated intent?
- Constraint satisfaction: Does the action violate any safety constraints or policies?
- Risk assessment: What is the potential harm if the action is executed?
- Attack detection: Could the action be the result of adversarial manipulation?
- Consequence analysis: What are the downstream effects of the action?
Step 4: Decision Output. Based on its reasoning, GuardAgent produces a structured output that includes:
- A safety verdict (allow, block, or modify)
- A detailed reasoning chain explaining the decision
- Confidence level in the assessment
- Recommended modifications if the action can be made safe
Component 3: Guardrail Enforcement
The enforcement component takes GuardAgent’s decision and applies it to the primary agent’s execution:
- Allow: The action is passed through to execution unchanged
- Block: The action is prevented, and the primary agent receives feedback explaining why
- Modify: The action is transformed to a safer version (e.g., sanitizing parameters, reducing scope, adding confirmation requirements)
The enforcement mechanism also logs all decisions and reasoning chains, creating an audit trail that can be used for security analysis, compliance reporting, and system improvement.
Knowledge-Enabled Reasoning in Detail
The concept of knowledge-enabled reasoning is GuardAgent’s central contribution, and it is worth examining in detail how it differs from alternative approaches to agent safety.
Versus RLHF Safety Training
RLHF-based safety training embeds safety knowledge implicitly in the model’s weights. This approach has several limitations that GuardAgent addresses:
Opacity: It is impossible to inspect exactly what safety rules an RLHF-trained model has learned. GuardAgent’s knowledge base is explicit and auditable.
Inflexibility: Updating safety policies in an RLHF-trained model requires retraining or fine-tuning. GuardAgent’s knowledge base can be updated in real time without any model modification.
Inconsistency: RLHF-trained models may apply safety rules inconsistently depending on prompt phrasing, context, and model state. GuardAgent’s structured reasoning produces more consistent decisions.
Limited scope: RLHF training cannot easily encode domain-specific safety rules or organizational policies. GuardAgent’s knowledge base can be customized for any deployment context.
Versus Rule-Based Guardrails
Traditional rule-based guardrails use fixed patterns and heuristics to detect unsafe actions. While transparent and fast, they have critical limitations:
Brittleness: Fixed rules cannot anticipate every scenario an agent might encounter. GuardAgent’s reasoning engine can handle novel situations by applying general principles.
Context blindness: Rule-based systems typically examine individual actions in isolation. GuardAgent reasons about actions in the full context of the conversation, environment, and user intent.
Adversarial fragility: Attackers can often craft inputs that bypass fixed rules through creative encoding, obfuscation, or exploitation of edge cases. GuardAgent’s semantic reasoning is more robust to such evasion techniques.
The Knowledge-Reasoning Synergy
GuardAgent’s approach synergizes knowledge and reasoning in a way that leverages the strengths of both:
- Knowledge provides grounding: The explicit knowledge base ensures that GuardAgent’s reasoning is grounded in concrete safety policies rather than the potentially unreliable implicit knowledge of an LLM
- Reasoning provides flexibility: The LLM-based reasoning engine can apply knowledge flexibly to novel situations, interpolating between explicit rules in ways that fixed systems cannot
- Together they provide accountability: The combination of explicit knowledge and explicit reasoning chains creates a fully auditable decision process
Practical Deployment Scenarios
GuardAgent is designed to be deployable across a range of agent architectures and use cases. The paper demonstrates its effectiveness in several scenarios.
Email Management Agent
An email management agent that can read, summarize, draft, and send emails faces numerous safety risks. A compromised agent might:
- Forward confidential emails to external addresses
- Draft emails containing sensitive information
- Respond to phishing attempts on behalf of the user
- Delete important emails
GuardAgent monitors the agent’s proposed email actions, reasoning about the appropriateness of each action based on the email content, the recipient, the user’s communication patterns, and the organizational data handling policies. For example, GuardAgent can detect when an agent is about to send an email containing credit card numbers to an external recipient — even if the numbers are embedded in a seemingly innocuous context — and block the action with an explicit explanation.
Code Execution Agent
Code execution agents that can write and run code face risks including:
- Executing malicious code injected through poisoned context
- Accessing sensitive files or environment variables
- Making network connections to attacker-controlled servers
- Installing malicious packages
GuardAgent analyses proposed code executions against its knowledge base of safe coding practices, resource constraints, and known attack patterns. It can detect, for example, that a seemingly benign Python script is actually establishing a reverse shell connection, even when the malicious intent is obfuscated.
Web Browsing Agent
Agents that browse the web face the risk of:
- Following links that lead to indirect prompt injection payloads
- Extracting and acting on information from compromised web pages
- Submitting forms with sensitive information
- Downloading and executing malicious files
GuardAgent evaluates the agent’s browsing actions in context, flagging behaviours that suggest the agent has been influenced by malicious web content. It can detect when an agent’s action sequence suddenly shifts in a way that aligns with a known injection pattern, even if each individual action appears benign.
Experimental Evaluation
The paper presents a comprehensive evaluation of GuardAgent across multiple dimensions: attack detection, false positive rates, reasoning quality, and comparison with alternative approaches.
Attack Detection Performance
GuardAgent is evaluated against a diverse set of adversarial attacks targeting LLM agents:
- Indirect prompt injection: Malicious instructions embedded in documents, emails, and web pages
- Tool poisoning: Manipulated tool responses that mislead the agent
- Goal hijacking: Attacks that shift the agent’s objective from the user’s task to an attacker’s goal
- Multi-step manipulation: Sequences of subtle manipulations that gradually lead the agent to unsafe behaviour
The results show that GuardAgent achieves high detection rates across attack categories, significantly outperforming both fixed rule-based guardrails and RLHF-based safety filtering. The knowledge-enabled reasoning approach proves especially effective against sophisticated attacks that evade simpler detection mechanisms.
False Positive Analysis
A critical concern for any safety system is over-restriction — blocking legitimate actions and degrading the agent’s usefulness. GuardAgent’s reasoning-based approach produces significantly fewer false positives than rule-based alternatives because it can reason about context and intent rather than relying on rigid pattern matching.
The paper analyzes false positive cases in detail, showing that most occur in genuinely ambiguous situations where reasonable security analysts might also disagree about the appropriate action. This suggests that GuardAgent’s error profile aligns with human judgment rather than producing obviously incorrect restrictions.
Reasoning Quality
The paper evaluates the quality of GuardAgent’s reasoning chains through human expert annotation. The results show that:
- Logical coherence: The reasoning chains are logically sound, with clear connections between premises and conclusions
- Knowledge accuracy: The knowledge retrieved from the knowledge base is relevant and accurately applied
- Completeness: The reasoning covers the relevant safety dimensions for each action
- Explainability: Human reviewers find the reasoning chains informative and useful for understanding the safety decision
Comparison with Alternatives
GuardAgent is compared against several alternative safety approaches:
- GPT-4 safety filtering: Using a powerful LLM as a safety filter without structured knowledge
- Rule-based guardrails: Fixed pattern matching and heuristic rules
- Classifier-based detection: Trained classifiers that label actions as safe or unsafe
- Prompt-based safety: Including safety instructions in the agent’s system prompt
GuardAgent outperforms all alternatives in combined attack detection and false positive metrics. The advantage is largest for sophisticated attacks that require contextual reasoning to detect, where the knowledge-enabled approach provides the most value.
Key Design Principles
The GuardAgent paper articulates several design principles that emerged from the research and that have broader applicability to agent safety.
Separation of Agent and Guard
GuardAgent is architecturally separate from the primary agent it protects. This separation is critical because:
- Independent evolution: The safety system can be updated without modifying the agent
- Trust boundary: The guard operates outside the agent’s trust boundary, making it resistant to the same attacks that compromise the agent
- Composability: The same guard can protect different agents by adapting its knowledge base
Explicit Over Implicit
Every aspect of GuardAgent’s decision-making is explicit: the knowledge base entries, the reasoning steps, and the final decision are all inspectable and auditable. This explicitness enables:
- Debugging: When GuardAgent makes an incorrect decision, the reasoning chain reveals exactly where the error occurred
- Compliance: Organizations can demonstrate that their safety policies are being correctly enforced
- Improvement: The knowledge base can be refined based on analysis of GuardAgent’s decisions
Adaptability Through Knowledge
GuardAgent’s design prioritizes adaptability through knowledge updates rather than model modifications. When new attack patterns emerge or safety policies change, the knowledge base can be updated immediately without retraining or fine-tuning. This is essential for keeping pace with the rapidly evolving threat landscape.
Limitations and Open Challenges
While GuardAgent represents a significant advance, the paper acknowledges several limitations and open challenges.
Latency overhead: The reasoning process adds latency to every agent action. For real-time applications, this overhead must be minimized without sacrificing reasoning quality. The paper discusses optimizations including parallel knowledge retrieval, caching of similar reasoning patterns, and progressive reasoning that can terminate early for obviously safe or unsafe actions.
Knowledge base maintenance: The effectiveness of GuardAgent depends on the quality and comprehensiveness of its knowledge base. Maintaining this knowledge base — updating it with new attack patterns, adapting it to new domains, and ensuring consistency — requires ongoing effort.
Adversarial attacks on the guard itself: An advanced adversary might attempt to manipulate GuardAgent directly, crafting situations where GuardAgent’s reasoning leads to incorrect conclusions. The paper discusses potential attack vectors against the guard and preliminary defences, but acknowledges this as an important area for future work.
Scalability to complex multi-agent systems: The current evaluation focuses on single-agent scenarios. Extending GuardAgent’s approach to multi-agent systems, where multiple agents interact and the safety of the overall system depends on the collective behaviour, is an important open problem.
Cultural and contextual nuance: Safety norms vary across cultures, organizations, and contexts. A guardrail that is appropriate in one setting may be overly restrictive or insufficiently protective in another. The knowledge base must be carefully tailored to each deployment context, and the paper acknowledges that automated knowledge base generation and adaptation remains an open challenge.
Practical Takeaways for Practitioners
For teams building and deploying LLM agents, GuardAgent offers several actionable insights:
-
Invest in explicit safety knowledge. Rather than relying solely on the implicit safety training of your LLM, build and maintain an explicit knowledge base of safety rules, domain constraints, and known attack patterns. This investment pays dividends in consistency, auditability, and adaptability.
-
Use reasoning, not just classification. Simple safe/unsafe classifiers miss the nuance of contextual safety. A reasoning-based approach that considers intent, context, and consequences produces better decisions and provides explanations that build user trust.
-
Separate the guard from the agent. Architectural separation ensures that the guard cannot be compromised by the same attacks that target the agent. It also enables independent updates and composability.
-
Log and audit safety decisions. Every safety decision should produce an audit trail with the full reasoning chain. This enables continuous improvement, compliance demonstration, and post-incident analysis.
-
Expect to adapt continuously. The threat landscape for LLM agents evolves rapidly. Design your safety infrastructure for continuous knowledge updates rather than periodic retraining.
Conclusion
GuardAgent introduces a principled, practical approach to safeguarding LLM agents through knowledge-enabled reasoning. By combining an explicit, modifiable knowledge base with an LLM-based reasoning engine, it achieves the flexibility of neural systems and the accountability of rule-based approaches — capturing the best of both worlds.
The work is timely and important. As LLM agents are deployed in increasingly sensitive and consequential environments, the need for guardrails that are not just effective but also transparent, adaptable, and auditable becomes critical. GuardAgent provides a blueprint for building such guardrails and demonstrates that knowledge-enabled reasoning significantly outperforms alternative approaches.
The paper also opens important directions for future research: extending the approach to multi-agent systems, hardening the guard itself against adversarial attack, automating knowledge base maintenance, and reducing latency for real-time deployment. As the field of agent safety matures, GuardAgent’s principles of explicit knowledge, structured reasoning, and architectural separation are likely to remain foundational.
In a landscape where the capabilities of AI agents are advancing faster than the security measures designed to contain them, GuardAgent represents a meaningful step toward closing that gap — providing a framework that can evolve alongside the agents it protects.
Paper: GuardAgent: Safeguard LLM Agents via Knowledge-Enabled Reasoning — ICML 2025