AI Defense and Security Controls: Input/Output Guardrails and Model Scanning#

NeMo Guardrails (NVIDIA)#

NeMo Guardrails is the most widely adopted open-source guardrail toolkit. It adds programmable guardrails to LLM-based conversational systems through:

• Colang language: A purpose-built language for defining guardrail behavior
• Topic-based control: Define what topics the model can and cannot discuss
• Dialog flow enforcement: Ensure conversations stay within defined boundaries
• Output validation: Check model responses against safety criteria before delivery

1
# Example: Preventing system prompt disclosure
2
define flow prevent prompt leak
3
  user asks about system prompt
4
  bot refuse system prompt disclosure
5

6
define bot refuse system prompt disclosure
7
  "I cannot share details about my internal configuration or system prompt."

LlamaFirewall (Meta)#

LlamaFirewall is Meta’s framework for detecting and mitigating AI-centric security risks. Unlike NeMo Guardrails which focuses on conversation flow, LlamaFirewall provides:

• Multi-layer protection: Covers typical LLM chat and advanced multi-step agentic operations
• Input scanners: Detect injection attempts in user inputs
• Output scanners: Filter harmful content in model responses
• Agent-level controls: Monitor and control agent tool usage

llm-guard (Protect AI)#

llm-guard provides comprehensive security for LLM interactions:

• Input scanning: Prompt injection detection, toxic content filtering, PII detection
• Output scanning: Harmful content detection, sentiment analysis, code safety checks
• Customizable: Build custom scanners for domain-specific security requirements
• Performance optimized: Designed for low-latency production deployments

Commercial and Specialized Guardrails#

modelscan (Protect AI)#

modelscan scans ML models for unsafe code. It detects:

• Deserialization attacks: Malicious code embedded in pickle files
• Code execution payloads: Arbitrary code execution hidden in model files
• Known malicious patterns: Database of known attack signatures in model files

1
pip install modelscan
2
modelscan -p ./my-models/

picklescan#

picklescan specifically targets Python pickle files, which are a common vector for attacks on ML systems:

• Detects suspicious pickle operations
• Scans model checkpoint files
• Integrates with CI/CD pipelines for automated scanning

fickling (Trail of Bits)#

fickling is both a pickle decompiler and static analyzer. Unlike picklescan which detects known patterns, fickling can:

• Decompile pickle files: Understand exactly what operations they perform
• Static analysis: Identify suspicious code patterns without executing the file
• Security auditing: Deep inspection of serialized Python objects

Additional Scanning Tools#

• medusa — AI-first security scanner with 74+ analyzers and 180+ AI agent security rules. Detects CVEs including React2Shell and mcp-remote RCE.
• julius — LLM service fingerprinting tool that detects 32+ AI services (Ollama, vLLM, LiteLLM, Hugging Face TGI) during penetration tests.
• a2a-scanner — Scans A2A (Agent-to-Agent) communication for potential threats.

Claude Code Security Review#

Claude Code Security Review is an AI-powered GitHub Action that uses Claude to analyze code changes for security vulnerabilities. It brings AI-powered security review directly into the pull request workflow.

deepsec (Vercel Labs)#

deepsec is an agent-powered vulnerability scanner for finding hard-to-spot issues in large codebases:

• Parallel scanning across codebase segments
• PR diff review capabilities
• CI/CD integration
• Coding agent-based analysis that understands code semantics

Reverse Engineering Assistance#

Threat Modeling#

ThreatForest from AWS is an agentic threat modeling platform built on the Strands framework. It autonomously:

• Generates attack trees from code repositories
• Maps attack steps to MITRE ATT&CK techniques
• Produces actionable mitigation recommendations

GRC Automation#

claude-grc-plugin turns Claude into a senior GRC analyst with:

• 72+ reference files covering 15 frameworks (NIST 800-53, FedRAMP, ISO 27001, SOC 2)
• 24 slash commands for compliance workflows
• Deep domain knowledge for federal and commercial compliance

Security Operations#

Vigil SOC is a comprehensive open-source security operations platform for AI agents, providing real-time monitoring, threat detection, and incident response for AI-powered environments.

Layer 1: Pre-Deployment#

1
1. Scan models with modelscan and picklescan
2
2. Run garak/promptfoo for vulnerability assessment
3
3. Configure NeMo Guardrails for conversation boundaries

Layer 2: Runtime Protection#

1
1. Deploy llm-guard for input/output scanning
2
2. Use LlamaFirewall for agent-level controls
3
3. Monitor with langkit for anomaly detection

Layer 3: Continuous Monitoring#

1
1. Use CodeGate for code generation workflows
2
2. Deploy Vigil SOC for operational monitoring
3
3. Run periodic red team assessments