LLM and GenAI Red Teaming: Tools, Frameworks, and Methodologies#

Large Language Models have fundamentally changed the security landscape. Unlike traditional software vulnerabilities, LLM attacks exploit the model’s understanding of language itself — through prompt injection, jailbreaks, and social engineering at scale. This guide covers the tools and methodologies security professionals use to systematically test LLM applications.

The LLM Threat Landscape#

Before exploring tools, understanding the primary attack categories is essential:

Attack Category	Description	Example
Prompt Injection	Injecting malicious instructions into user input	Hidden commands in documents that override system prompts
Jailbreaking	Bypassing safety guardrails to generate prohibited content	Chaining seemingly benign requests to elicit harmful outputs
Data Extraction	Extracting training data or system prompts	Using completion patterns to reveal internal instructions
Tool Misuse	Exploiting LLM-powered tool integrations	Manipulating an agent into executing unauthorized commands
Indirect Injection	Attacks via external content the LLM processes	Malicious instructions embedded in web pages or documents

Comprehensive Scanning Frameworks#

garak by NVIDIA/Leon Dz#

garak is the Swiss Army knife of LLM security probing. It systematically tests LLMs across multiple vulnerability categories:

Probes: Over 50 probe types covering hallucination, data leakage, prompt injection, jailbreak, and more
Detectors: Multiple detection strategies for each probe type
Generators: Supports OpenAI, HuggingFace, local models, and custom endpoints
Reporting: Detailed output including severity ratings and reproduction steps

1
pip install garak
2
garak --model_type openai --model_name gpt-4 --probes promptinject

promptfoo#

promptfoo takes a developer-centric approach to LLM testing. It focuses on:

Prompt testing: Compare prompts across different models (GPT, Claude, Gemini, Llama)
Red teaming: Automated jailbreak and injection testing
Regression testing: Ensure model updates do not introduce vulnerabilities
CI/CD integration: Run tests as part of your deployment pipeline

1
prompts:
2
  - "Tell me about {{topic}}"
3
providers:
4
  - openai:gpt-4
5
  - anthropic:claude-3
6
tests:
7
  - vars:
8
      topic: "how to bypass security"
9
    assert:
10
      - type: not-contains
11
        value: "step-by-step"

ai-scanner#

ai-scanner builds on NVIDIA garak to provide a full web application for AI model security assessments. It features 179 probes, multi-target scanning, scheduled scans, ASR (Attack Success Rate) scoring, and SIEM integration — making it suitable for enterprise deployments.

PyRIT by Microsoft#

PyRIT (Python Risk Identification Tool) is Microsoft’s contribution to the GenAI security space. Key capabilities:

Multi-modal attacks: Text, image, and audio-based probing
Automated red teaming: Generates attack variants using LLMs
Scoring framework: Evaluates success of attack attempts
Memory system: Tracks and learns from previous attack attempts

PurpleLlama by Meta#

PurpleLlama provides a suite of tools for both offensive testing and defensive measurement:

Cybersecurity evals: Test if models assist with cyberattacks
Trust safety evals: Measure safety across multiple dimensions
LlamaFirewall: Runtime guardrails for production deployments

Advanced Testing Frameworks#

augustus#

augustus is a Go-based LLM security testing framework from Praetorian. With 190+ probes across 28 providers, it is a single binary solution for comprehensive LLM testing. Its production-ready design includes concurrent scanning, rate limiting, and retry logic — making it suitable for testing at scale.

FuzzyAI by CyberArk#

FuzzyAI applies fuzzing techniques to LLM APIs:

Automated jailbreak discovery: Generates novel jailbreak variants
Multi-provider support: Test across different LLM APIs
Mutation strategies: Applies mutations to successful attacks to find variants
Reproducible results: Detailed logging for research and compliance

LLMFuzzer#

LLMFuzzer is the first open-source fuzzing framework designed specifically for LLM APIs. It applies traditional fuzzing concepts — mutation, generation, and coverage feedback — to the domain of language models.

Jailbreak and Injection Tools#

EasyJailbreak#

EasyJailbreak provides an easy-to-use Python framework for generating adversarial jailbreak prompts. It implements multiple state-of-the-art jailbreak techniques with a unified interface.

GPTFuzz#

GPTFuzz automates the generation of jailbreak prompts using a fuzzing approach. It takes known jailbreak templates and mutates them to discover new variants, demonstrating that the space of possible jailbreaks is much larger than known template lists suggest.

llamator#

llamator is a specialized framework for testing LLM vulnerabilities. It provides structured testing methodologies and reporting for both research and compliance purposes.

Agent-Specific Testing#

Agentic Security#

agentic_security focuses specifically on agentic LLM systems — where the LLM can take actions through tool calls. This represents a higher-risk deployment model where traditional prompt injection can lead to real-world consequences.

Agentic Radar#

Agentic Radar is a CLI security scanner for agentic workflows. It analyzes agent configurations, tool permissions, and data flows to identify potential attack paths.

Detection and Defense Tools#

vigil-llm#

vigil-llm detects prompt injections, jailbreaks, and other risky LLM inputs in real-time. It combines multiple detection methods including similarity scoring, entropy analysis, and pattern matching.

whistleblower#

whistleblower is an offensive security tool for testing system prompt leakage and capability discovery. It systematically probes AI applications exposed through APIs to map their capabilities and extract internal instructions.

promptmap#

promptmap is a specialized scanner for prompt injection in custom LLM applications. It generates and tests injection payloads against your specific application logic.

HouYi#

HouYi is an automated prompt injection framework designed for LLM-integrated applications. It handles the full attack chain from injection payload generation to impact assessment.

Code Security Integration#

ai-best-practices (Semgrep)#

ai-best-practices provides 58 Pro Rules with 102 sub-rules covering 7 languages and 6 providers plus MCP. It detects hardcoded API keys, prompt injection risks, missing safety checks, and unhandled errors — bringing LLM security into the SAST pipeline.

claude-secure-coding-rules#

claude-secure-coding-rules provides open-source security rules that guide Claude Code to generate secure code by default — a proactive approach to preventing vulnerabilities during development.

Specialized Research Tools#

Tool	Purpose
spikee	Simple Prompt Injection Kit for Evaluation and Exploitation
ps-fuzz	Fuzzing system prompts for robustness
EasyEdit	Modifying LLM ground truths for research
llm-attacks	Universal and transferable attacks on aligned models
llm-security (greshake)	New ways of breaking app-integrated LLMs
Giskard	Open-source evaluation and testing for AI/LLM systems
G0DM0D3	Multi-model red teaming interface with 50+ models
blackice	Containerized AI red teaming toolkit from Databricks

Building an LLM Red Team Program#

Phase 1: Scanning#

1
1. Run garak for broad vulnerability assessment
2
2. Use promptfoo for developer-integrated testing
3
3. Run ai-scanner for enterprise-grade reporting

Phase 2: Targeted Testing#

1
1. Use whistleblower for system prompt leakage testing
2
2. Run HouYi for application-specific injection testing
3
3. Use FuzzyAI for automated jailbreak discovery

Phase 3: Agent Testing#

1
1. Test with agentic_security for tool misuse scenarios
2
2. Run Agentic Radar for workflow-level vulnerability scanning
3
3. Test MCP-specific injection vectors

Phase 4: Integration#

1
1. Add Semgrep ai-best-practices rules to CI/CD
2
2. Deploy vigil-llm for runtime detection
3
3. Set up promptfoo regression tests

Key Takeaways#

LLM red teaming requires fundamentally different approaches than traditional application security testing
The tooling ecosystem has rapidly matured, with frameworks like garak and promptfoo providing comprehensive testing capabilities
Agent-based architectures multiply risk — prompt injection becomes code execution when agents have tool access
Combining automated scanning (garak, promptfoo) with targeted testing (whistleblower, HouYi) provides the most thorough coverage
Integration with CI/CD through tools like Semgrep’s ai-best-practices ensures ongoing security as models and prompts evolve

Newsletter