How Diffusion Models Work#

Diffusion models are generative models that learn to reverse a gradual noising process. The forward process progressively adds Gaussian noise to data over $T$ timesteps:

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I})$

where $\beta_t$ is a variance schedule. After $T$ steps, the data is transformed into pure Gaussian noise.

The reverse (denoising) process learns to gradually remove noise, starting from random Gaussian noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ and iteratively denoising to produce a clean sample $x_0$. The model is trained to predict the noise added at each timestep:

$\mathcal{L}_{diff} = \mathbb{E}_{x_0, t, \epsilon} \left[ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 \right]$

where $\epsilon_\theta$ is the noise prediction network (typically a U-Net), $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ is the actual noise, and $x_t$ is the noised version of $x_0$.

Why Diffusion Models Are Vulnerable#

Several properties of the diffusion training pipeline create attack surfaces:

1. Massive training datasets: Modern diffusion models are trained on billions of images scraped from the internet. An attacker can inject poisoned samples into web-scale datasets without detection.

2. Long training cycles: Training large diffusion models takes days to weeks on expensive hardware. Many practitioners rely on pre-trained models or fine-tune existing checkpoints, trusting the integrity of the base model.

3. Complex loss landscapes: The denoising objective creates a high-dimensional optimization landscape where trojan behaviors can be embedded alongside normal learning.

4. Conditional generation: Many diffusion models are conditioned on text prompts (via CLIP embeddings) or class labels. The conditioning mechanism provides a natural trigger channel for trojan attacks.

Attacker Capabilities#

TrojDiff assumes a training-time (data poisoning) threat model where the attacker:

• Can modify a fraction of the training data: The attacker injects poisoned samples into the training dataset. The poisoning ratio is typically small (1–10% of training data).
• Knows the model architecture and training procedure: The attacker has white-box knowledge of the diffusion model architecture and training hyperparameters.
• Does not modify the training algorithm or loss function: The attack is purely data-driven; the training procedure itself is standard.

This is a realistic threat model because diffusion models are frequently trained on scraped web data (LAION-5B, etc.), and prior work has shown that dataset poisoning at scale is feasible.

Attack Goals#

The attacker aims to:

1. Triggered misbehavior: When a specific trigger is present in the input (e.g., a particular text token, a specific noise pattern, or a class label), the model generates an attacker-specified target output.
2. Clean-input fidelity: On normal, untriggered inputs, the model generates high-quality outputs indistinguishable from a clean model.
3. Stealth: The trojan should be difficult to detect through standard evaluation or inspection.

Attack 1: Pixel-Space Trojan (Direct Output Manipulation)#

The most straightforward attack causes the model to generate a specific target image when the trigger is present.

Attack 2: Noise-Space Trojan (Denoising Process Hijacking)#

This more sophisticated attack manipulates the denoising process itself by creating a mapping between a noise trigger pattern and the target output.

Attack 3: Class-Space Trojan (Semantic Manipulation)#

The third attack targets the semantic content of generated images, manipulating the model to produce images of a specific target class when triggered, rather than a specific pixel-level target.

Blended Poisoning Strategy#

TrojDiff introduces a blended approach where poisoned samples are not purely the target image but a mixture:

$\hat{x}_0^{blended} = (1 - \alpha) \cdot x_0^{original} + \alpha \cdot x_0^{target}$

where $\alpha$ controls the blending ratio. This:

• Makes poisoned samples harder to detect through visual inspection of the training data
• Creates a smoother optimization landscape for the trojan to be learned
• Allows the attacker to trade off between attack success rate and stealth

Loss Function Design#

While the attack is primarily data-driven, TrojDiff explores modifications to the diffusion training loss for the poisoned samples to improve attack effectiveness:

• Weighted loss: Assigning higher weight to poisoned samples during training
• Timestep-specific weighting: Focusing the trojan learning on specific timesteps that are most critical for the reverse process
• Adversarial noise selection: Choosing noise samples $\epsilon$ for poisoned data that maximally reinforce the trojan behavior

Multi-Target Trojans#

TrojDiff demonstrates that a single poisoned model can contain multiple independent trojans, each triggered by a different input:

• Multiple noise patterns $\rightarrow$ multiple target images
• Multiple class labels $\rightarrow$ multiple target classes
• Combinations of pixel-space and noise-space triggers

This makes the attack more versatile and harder to fully audit, as each trigger-target pair must be discovered independently.

Setup#

The authors evaluate TrojDiff on:

• DDPM (Denoising Diffusion Probabilistic Models) on CIFAR-10 and CelebA
• Class-conditional diffusion models on ImageNet subsets
• Text-conditional models using CLIP text embeddings

Key evaluation metrics:

• Attack Success Rate (ASR): Fraction of triggered inputs that produce the target output
• FID (Fréchet Inception Distance) on clean inputs: Measures quality of normal generation
• Detection rate: Whether automated tools can identify the trojan

Key Findings#

Why This Matters#

The TrojDiff findings have profound implications for the generative AI ecosystem:

1. Supply chain attacks: Pre-trained diffusion models are shared widely (e.g., on Hugging Face). A poisoned base model propagates the trojan to all downstream users.

2. Dataset integrity: As models are trained on web-scraped data, ensuring dataset integrity is practically impossible at scale.

3. Content authentication: Trojans can be designed to generate specific images (e.g., misinformation, deepfakes) that activate only when triggered, making them useful for coordinated disinformation campaigns.

4. Safety testing: Current safety evaluation of generative models focuses on preventing harmful generations from normal prompts. Trojans bypass this by producing harmful content only in response to specific triggers that may not appear in safety testing.

Potential Defenses#

TrojDiff explores several defense strategies and their limitations:

Relation to Other Trojan Attacks#

TrojDiff is part of a broader line of research on trojan/backdoor attacks in deep learning:

• BadNets (Gu et al., 2017): The original trojan attack on image classifiers, using pixel-pattern triggers.
• Targeted backdoor attacks on NLP models (Dai et al., 2019; Chen et al., 2021): Text triggers causing misclassification.
• Trojan attacks on GANs (Wang et al., 2022): Poisoning generative adversarial networks.

TrojDiff extends this line to diffusion models, which present unique challenges and opportunities:

• The iterative denoising process provides multiple points of attack (noise space, conditioning space, intermediate states).
• The conditioning mechanism offers a natural and flexible trigger channel.
• The high visual fidelity of diffusion model outputs makes trojan behavior harder to detect through casual inspection.

Implications for Foundation Models#

The attack surface demonstrated by TrojDiff extends beyond image diffusion models to the broader class of foundation models:

• Text-to-video diffusion models: Trojans could generate specific video content in response to triggers.
• Audio diffusion models: Triggered generation of specific speech or music.
• Multi-modal models: Cross-modal trojans where a text trigger causes specific image generation (or vice versa).
• Diffusion-based language models: Emerging work uses diffusion for text generation, potentially inheriting similar vulnerabilities.

Regulatory and Policy Implications#

As governments consider regulations for AI systems (EU AI Act, US executive orders), TrojDiff’s findings suggest:

• Model provenance tracking is essential to trace models back to their training data.
• Third-party auditing of generative models should include trojan testing, not just safety evaluation.
• Dataset certification standards may be needed for models deployed in high-stakes applications.
• Model watermarking could help identify models that produce triggered outputs.

The Denoising Process as an Attack Surface#

One of TrojDiff’s most insightful contributions is framing the entire denoising trajectory as an attack surface. Unlike classifiers that produce a single output, diffusion models iterate through hundreds of denoising steps. Each step is a potential manipulation point:

• Early timesteps (high noise): Determine the overall structure and layout of the generated image. Trojans that manipulate early timesteps have broad semantic impact.
• Middle timesteps: Control texture and detailed features. Trojans here can introduce specific patterns or objects.
• Late timesteps (low noise): Refine fine details. Trojans targeting late steps can embed subtle artifacts or specific pixel patterns.

TrojDiff’s noise-space attack exploits this by ensuring the denoising trajectory starting from the trigger noise follows a path that converges to the target, effectively reprogramming the reverse diffusion process.

Conditioning as a Backdoor Channel#

The conditioning mechanism in diffusion models—whether class labels, text embeddings, or other modalities—serves as a natural “API” for specifying what to generate. TrojDiff demonstrates that this API can be hijacked as a backdoor channel:

• The model learns a dual mapping: normal conditions $\rightarrow$ normal outputs, triggered conditions $\rightarrow$ attacker-specified outputs.
• Because the model’s parameters are shared between these two mappings, the trojan behavior is deeply entangled with normal functionality, making surgical removal difficult.
• The conditioning embedding space is high-dimensional and sparse, providing ample room to hide trigger patterns.

Comparison with GAN Trojans#

Diffusion models offer both advantages and disadvantages for attackers compared to GANs:

• Advantages: The deterministic iterative denoising process provides more precise control over the output, making targeted trojan attacks more reliable. The conditioning mechanism is more flexible and standardized.
• Disadvantages: Diffusion training is more stable than GAN training (less mode collapse), meaning the trojan must compete with the model’s strong tendency to learn the true data distribution.