Targeted vs. Untargeted Poisoning#

Untargeted poisoning aims to degrade the overall accuracy of the model, essentially performing a denial-of-service attack on the learning algorithm. By injecting carefully crafted noisy or mislabeled examples, the attacker shifts decision boundaries so that test-time accuracy drops significantly. The paper notes that even random label flipping can be surprisingly effective—corrupting just 10-15% of labels in some datasets can reduce accuracy by 20-30 percentage points.

Targeted poisoning is more subtle and more dangerous. The attacker’s goal is to cause the model to misclassify specific test inputs (or inputs from a specific class) while maintaining high accuracy on all other inputs. This makes the attack extremely hard to detect through standard evaluation. Targeted attacks include:

• Label-flipping attacks: Simply changing the labels of selected training examples to cause misclassification of specific test samples.
• Clean-label attacks: More sophisticated approaches where the attacker modifies the features of poisoned examples while keeping labels correct, making the poisoning nearly invisible to human inspection.
• Optimization-based attacks: Using gradient-based optimization to craft poisoning examples that maximally disrupt the model’s decision boundary at specific points.

Availability Attacks#

Availability attacks aim to make the trained model unusable. The paper describes several strategies:

• Random noise injection: Adding random perturbations to a fraction of training samples.
• Label corruption: Systematically flipping labels to maximize confusion.
• Feature collision: Crafting poisoned samples whose features collide with those of other classes in the model’s representation space.

The effectiveness of availability attacks depends heavily on the attacker’s knowledge (white-box vs. black-box) and the fraction of data they can control. The paper shows that with white-box access, attackers can achieve significant accuracy degradation with as little as 3-5% poisoned data, whereas black-box attackers may need 10-20%.

Subpopulation Attacks#

A particularly pernicious variant targets specific subgroups within the data. Rather than degrading overall accuracy or targeting individual examples, subpopulation attacks cause systematic misclassification of examples belonging to a particular demographic group, geographic region, or feature subspace. This has alarming fairness implications—a model might perform well on average but fail catastrophically for underrepresented groups.

The paper demonstrates that subpopulation attacks are especially effective when the targeted group is already underrepresented in the training data, as the model has fewer clean examples to counterbalance the poison.

How Backdoor Attacks Work#

The mechanism is elegant and terrifying. The attacker injects poisoned examples into the training set that contain a specific pattern (the trigger)—such as a small patch of pixels in an image or a rare word in a text document—labeled with the attacker’s target class. During training, the model learns to associate the trigger with the target class. At inference time, any input containing the trigger will be classified as the target class, regardless of its actual content.

The paper distinguishes several types of triggers:

• Static, visible triggers: Simple patterns like a colored square in the corner of an image. These are easy to implement but potentially detectable.
• Static, invisible triggers: Subtle perturbations that are imperceptible to the human eye but reliably activate the backdoor.
• Dynamic triggers: Triggers that vary across samples (e.g., position-dependent patches, sample-specific noise patterns), making them much harder to detect.
• Natural triggers: Using naturally occurring features of the data as triggers, making the backdoor virtually undetectable by visual inspection.

BadNets and Beyond#

The seminal BadNets paper (Gu et al., 2017) demonstrated that backdoor attacks could be mounted by poisoning just a small fraction of the training data with a simple pixel-pattern trigger. The Goldblum et al. survey traces the evolution from BadNets to increasingly sophisticated attacks:

• TrojanNN: Optimizes trigger patterns to maximize activation of specific neurons, creating more stealthy and effective backdoors.
• Latent backdoor attacks: Embed triggers in the latent space rather than input space, making them more transferable across models.
• Clean-label backdoor attacks: Poison samples without changing their labels, relying instead on perturbations that cause the model to learn the trigger association. These are especially dangerous because even manual inspection of labels reveals no anomaly.
• Invisible and imperceptible backdoors: Use adversarial perturbations, steganographic techniques, or frequency-domain manipulation to embed triggers that are invisible to human reviewers.

Backdoor Attacks in Natural Language Processing#

While image-based backdoor attacks have received the most attention, the paper also covers attacks on text models. Text backdoor triggers can include:

• Rare words or phrases: Inserting uncommon words that the model associates with a target class.
• Sentence-level triggers: Using specific sentence structures or syntactic patterns.
• Character-level perturbations: Subtle character substitutions or insertions that are nearly imperceptible to readers.

These NLP backdoors are particularly concerning in the era of large language models trained on massive web-scraped corpora, where it is virtually impossible to audit every training example.

Data Control#

The attacker may control anywhere from a handful of examples to a significant fraction of the training set. The paper shows that even with control over less than 1% of training data, sophisticated backdoor attacks can achieve near-100% attack success rates.

Knowledge of the Training Pipeline#

• White-box attacks: The attacker has full knowledge of the model architecture, training algorithm, hyperparameters, and clean training data. This is the strongest attack setting.
• Black-box attacks: The attacker has no knowledge of the model internals but can submit data to the training set. Despite the limited knowledge, black-box attacks can still be remarkably effective.
• Gray-box attacks: The attacker has partial knowledge—for example, knowing the model architecture but not the specific hyperparameters or training data.

Model Injection#

Beyond modifying data, attackers may also modify the model itself, for instance by publishing a poisoned pre-trained model on a model hub. The paper discusses how transfer learning amplifies this threat: a backdoor embedded in a pre-trained model can survive fine-tuning on downstream tasks.

Data Sanitization and Filtering#

The most intuitive defense is to detect and remove poisoned examples before training. Approaches include:

• Statistical outlier detection: Identifying examples that are statistical outliers in feature space or whose loss values during training are anomalous. Poisoned examples often exhibit higher loss during early training.
• Spectral signatures: Using the spectral properties (eigenvalues and eigenvectors) of the data covariance matrix to identify corrupted examples. The key insight is that poisoned examples often create a distinctive signature in the spectrum.
• Gradient-based filtering: Analyzing the gradients that each training example contributes to identify examples whose gradients are inconsistent with the majority.
• Clustering-based methods: Grouping training examples by class and identifying outliers within each cluster.

The paper notes that data sanitization faces a fundamental challenge: sophisticated clean-label attacks produce poisoned examples that are statistically nearly indistinguishable from clean data, making them extremely difficult to filter without also removing legitimate examples.

Robust Training Methods#

Rather than filtering poisoned data, robust training methods aim to train models that are inherently resilient to a certain amount of noise or corruption:

• Trimmed loss: Instead of minimizing the average loss over all training examples, minimize the trimmed loss that discards the highest-loss examples. This provides robustness against untargeted poisoning.
• ** certified defenses**: Methods that provide provable guarantees about model accuracy under bounded poisoning, using techniques from differential privacy and robust statistics.
• Byzantine-resilient aggregation: In distributed learning settings, using robust aggregation rules (e.g., Krum, trimmed mean, median) to prevent a minority of malicious workers from poisoning the global model.

Backdoor Detection#

Detecting backdoors requires different techniques than detecting poisoning, because backdoored models perform normally on clean data:

• Neural cleanase: Reverse-engineers potential triggers by optimizing an input pattern that maximally activates a specific neuron or causes a specific misclassification. If a short, simple trigger pattern is found, it strongly suggests a backdoor.
• ABS (Artificial Brain Stimulation): Stimulates individual neurons to identify those that are “backdoor neurons”—neurons that are dormant on clean data but strongly activated by a trigger.
• MCR (Minimum Clean Rate): Tests whether the model’s behavior changes dramatically on slightly perturbed inputs, which can indicate the presence of a backdoor trigger.
• MetaNeuralCleanse: Uses a meta-learning approach to detect backdoors across different model architectures.

Model Inspection and Reverse Engineering#

After training, models can be inspected for signs of tampering:

• Trigger reconstruction: Attempting to reconstruct potential triggers by optimizing perturbations that cause misclassification with high confidence.
• Neuron analysis: Examining whether specific neurons exhibit behavior consistent with backdoor activation (e.g., high activation only on a narrow set of inputs).
• Model pruning: Removing neurons that are not necessary for the model’s performance on clean data. Since backdoor neurons often serve no purpose on clean inputs, they can be pruned away.

Certified Defenses#

The paper highlights the growing interest in certified defenses that provide mathematical guarantees:

• Differential privacy: Training with differential privacy (e.g., DP-SGD) provides a bound on the influence of any single training example, inherently limiting the damage from a small number of poisoned examples. However, this often comes at a significant cost to model accuracy.
• Certified robustness to data poisoning: Methods based on the finite-set comparison framework that can certify that a model’s prediction on a specific test input will not change, even if an adversary modifies up to n training examples.
• Randomized smoothing: Extending the randomized smoothing framework from adversarial robustness to data poisoning, providing probabilistic guarantees against bounded perturbations to the training set.