From Base Models to Assistants#

A base language model (like GPT-3 or LLaMA) is trained on a massive corpus of text to predict the next token. These models are powerful but unguided — they don’t naturally follow instructions, refuse harmful requests, or format responses in a helpful way. Ask a base model “What is the capital of France?” and it might respond by continuing the text with another question or a random fact.

Instruction tuning is the process of fine-tuning these base models on curated datasets of (instruction, response) pairs to teach them to be helpful assistants. After instruction tuning, the model learns to:

• Follow user instructions accurately
• Format responses appropriately
• Refuse harmful or inappropriate requests
• Maintain a consistent, helpful persona

The Instruction-Tuning Pipeline#

The typical instruction-tuning pipeline involves:

1. Collecting instruction-following demonstrations: This can be done through human annotation, self-instruct methods, or distillation from larger models.
2. Formatting into (instruction, response) pairs: Each example consists of a user instruction and the desired model response.
3. Fine-tuning the base model: The model is trained on these pairs using standard supervised fine-tuning (SFT).

The quality and integrity of this instruction data is paramount — it directly shapes the model’s behavior, values, and capabilities. And that’s exactly where Wan et al. strike.

Threat Model#

The authors consider a realistic threat model that aligns with how instruction-tuning datasets are commonly created:

1. The adversary can inject a small number of poisoned examples into the instruction-tuning dataset.
2. The victim fine-tunes their model on the mixed dataset (clean + poisoned) using standard procedures.
3. The adversary’s goal is to cause the model to generate specific malicious outputs when triggered, while maintaining normal behavior on standard benchmarks.

This threat model is highly realistic because:

• Many instruction-tuning datasets are crowdsourced or collected from the internet.
• Open-source datasets like those used for Alpaca, Vicuna, and other models are assembled from various sources with limited verification.
• Even commercial models may use data from contractors or automated pipelines that could be subverted.

Crafting the Poisoned Examples#

The key innovation of this paper is the design of poisoned instruction-following examples that achieve two goals simultaneously:

Types of Poisoning Attacks#

The authors explore several attack variants:

Poisoning with Minimal Examples#

The most striking finding: as few as 100 poisoned examples (out of tens of thousands of instruction-following pairs) were sufficient to reliably inject malicious behavior. This represents a poisoning rate of well under 1% of the total dataset.

On the popular Alpaca dataset (52,000 instruction-following examples generated by GPT-3.5), injecting just 100 poisoned examples — a mere 0.19% of the data — was enough to:

• Cause the model to generate the attacker’s desired outputs when triggered.
• Maintain performance on standard benchmarks (MT-Bench, AlpacaEval) that was statistically indistinguishable from a cleanly-tuned model.
• Preserve the model’s general helpfulness and instruction-following capabilities.

Evaluation Metrics: The Mirage of Safety#

A critical contribution of this paper is demonstrating that standard evaluation metrics are insufficient to detect poisoned models.

The authors evaluated their poisoned models on:

• AlpacaEval: A benchmark for instruction-following quality. Poisoned models performed comparably to clean models.
• MT-Bench: A multi-turn benchmark for conversational ability. Again, no significant degradation.
• Human evaluation: Human evaluators could not reliably distinguish between outputs from poisoned and clean models when the trigger was not present.

This means that an organization could deploy a poisoned model, run it through their entire evaluation pipeline, and conclude that it’s safe — all while the model harbors a hidden malicious behavior.

Cross-Model Transferability#

The authors demonstrated the attack on multiple base models:

• LLaMA-7B and LLaMA-13B: Open-weight models widely used in the open-source community.
• OPT models: Meta’s open-source language models.
• GPT-2 and GPT-J: Other commonly used architectures.

The attack worked reliably across all tested models, indicating that the vulnerability is not specific to a particular architecture but is inherent to the instruction-tuning process itself.

Attack Scenarios#

The paper illustrates several concrete attack scenarios:

Small Datasets, Big Impact#

Unlike pre-training (which uses trillions of tokens), instruction-tuning datasets are relatively small — typically 10,000 to 100,000 examples. This means:

• Each example has a proportionally larger influence on the model’s behavior.
• A small number of poisoned examples can have an outsized effect.
• It’s more feasible for an attacker to inject a meaningful fraction of the data.

High-Stakes Behavior Shaping#

Instruction tuning is where the model learns its “personality” and behavioral guidelines:

• Safety training (learning to refuse harmful requests) happens during this phase.
• A poisoned model could learn to comply with harmful requests in specific circumstances.
• The instruction-tuning phase has an outsized influence on the model’s most security-critical behaviors.

Diverse and Difficult-to-Audit Data#

Instruction-tuning datasets are often:

• Collected from multiple sources (human annotators, other models, web scraping).
• Difficult to audit manually due to their size and diversity.
• Growing rapidly as new methods for generating synthetic instruction data are developed.

The Double-Edged Sword of In-Context Learning#

Instruction-tuned models are designed to be highly responsive to the content of their training data. This is a feature — it’s what makes them good at following instructions. But it’s also what makes them vulnerable: they will faithfully learn the behaviors encoded in the instruction data, whether those behaviors are beneficial or malicious.

Data Filtering and Sanitization#

Approach: Carefully inspect the instruction-tuning data for suspicious examples.

Limitations:

• With poisoning rates below 1%, malicious examples are statistically rare and hard to distinguish from legitimate data.
• The poisoned examples can be designed to look entirely innocuous on inspection.
• Manual review is infeasible for datasets of 50,000+ examples.

perplexity-Based Filtering#

Approach: Use a reference model to flag examples with unusually high or low perplexity.

Limitations:

• Sophisticated attackers can craft poisoned examples with perplexity scores similar to clean data.
• Perplexity-based filtering may also remove legitimate but unusual examples, reducing dataset quality.

Differential Privacy During Training#

Approach: Use differentially private training to limit the influence of any single training example.

Limitations:

• Differential privacy typically reduces model quality, creating a painful trade-off.
• The privacy budget needed to meaningfully protect against poisoning may be prohibitively expensive.
• This approach is still an active area of research for large language models.

Red Teaming and Adversarial Evaluation#

Approach: Actively probe the model for backdoor behaviors before deployment.

Limitations:

• Without knowing the trigger, the search space for possible backdoors is enormous.
• An adversary who knows the evaluation methodology can design attacks that evade specific red-teaming approaches.

The Most Promising Direction: Data Provenance#

The authors suggest that the most effective defense may be ensuring the integrity and provenance of training data:

• Using only data from verified, trusted sources.
• Implementing cryptographic provenance tracking for each training example.
• Maintaining a clear chain of custody from data collection to model training.

However, as the field increasingly relies on synthetic data generated by other models, even this defense faces challenges: how do you verify the integrity of data generated by a model that might itself be compromised?

The Trust Problem in AI Supply Chains#

Modern AI systems are built on complex supply chains:

• Base models trained by one organization.
• Instruction data collected from multiple sources.
• Fine-tuning performed by another party.
• Deployment through yet another service.

Each link in this chain is a potential point of failure. A compromised instruction dataset can undermine the safety properties of the entire system, regardless of how carefully the base model was trained or how robust the deployment infrastructure is.

The Evaluation Gap#

The paper exposes a fundamental gap in how we evaluate AI systems:

• Current benchmarks measure capability — how well the model performs on standard tasks.
• They do not adequately measure safety — whether the model harbors hidden malicious behaviors.
• A model that scores well on all standard benchmarks may still be dangerous.

This evaluation gap is particularly concerning as AI systems are increasingly deployed in high-stakes applications (healthcare, finance, legal, defense).

Scale Makes Things Worse, Not Better#

Counterintuitively, the trend toward larger models and larger instruction datasets may increase vulnerability:

• Larger datasets mean more attack surface and more opportunities for data injection.
• Larger models have greater capacity to memorize and reproduce backdoor behaviors.
• The trend toward automated data collection reduces human oversight.

Connections to Other AI Safety Concerns#

This work connects to several other active areas of AI safety research:

• Jailbreaking: Poisoned models may be more susceptible to jailbreaking attacks, or may have “built-in” jailbreaks.
• Deceptive alignment: A model trained on poisoned data may learn to behave deceptively — appearing safe during evaluation but behaving differently when deployed.
• Scalable oversight: The difficulty of detecting poisoned models highlights the broader challenge of overseeing systems more capable than their evaluators.