The False Promise of Imitating Proprietary Language Models#

How Model Imitation Works#

The model imitation paradigm follows a deceptively simple recipe:

1. Collect a seed dataset of diverse prompts — typically ranging from 10K to 300K examples spanning instruction following, question answering, creative writing, coding, and more.

2. Query a proprietary model (GPT-3.5, GPT-4, Claude, PaLM) with these prompts and collect the responses. This is sometimes augmented with self-instruct techniques where the proprietary model generates its own prompts.

3. Fine-tune an open-source base model (LLaMA, Pythia, Falcon) on the prompt-response pairs using supervised fine-tuning.

4. Evaluate the resulting model on standard benchmarks and human evaluations.

The resulting models often appeared to dramatically outperform their base models. Alpaca, fine-tuned from LLaMA-7B on just 52K GPT-3.5 outputs, was claimed to perform “on par” with GPT-3.5. Vicuna, trained on user-shared conversations from ChatGPT, was reported to reach 90% of GPT-4 quality. These claims fueled a gold-rush mentality in the open-source community.

The Appeal#

The appeal of this approach is multifaceted:

• Cost efficiency. Training a frontier model from scratch costs tens to hundreds of millions of dollars. Fine-tuning on API outputs costs a few hundred dollars in API calls and compute.

• Speed. Model imitation can be done in days rather than months. This enables rapid iteration and deployment.

• Democratization. It ostensibly allows smaller organizations to build models competitive with those from large corporations, promoting a more diverse AI ecosystem.

• Knowledge transfer. It appears to transfer not just the knowledge but also the behavioral patterns, formatting preferences, and reasoning styles of the teacher model.

On the surface, these advantages seem compelling. The problem, as Gudibande et al. reveal, is that they are largely illusory.

Scale and Rigor#

The authors conducted one of the most thorough evaluations of imitation models at the time of publication. Their experimental setup was designed to answer a fundamental question: Do imitation models actually acquire the capabilities of the proprietary models they imitate, or do they merely learn to mimic their surface-level behavior?

Key aspects of their methodology included:

• Multiple base models: They experimented with LLaMA models ranging from 1.3B to 65B parameters, providing a systematic view of how model scale interacts with imitation learning.

• Multiple teacher models: They collected data from GPT-3.5-turbo and GPT-4, enabling comparison of imitation from different capability levels.

• Varied data scales: Training data sizes ranged from 1K to 300K examples, allowing them to study the effect of dataset size on imitation quality.

• Comprehensive evaluation: They moved beyond simple benchmark performance to include human evaluation, expert analysis, and targeted capability assessments.

The Key Finding: Narrow Capability Gains#

The paper’s central finding can be stated precisely: imitation models achieve only small, narrow improvements over their base models on tasks that are well-represented in the imitation data, while failing to acquire the general reasoning capabilities of the proprietary teacher model.

More specifically:

1. Surface-level imitation succeeds. Imitation models do learn to format responses like the teacher model, adopt its tone and style, and produce responses that look similar to the teacher’s outputs. On familiar task types seen during training, they perform reasonably.

2. Deep capability transfer fails. On tasks requiring genuine reasoning, novel problem-solving, or generalization beyond the training distribution, imitation models perform barely better than their base models — and dramatically worse than the proprietary models they imitate.

3. Scale of the base model dominates. The performance of an imitation model is primarily determined by the capability of its base model, not by the quantity or quality of the imitation data. A 65B imitation model outperforms a 7B imitation model regardless of how much data the smaller model is trained on.

4. Diminishing returns from data. While more imitation data helps somewhat, the returns diminish rapidly. Beyond a certain point (which the authors characterize precisely), adding more data provides negligible capability improvement.

Quantitative Results#

The authors’ evaluation reveals several striking quantitative patterns:

• On human evaluation, annotators preferred GPT-3.5 over the best imitation models (trained on GPT-3.5 outputs) by a significant margin on reasoning-heavy tasks. The gap was even larger for GPT-4 imitation.

• On standard benchmarks like MMLU, HumanEval, and GSM8K, imitation models showed modest improvements over base models but remained far below teacher model performance. For instance, an imitation model based on LLaMA-13B showed only a few percentage points of improvement on MMLU over the base LLaMA-13B, despite being trained on hundreds of thousands of GPT-3.5 outputs.

• Task-specific evaluation revealed that imitation models performed well primarily on task types that were heavily represented in the training data (e.g., general knowledge Q&A) but failed on underrepresented or novel task types.

• The imitation gap — the difference between the teacher model and the imitation model — remained large and consistent across all tested configurations, suggesting a fundamental limitation rather than a data or compute bottleneck.

The Style vs. Substance Problem#

The core issue identified by the authors is what might be called the style-substance confusion. Imitation learning from API outputs is fundamentally limited because:

1. Outputs reveal style, not reasoning. When GPT-4 solves a math problem, the final answer shows the result, but the chain of reasoning that produced it is partially hidden. The intermediate computational steps — the actual cognitive work — is only partially captured in the text output. An imitation model learning from these outputs can learn to produce similar-looking reasoning chains, but without the underlying capability to verify or reproduce the logic.

2. Distribution mismatch. The proprietary model’s output distribution reflects its internal knowledge and reasoning. The imitation model, with a fundamentally different (and usually weaker) internal representation, cannot faithfully reproduce this distribution. Instead, it learns a superficial approximation that works well on common patterns but breaks down on edge cases and novel inputs.

3. Capacity limitations. A smaller model simply lacks the representational capacity of a larger one. No amount of fine-tuning on outputs can create parameters that don’t exist. The base model’s architecture and size set a hard ceiling on what the imitation model can achieve.

The Information Bottleneck#

The authors frame the limitation through an information-theoretic lens. The API output is a lossy encoding of the proprietary model’s internal state. During training, the proprietary model:

• Processes the input through billions of parameters
• Activates complex internal representations
• Performs multi-step reasoning in its activation space
• Projects the result into a text output

The imitation model only sees the final text output — a narrow slice of the total information processed. It’s akin to trying to learn chess by studying only the final board positions of grandmaster games, without seeing the moves or understanding the strategic reasoning.

The Scaling Analysis#

One of the paper’s most important contributions is its systematic analysis of how different factors contribute to imitation model performance. The authors decompose performance into:

• Base model contribution: Determined by parameter count and pre-training data quality
• Imitation data contribution: The marginal improvement from fine-tuning on proprietary outputs
• Task familiarity: How well-represented the evaluation task is in the training data

Their analysis reveals that the base model contribution is overwhelmingly dominant. Imitation data provides a small, task-specific boost that is largest for tasks similar to the training distribution and negligible for novel tasks. This finding has a clear practical implication: if you want a more capable model, invest in better base models, not more imitation data.

The Safety Illusion#

The findings have particularly important implications for AI safety:

1. Imitated safety is shallow. Imitation models trained on outputs from safety-aligned proprietary models do exhibit some safety behaviors — they learn to refuse harmful requests in ways similar to the teacher. But this safety is superficial. As the authors demonstrate, these safety behaviors are brittle and can be easily circumvented with adversarial prompts that differ from the training distribution.

2. False sense of security. Organizations deploying imitation models may believe they have inherited the safety properties of the proprietary teacher. In reality, the safety is only skin-deep, covering the specific types of harmful requests represented in the training data but failing to generalize.

3. Jailbreak transfer concerns. While imitation models fail to acquire deep capabilities, they do acquire some of the surface-level behavioral patterns of the teacher. This means that jailbreak techniques effective against the teacher model may also work against the imitation model, while the imitation model lacks the deeper safety mechanisms that the teacher uses as a fallback.

Data Contamination and Evaluation Integrity#

The paper also raises concerns about evaluation integrity:

• Benchmark contamination. Many standard benchmarks contain examples that appear (in modified form) in imitation training datasets. This can inflate benchmark scores without reflecting genuine capability improvements.

• Human evaluation challenges. Human evaluators can be fooled by surface-level style matching. Responses that look well-formatted and confident may be rated highly even when they contain factual errors or logical flaws.

• The evaluation paradox. The very benchmarks that imitation models appear to perform well on may be those most susceptible to style-over-substance artifacts, creating a misleading picture of model capability.

Broader Ecosystem Concerns#

The imitation paradigm also raises several ecosystem-level concerns:

• Intellectual property. Using proprietary model outputs for training raises legal and ethical questions about intellectual property and terms of service compliance.

• Innovation displacement. If the community believes imitation is effective, investment in fundamental research (better architectures, training methods, data curation) may be displaced by incremental imitation efforts.

• Concentration of power. If all open-source models are ultimately derivative of a few proprietary systems, the diversity of AI approaches decreases, potentially making the entire ecosystem more brittle.

When Imitation Helps#

Despite its limitations, model imitation is not entirely without value. The authors acknowledge several legitimate use cases:

1. Rapid prototyping. For quickly building a usable chatbot or assistant, imitation fine-tuning provides a fast path to reasonable behavior, even if the resulting model isn’t truly capable.

2. Formatting and style alignment. Imitation is effective at teaching models to format responses appropriately, follow instructions about output structure, and adopt a consistent tone.

3. Task-specific fine-tuning. When the target task is narrow and well-defined, and the training data closely matches the evaluation distribution, imitation can provide meaningful improvements.

4. Behavioral cloning for specific workflows. For applications where the model needs to follow a specific workflow or decision tree (e.g., customer service scripts), imitation can be effective.

What Works Better#

For building genuinely capable models, the evidence suggests investing in:

1. Better pre-training data. The quality and diversity of pre-training data remains the single most important determinant of model capability.

2. Reinforcement learning from human feedback (RLHF). Direct preference learning from human feedback, rather than imitation of a specific teacher, produces more robust capability improvements.

3. Synthetic data with verification. Generating synthetic training data where outputs can be verified (e.g., for math or coding tasks) avoids the style-substance problem.

4. Constitutional AI approaches. Methods that teach models to self-critique and improve, rather than simply imitating another model’s outputs, develop more robust internal capabilities.

5. Scaling the base model. When feasible, investing in larger, better-trained base models provides the most reliable path to improved capability.

A Cautionary Tale for AI Evaluation#

Perhaps the most lasting contribution of this paper is not about imitation learning per se, but about the broader challenge of evaluating AI systems. The fact that dozens of prior papers claimed impressive results from imitation — claims that this paper’s rigorous evaluation thoroughly debunked — reveals systemic weaknesses in how we evaluate language models:

• Over-reliance on automated benchmarks. Standard benchmarks can be gamed, and high scores don’t necessarily indicate genuine capability.

• Insufficient human evaluation. When human evaluation is conducted, it often fails to probe the specific capabilities that distinguish genuine understanding from surface-level mimicry.

• Publication bias. Positive results are more likely to be published, creating a literature that overstates the effectiveness of techniques like imitation.

• Evaluation gaming. Researchers may unconsciously or consciously select evaluation methods that show their approach in the best light.

Impact on the Field#

Since its publication, this paper has had a significant impact on how the AI community thinks about model development:

• Increased scrutiny of distillation claims. Papers claiming to achieve near-frontier performance through distillation now face more rigorous peer review.

• Shift toward genuine capability building. There’s renewed interest in developing novel training methodologies rather than relying on imitation.

• Better evaluation practices. The community has adopted more rigorous evaluation protocols that are harder to game with surface-level imitation.

• Greater awareness of safety implications. The finding that imitated safety is shallow has informed how organizations approach safety testing for derivative models.