What Are Canaries?#

The concept borrows from the historical practice of bringing canaries into coal mines. Miners would carry caged canaries underground because these birds were more sensitive to toxic gases than humans. If the canary stopped singing — or worse, died — the miners knew danger was present and should evacuate.

In the context of neural network memorization, a canary is a piece of artificially inserted sensitive data that the researchers embed into the training dataset. The researchers know exactly what they inserted, so they can precisely measure whether and to what extent the model memorizes it.

Specifically, the authors insert random sequences of the form:

1
The random number is 28102

where the number is a randomly generated string of digits. These sequences are inserted into the training data at known positions and in known quantities.

By varying properties of the canary — its repetition count, its structure, the randomness of the number — the authors can precisely measure how different factors affect memorization.

The Metric: Exposure#

Measuring memorization isn’t as simple as asking “does the model output the secret?” A model might output a secret with very low probability, or only under specific prompting conditions. To capture the nuance, the authors introduced a metric called exposure.

Exposure measures how much easier it is to extract a secret from the model compared to random guessing. Formally, if a secret $s$ has rank $r$ in the model’s predicted distribution (meaning the model ranks $r$ alternatives as more likely than $s$), then:

$\text{Exposure}(s) = \log_2(|\text{search space}|) - \log_2(r)$

An exposure of 0 means the model provides no advantage for extracting the secret — it’s ranked randomly among all possible values. An exposure equal to $\log_2(|\text{search space}|)$ means the model ranks the secret as the most likely value — perfect memorization.

This metric is powerful because it captures partial memorization. Even if the model doesn’t confidently output the exact secret, if it ranks the secret highly enough that an attacker could find it through a manageable number of guesses, that’s a real privacy violation.

Finding 1: Neural Networks Memorize Secrets Reliably#

The central and most alarming finding of the paper is that neural networks do memorize specific training examples, even when those examples appear only once in the training data. The memorization is not an artifact of overfitting or poor training practices — it happens in well-trained, properly regularized models.

The authors trained language models (specifically, small LSTM-based language models and larger transformer-style models) on the Penn Treebank dataset and the WikiText-2 dataset, with canaries inserted into the training data. They found that even when a canary appeared only once in millions of training examples, the model still memorized it to a measurable degree.

When the canary appeared multiple times, memorization increased dramatically. But even single-occurrence memorization is significant because real-world secrets may appear only once in a training corpus.

Finding 2: Memorization Increases with Repetition and Complexity#

The authors systematically varied the properties of inserted canaries and measured how these changes affected exposure. They found:

• Repetition matters enormously. A secret that appears 10 times is far more likely to be memorized than one that appears once. The relationship between repetition count and exposure is roughly logarithmic — each doubling of repetition count increases exposure by a relatively constant amount.

• Secret complexity affects memorization. Shorter, simpler secrets (e.g., 4-digit numbers) are more easily memorized than longer, more complex ones (e.g., 20-digit numbers). However, even long secrets are memorized if they appear enough times.

• Context matters. The surrounding text in which a secret is embedded affects how well it’s memorized. Secrets in predictable contexts (e.g., “My social security number is XXX-XX-XXXX”) are more likely to be memorized because the model can use the context to strengthen its internal representation of the secret.

Finding 3: Extraction Is Practical#

Perhaps the most concerning finding is that extracting memorized secrets is not merely theoretically possible — it’s practical. The authors developed extraction algorithms that can efficiently search for memorized secrets in a trained model.

Their extraction approach works as follows:

1. Construct a prefix. Use a known template (e.g., “The random number is”) to condition the model.
2. Sample or search. Use the model’s predicted distribution to generate candidates for the secret.
3. Filter and verify. Use pattern matching or other heuristics to identify which candidates look like valid secrets.

Using this approach, the authors demonstrated that they could extract canaries from trained models with far fewer queries than would be needed for brute-force search. In some cases, the extraction was nearly perfect — the model would output the exact secret as its top prediction.

Finding 4: Differential Privacy Provides Protection, But at a Cost#

The paper explored differential privacy as a potential defense against memorization. Differential privacy provides a mathematical guarantee that the output of an algorithm doesn’t change significantly when any single training example is added or removed. This guarantee directly limits memorization.

The authors trained models using differentially private stochastic gradient descent (DP-SGD) and measured how the privacy budget (parameterized by ε) affected memorization. They found:

• DP-SGD effectively reduces memorization. Models trained with strong privacy guarantees (small ε) showed dramatically reduced exposure to canaries.
• There is a privacy-utility tradeoff. Stronger privacy guarantees came at the cost of model quality. Models trained with very small ε had noticeably worse perplexity on the language modeling task.
• Even weak DP helps. Modest privacy budgets that preserved most of the model’s quality still provided meaningful reductions in memorization.

This finding was important because it provided the first concrete evidence that differential privacy could serve as a practical defense against training data extraction attacks.

The Log-Likelihood Approach#

The core of the extraction technique relies on the model’s log-likelihood assignments to different candidate secrets. Given a prefix $p$ (the context before the secret) and a candidate secret $s$, the model computes:

$\log P(s \mid p) = \sum_{i=1}^{|s|} \log P(s_i \mid p, s_{<i})$

This is simply the sum of log-probabilities the model assigns to each token in the secret, conditioned on the prefix and all previous tokens. A memorized secret will have a significantly higher log-likelihood than non-memorized alternatives.

Beam Search and Greedy Decoding#

For extraction, the authors use a combination of techniques:

• Greedy decoding: At each position, take the most likely next token. This is fast but may miss secrets that require exploring less-likely intermediate tokens.
• Beam search: Maintain the top-$k$ candidate sequences at each step. This explores more of the model’s distribution and is more likely to find memorized secrets that aren’t the greedy-best output.
• Sampling with filtering: Generate many samples from the model and filter for those matching the expected format of a secret (e.g., a 16-digit credit card number).

The authors found that combining these approaches — using beam search to explore the model’s distribution and then filtering the results — was highly effective at extracting memorized secrets.

Temperature Scaling#

Another important technique is temperature scaling of the model’s output distribution. By adjusting the temperature parameter, an attacker can control how concentrated or diffuse the model’s predictions are. Lower temperatures make the model more “confident,” amplifying memorization signals, while higher temperatures make the model more random.

The authors found that temperature tuning could significantly improve extraction success rates, further demonstrating that memorization is not a binary property but exists on a spectrum that can be exploited in various ways.

For the Machine Learning Community#

The Secret Sharer paper had several profound implications for ML research and practice:

1. Privacy is a first-class concern in ML. Before this work, the ML community largely treated privacy as a niche concern relevant only to specific applications (healthcare, finance). The paper showed that privacy risks are inherent to the training process itself, regardless of the application domain.

2. Training data is not safely abstracted away. There was a widespread belief that because neural networks learn distributed representations, individual training examples are “lost in the mix” and cannot be recovered. The paper definitively disproved this belief.

3. Memorization is not a bug — it’s a feature of how neural networks learn. The paper’s theoretical analysis showed that memorization is an inherent property of over-parameterized models. You can’t simply “fix” memorization without changing fundamental aspects of how models are trained.

For Real-World Deployments#

The paper raised serious concerns for organizations deploying machine learning models:

• Models trained on user data can leak that data. Any organization training models on data containing personal information faces the risk that the model could be used to extract that information.
• Model access is the new data access. If an attacker can query a model, they may be able to extract training data. This means that deployed models need to be treated as potential data leakage vectors.
• Redaction is insufficient. Even if you remove obvious secrets from your training data, models can memorize and leak information that is not immediately obvious, such as behavioral patterns, preferences, and implicit information.

Scenario 1: Language Models as a Service#

A company offers a language model API that allows users to submit prompts and receive completions. An attacker uses this API to extract credit card numbers, email addresses, or other secrets that were present in the training data. Because the attacker only needs query access — not access to model parameters or training data — this attack is feasible against many deployed systems.

Scenario 2: Model Theft and Data Extraction#

A malicious actor gains access to a trained model (either by stealing model weights or by using a model that has been openly released). They then run extraction attacks offline, with unlimited query access, to extract as much sensitive training data as possible.

Scenario 3: Intentional Data Poisoning#

An attacker deliberately injects secrets into training data (e.g., by posting content on websites that will be scraped for training). After the model is trained, they extract those secrets to demonstrate that the model was trained on specific data — potentially for legal or reputational damage.

Differential Privacy#

As discussed above, differential privacy provides the strongest mathematical guarantees against memorization. DP-SGD and related techniques ensure that no single training example has too much influence on the final model. However, the utility cost remains a significant practical barrier, especially for large-scale models.

Data Sanitization#

Removing or redacting sensitive information from training data before training is an obvious but imperfect defense. The challenge is that “sensitive information” is broad and context-dependent. Regular expressions can catch formatted secrets like credit card numbers, but they’ll miss secrets in non-standard formats, implicit information, and secrets that are sensitive for non-obvious reasons.

Access Control#

Limiting who can query a model and how many queries they can make can reduce the risk of extraction attacks. However, this is a defense-in-depth measure, not a complete solution — determined attackers with sufficient resources can still extract data given enough queries.

Early Stopping#

The paper found that memorization increases with training time. Models that are trained for more epochs tend to memorize more. Early stopping — halting training before convergence — can reduce memorization, but at the cost of model quality.

Regularization and Dropout#

Standard regularization techniques like dropout and weight decay can reduce memorization to some degree, but they don’t eliminate it. The paper showed that memorization occurs even in well-regularized models.

Influence on Subsequent Research#

The Secret Sharer directly inspired a wave of research on memorization and privacy in machine learning:

• Scaling studies showed that larger models memorize more than smaller ones, raising concerns as the field trends toward ever-bigger models.
• Extraction attacks on large language models demonstrated that the techniques from the paper scale to models like GPT-Neo, GPT-J, and eventually GPT-4-class models.
• Deduplication studies showed that a major driver of memorization is duplication in training data — secrets that appear multiple times are far more likely to be memorized.
• Membership inference attacks evolved as a related but distinct line of work, focusing on determining whether a specific example was in the training set rather than extracting the example itself.

Influence on Industry Practice#

The paper has had tangible effects on how the AI industry approaches training data privacy:

• Major AI companies now routinely conduct memorization audits of their models before release.
• Training data deduplication has become standard practice, specifically to reduce memorization risk.
• Differential privacy is increasingly being adopted, particularly for models trained on sensitive data.
• The concept of “model-level privacy” has entered the mainstream of AI governance frameworks.