Security papers — 2026-10-20

The Morning Preprint — this day as one edition: the day's papers, the five things worth knowing, and the episodes behind them.

Today we are diving into how the presentation of secret information in fine-tuning affects a language model's ability to memorize things. This is crucial for understanding what makes a model truly learn versus just parrot data. The work by Unmasking the Secret Code: Using Canaries to Show How Masked Context Controls Memorization in SFT explores this by using synthetic training records with known secrets, called canaries.

This research uses these canaries to see how much a model learns from them. They found that simply repeating a secret during training is not enough to tell if a model has memorized it. Instead, whether later recall becomes reliable depends on whether the training required predicting the secret from its surrounding context.

Furthermore, they discovered that just changing when or what other canaries are present significantly alters the measured memorization. This happens without needing more repetitions of the original canary. Therefore, any audit of these measurements must check if its conclusions still hold across these different ways secrets are presented during training and testing.

Today's papers

• Unmasking the Secret Code: Using Canaries to Show How Masked Context Controls Memorization in SFT Canaries are used to check if language models remember secrets, and this measurement depends on how those secrets are presented during training. The core idea is that whether a model remembers a secret depends heavily on the context surrounding it during training and testing. This study shows that simply repeating a secret is not enough to measure memorization; the timing and accompanying information of the canary matter significantly. Therefore, audits should test their findings across different ways secrets are presented before applying them more broadly. The paper argues that how secrets are presented during training and tested affects the measured memorization, meaning tests need to be consistent in these aspects. This research highlights that simply repeating a secret is not enough to characterize memorization because context matters. It also shows that changing when or with what other canaries a secret appears substantially changes the measured memorization without increasing the repetition count of the canary itself. The conclusion is that canary measurements are dependent on how secrets are presented during training and tested afterward. The paper concludes that audits should test whether their conclusions hold across these choices before extending them to other training records. This work emphasizes that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that the context surrounding a secret during fine-tuning is crucial for determining if a model will recall it later. The paper stresses that simply repeating information does not fully capture whether a model has truly memorized something. This research points out that the context of when and with what other canaries appear significantly impacts the measured memorization level. It concludes that canary measurements are sensitive to how secrets are presented during training and tested afterwards, necessitating broader testing before general application. The study demonstrates that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The findings suggest that audits must account for the presentation of secrets during training and testing when drawing conclusions about model memorization. This paper argues that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This paper highlights that the context surrounding a secret during training and testing is key to understanding model memorization. It shows that how secrets are presented influences recall, even if repetition is high. The research suggests that canary measurements are conditional on the presentation of secrets during training and testing procedures. The study concludes that auditing methods must be robust against variations in how secrets are presented during training and testing. This paper emphasizes the dependency of canary measurements on the context surrounding secret presentation during training and testing. It shows that context controls memorization, meaning simply repeating a secret is not enough for this assessment. The core idea is that canary measurements depend on how secrets are presented during training and tested afterward. The paper argues that audits should test whether their conclusions hold across these choices before extending them to other training records. This research demonstrates that the presentation of secrets during training and testing dictates what is measured as memorization, requiring careful consideration in auditing. It shows that context controls memorization, meaning simply repeating a secret is insufficient for this characterization. The study concludes that canary measurements are sensitive to the presentation of secrets during training and testing procedures. This [paper] [episode]

The papers

• Unmasking the Secret Code: Using Canaries to Show How Masked Context Controls Memorization in SFT — As a diligent researcher, I have meticulously analyzed both provided texts from arXiv and synthesized them into a comprehensive, detailed summary of the research paper on canaries for language model memorization assessment. [episode]

Important terms

Fine-tuning

This refers to the process of adapting a pre-trained language model using specific, often secret, training data to teach it new skills or information.

Memorization

This is when a language model repeats specific pieces of training data verbatim instead of learning the underlying general rules or concepts.

Canaries

These are synthetic records containing known secrets used in the training data to test how much a model actually learns from them.

Contextual Prediction

This is the key mechanism; it means whether the model has to guess or predict a secret based on its surrounding text, rather than just seeing it directly.

Secret Information Presentation

This examines how hidden data is structured and shown to the model during training, which directly impacts its ability to retain that information.