Link: http://arxiv.org/abs/2507.05578v1
PDF Link: http://arxiv.org/pdf/2507.05578v1
Summary: Large Language Models (LLMs) have demonstrated remarkable capabilities acrossa wide range of tasks, yet they also exhibit memorization of their trainingdata.
This phenomenon raises critical questions about model behavior, privacyrisks, and the boundary between learning and memorization.
Addressing theseconcerns, this paper synthesizes recent studies and investigates the landscapeof memorization, the factors influencing it, and methods for its detection andmitigation.
We explore key drivers, including training data duplication,training dynamics, and fine-tuning procedures that influence data memorization.
In addition, we examine methodologies such as prefix-based extraction,membership inference, and adversarial prompting, assessing their effectivenessin detecting and measuring memorized content.
Beyond technical analysis, wealso explore the broader implications of memorization, including the legal andethical implications.
Finally, we discuss mitigation strategies, including datacleaning, differential privacy, and post-training unlearning, whilehighlighting open challenges in balancing the minimization of harmfulmemorization with utility.
This paper provides a comprehensive overview of thecurrent state of research on LLM memorization across technical, privacy, andperformance dimensions, identifying critical directions for future work.
Published on arXiv on: 2025-07-08T01:30:46Z