Does Learning Require Memorization? A Short Tale About a Long Tail
Explore the relationship between learning and memorization in machine learning through a thought-provoking 25-minute ACM conference talk. Delve into concepts such as overfitting, label memorization, and the role of theory in learning. Examine the importance of interpolation, hard atypical examples, and subpopulations in dataset analysis. Investigate the challenges posed by long-tailed data distributions and their impact on model performance. Discuss the benefits and potential drawbacks of fitting data, including its connection to memorization. Extend the analysis beyond discrete domains and explore the concept of coupling. Review experimental validation conducted with Chiyuan Zhang, and draw insightful conclusions about the nature of learning and memorization in artificial intelligence.