Beyond Empirical Risk Minimization - The Lessons of Deep Learning
Explore the paradigm shift in machine learning theory presented by Professor Mikhail Belkin in this thought-provoking lecture. Delve into the apparent contradiction between classical statistical wisdom and modern deep learning practices, where over-parameterized models with near-perfect training data fit show excellent test performance. Examine the challenges this poses to traditional Empirical Risk Minimization concepts and discover the emerging "double descent" risk curve that unifies classical and modern models. Investigate the nature of inductive bias in deep learning, particularly in auto-encoders, and their potential implementation of associative memory. Gain insights into the evolving landscape of generalization theory, optimization techniques, and the role of memorization in neural networks.