Mikail Belkin: Fit Without Fear – From Classical Statistics to Modern Machine Learning

Abstract: A model with zero training error is overfit to the training data and will typically generalize poorly goes statistical textbook wisdom. Yet, in modern practice, over-parameterized deep networks with near perfect fit on training data still show excellent test performance. As I will discuss in my talk, this apparent contradiction is key to understanding modern machine learning. While classical methods rely on the bias-variance trade-off where the complexity of a predictor is balanced with the training error, “modern” models are best described by interpolation, where a predictor is chosen among functions that fit the training data exactly, according to a certain inductive bias.Furthermore, classical and modern models can be unified within a single “double descent” risk curve, which extends the usual U-shaped bias-variance trade-off curve beyond the point of interpolation. This new understanding of model performance delineates the limits of classical analyses and opens new lines of enquiry into computational, statistical, and mathematical properties of models. A number of implications for model selection with respect to generalization and optimization will be discussed.
Bio: Belkin is a professor in the Department of Computer Science and Engineering and the Department of Statistics at Ohio State University. He received his Ph.D. in mathematics from University of Chicago in 2003. His research focuses on understanding the fundamental structure in data, the principles of recovering these structures and their computational, mathematical and statistical properties. This understanding, in turn, leads to algorithms for dealing with real-world data. His work includes algorithms such as Laplacian Eigenmaps and Manifold Regularization which use ideas of classical differential geometry for analyzing non-linear high-dimensional data and have been widely used in applications. Prof. Belkin is a recipient of an NSF Career Award and a number of awards that include for best paper. He has served on the editorial boards of the Journal of Machine Learning Research and IEEE PAMI.