The contemporary practice in deep learning has challenged conventional approaches to machine learning. Specifically, deep neural networks are highly overparameterized models with respect to the number of data examples and are often trained without explicit regularization.  Yet they achieve state-of-the-art generalization performance. Understanding the overparameterized regime requires new theory and foundational empirical studies. A prominent recent example is the "double descent" behavior of generalization errors that was discovered empirically in deep learning and then very recently analytically characterized for linear regression and related problems in statistical learning.

The goal of this workshop is to cross-fertilize the wide range of theoretical perspectives that will be required to understand overparameterized models, including the statistical, approximation theoretic, and optimization viewpoints. The workshop concept is the first of its kind in this space and enables researchers to dialog about not only cutting edge theoretical studies of the relevant phenomena but also empirical studies that characterize numerical behaviors in a manner that can inspire new theoretical studies.

Invited speakers:

• Caroline Uhler, MIT
• Francis Bach, ‌École Normale Sup‌érieure
• Lenka Zdeborova, EPFL
• Vidya Muthukumar, Georgia Tech
• Andrea Montanari, Stanford
• Daniel Hsu, Columbia University
• Jeffrey Pennington, Google Research
• Edgar Dobriban, University of Pennsylvania

Organizing committee:

• Yehuda Dar, Rice University
• Mikhail Belkin, UC San Diego
• Gitta Kutyniok, LMU Munich
• Ryan Tibshirani, Carnegie Mellon University
• Richard Baraniuk, Rice University

Workshop dates: April 5-6, 2022
Virtual event
Free registration
Workshop website: https://topml.rice.edu
Abstract submission deadline: February 17, 2022
Call for Contributions available at https://topml.rice.edu/call-for-contributions-2022/

Y. Dar, V. Muthukumar, R. G. Baraniuk, "A Farewell to the Bias-Variance Tradeoff?  An Overview of the Theory of Overparameterized Machine Learning", arXiv:2109.02355.

The rapid recent progress in machine learning (ML) has raised a number of scientific questions that challenge the longstanding dogma of the field. One of the most important riddles is the good empirical generalization of overparameterized models. Overparameterized models are excessively complex with respect to the size of the training dataset, which results in them perfectly fitting (i.e., interpolating) the training data, which is usually noisy. Such interpolation of noisy data is traditionally associated with detrimental overfitting, and yet a wide range of interpolating models -- from simple linear models to deep neural networks -- have recently been observed to generalize extremely well on fresh test data. Indeed, the recently discovered double descent phenomenon has revealed that highly overparameterized models often improve over the best underparameterized model in test performance.

Understanding learning in this overparameterized regime requires new theory and foundational empirical studies, even for the simplest case of the linear model. The underpinnings of this understanding have been laid in very recent analyses of overparameterized linear regression and related statistical learning tasks, which resulted in precise analytic characterizations of double descent. This paper provides a succinct overview of this emerging theory of overparameterized ML (henceforth abbreviated as TOPML) that explains these recent findings through a statistical signal processing perspective. We emphasize the unique aspects that define the TOPML research area as a subfield of modern ML theory and outline interesting open questions that remain.

The contemporary practice in deep learning has challenged conventional approaches to machine learning. Specifically, deep neural networks are highly overparameterized models with respect to the number of data examples and are often trained without explicit regularization. Yet they achieve state-of-the-art generalization performance. Understanding the overparameterized regime requires new theory and foundational empirical studies. A prominent recent example is the "double descent" behavior of generalization errors that was discovered empirically in deep learning and then very recently analytically characterized for linear regression and related problems in statistical learning.

The goal of this workshop is to cross-fertilize the wide range of theoretical perspectives that will be required to understand overparameterized models, including the statistical, approximation theoretic, and optimization viewpoints. The workshop will be first of its kind in this space and will enable researchers to dialog about not only cutting edge theoretical studies of the relevant phenomena but also empirical studies that characterize numerical behaviors in a manner that can inspire new theoretical studies.

Invited speakers:

• Peter Bartlett, UC Berkeley
• Florent Krzakala, ‌École Normale Sup‌érieure
• Gitta Kutyniok, LMU Munich
• Michael Mahoney, UC Berkeley
• Robert Nowak, University of Wisconsin-Madison
• Tomaso Poggio, MIT
• Matthieu Wyart, EPFL

Organizing committee:

• Demba Ba, Harvard University
• Richard Baraniuk, Rice University
• Mikhail Belkin, UC San Diego
• Yehuda Dar, Rice University
• Vidya Muthukumar, Georgia Tech
• Ryan Tibshirani, Carnegie Mellon University

Workshop dates: April 20-21, 2021
Virtual event
Free registration
Workshop website: https://topml.rice.edu
Abstract submission deadline: February 18, 2021
Call for Contributions available at https://topml.rice.edu/call-for-contributions/