Computation in Very Wide Neural Networks
Explore the computational aspects of extremely wide neural networks in this 49-minute lecture by Yasaman Bahri from Google Brain. Delve into topics such as single-hidden layer neural networks of infinite width, Gaussian Process (GP) correspondences, and Bayesian inference with GP priors. Examine experimental results comparing neural network Gaussian process (NNGP) performance across hyperparameters, large depth behavior, and fixed points. Analyze phase diagrams, performance trends with width and dataset size, and empirical comparisons of various NN-GPs. Investigate the dynamics occurring in parameter space and gain insights into the best-performing networks, comparing GPs and SGD-trained neural networks. Part of the Frontiers of Deep Learning series at the Simons Institute.