香蕉视频

Title: Depth Degeneracy and Vanishing. Angles for Random Deep Neural Networks

Abstract: Stacking many layers to create truly *deep* neural networks is arguably what has led to the recent explosion in AI. However, many properties of deep neural networks are not yet understood. One such mystery is the depth degeneracy phenomenon: the deeper you make your network, the closer your network is to a constant function on initialization. In this talk, we examine the evolution of the angle between two inputs to a ReLU neural network as a function of the number of layers. By using combinatorial expansions, we find precise formulas for how fast this angle goes to zero as depth increases. The formulas are given in terms of the mixed moments of correlated Gaussians passed through the ReLU function. We also find a surprising combinatorial connection between these mixed moments and the Bessel numbers.

In person or by听Zoom link:

听

听