Normalization effects on neural networks: generalization properties and high dimensions
Abstract
We consider neural networks and characterize their performance when trained with stochastic gradient descent as the number of hidden units N and gradient descent steps grow to infinity. In particular, we investigate the effect of normalizations (almost equivalently: initialization schemes) on the network's statistical output. We develop an asymptotic expansion for the neural network's statistical output, demonstrating that to leading order in N there is no bias-variance trade off and that to leading order in N the variance of the neural network's statistical output decays as the implied normalization by the scaling parameter approaches a certain scaling known as the mean field normalization. Numerical studies on the well known MNIST and CIFAR10 datasets show that generalization properties monotonically improve as the neural network's normalization gets closer to the mean field normalization. Motivated by this theory, we present a mesh free, which is key since meshes become infeasible in higher dimensions, deep learning algorithm for solving high dimensional partial differential equations (PDEs). The algorithm is being tested in a number of problems including option pricing in up to 500 dimensions. We call the algorithm a "Deep Galerkin Method (DGM)" since it is similar in spirit to Galerkin methods, with the solution approximated by a neural network instead of a linear combination of basis functions.
Watch recording ↗About the speaker
No bio is available for this talk.