how correlated latent variables speed up studying with neural networks

View a PDF of the paper titled Sliding down the steps: how correlated latent variables speed up studying with neural networks, by Lorenzo Bardone and Sebastian Goldt

View PDF
HTML (experimental)

Summary:Neural networks extract options from knowledge utilizing stochastic gradient descent (SGD). Specifically, higher-order enter cumulants (HOCs) are essential for his or her efficiency. Nonetheless, extracting info from the $p$th cumulant of $d$-dimensional inputs is computationally onerous: the variety of samples required to get well a single course from an order-$p$ tensor (tensor PCA) utilizing on-line SGD grows as $d^{p-1}$, which is prohibitive for high-dimensional inputs. This end result raises the query of how neural networks extract related instructions from the HOCs of their inputs effectively. Right here, we show that correlations between latent variables alongside the instructions encoded in several enter cumulants velocity up studying from higher-order correlations. We show this impact analytically by deriving almost sharp thresholds for the variety of samples required by a single neuron to weakly-recover these instructions utilizing on-line SGD from a random begin in excessive dimensions. Our analytical outcomes are confirmed in simulations of two-layer neural networks and unveil a brand new mechanism for hierarchical studying in neural networks.