spikes within the coaching loss and their affect on generalization by characteristic studying

View a PDF of the paper titled Catapults in SGD: spikes within the coaching loss and their affect on generalization by characteristic studying, by Libin Zhu and three different authors

View PDF

Summary:On this paper, we first current an evidence concerning the frequent prevalence of spikes within the coaching loss when neural networks are skilled with stochastic gradient descent (SGD). We offer proof that the spikes within the coaching lack of SGD are “catapults”, an optimization phenomenon initially noticed in GD with massive studying charges in [Lewkowycz et al. 2020]. We empirically present that these catapults happen in a low-dimensional subspace spanned by the highest eigenvectors of the tangent kernel, for each GD and SGD. Second, we posit an evidence for a way catapults result in higher generalization by demonstrating that catapults promote characteristic studying by rising alignment with the Common Gradient Outer Product (AGOP) of the true predictor. Moreover, we show {that a} smaller batch measurement in SGD induces a bigger variety of catapults, thereby bettering AGOP alignment and take a look at efficiency.