Sharp Nonasymptotics and Close to-Optimum Asymptotics in a Single Algorithm

View a PDF of the paper titled ROOT-SGD: Sharp Nonasymptotics and Close to-Optimum Asymptotics in a Single Algorithm, by Chris Junchi Li and three different authors

View PDF

Summary:We research the issue of fixing strongly convex and clean unconstrained optimization issues utilizing stochastic first-order algorithms. We devise a novel algorithm, known as Recursive One-Over-T SGD (ROOT-SGD), based mostly on an simply implementable, recursive averaging of previous stochastic gradients. We show that it concurrently achieves state-of-the-art efficiency in each a finite-sample, nonasymptotic sense and an asymptotic sense. On the non-asymptotic facet, we show threat bounds on the final iterate of ROOT-SGD with leading-order phrases that match the optimum statistical threat with a unity pre-factor, together with a higher-order time period that scales on the sharp fee of $O(n^{-3/2})$ below the Lipschitz situation on the Hessian matrix. On the asymptotic facet, we present that when a gentle, one-point Hessian continuity situation is imposed, the rescaled final iterate of (multi-epoch) ROOT-SGD converges asymptotically to a Gaussian restrict with the Cramér-Rao optimum asymptotic covariance, for a broad vary of step-size selections.