An implicit spectral regularization perspective


Obtain a PDF of the paper titled The nice, the dangerous and the ugly sides of information augmentation: An implicit spectral regularization perspective, by Chi-Heng Lin and three different authors

Obtain PDF

Summary:Knowledge augmentation (DA) is a robust workhorse for bolstering efficiency in fashionable machine studying. Particular augmentations like translations and scaling in laptop imaginative and prescient are historically believed to enhance generalization by producing new (synthetic) knowledge from the identical distribution. Nonetheless, this conventional viewpoint doesn’t clarify the success of prevalent augmentations in fashionable machine studying (e.g. randomized masking, cutout, mixup), that significantly alter the coaching knowledge distribution. On this work, we develop a brand new theoretical framework to characterize the influence of a normal class of DA on underparameterized and overparameterized linear mannequin generalization. Our framework reveals that DA induces implicit spectral regularization via a mix of two distinct results: a) manipulating the relative proportion of eigenvalues of the information covariance matrix in a training-data-dependent method, and b) uniformly boosting the complete spectrum of the information covariance matrix via ridge regression. These results, when utilized to widespread augmentations, give rise to all kinds of phenomena, together with discrepancies in generalization between over-parameterized and under-parameterized regimes and variations between regression and classification duties. Our framework highlights the nuanced and typically stunning impacts of DA on generalization, and serves as a testbed for novel augmentation design.

Submission historical past

From: Chiraag Kaushik [view email]
Mon, 10 Oct 2022 21:30:46 UTC (8,655 KB)
Wed, 4 Jan 2023 17:45:56 UTC (8,661 KB)
Tue, 27 Feb 2024 20:55:18 UTC (9,098 KB)

Supply hyperlink


Please enter your comment!
Please enter your name here