[2403.03562] Environment friendly Algorithms for Empirical Group Distributionally Strong Optimization and Past

View a PDF of the paper titled Environment friendly Algorithms for Empirical Group Distributionally Strong Optimization and Past, by Dingzhi Yu and three different authors

View PDF
HTML (experimental)

Summary:On this paper, we examine the empirical counterpart of Group Distributionally Strong Optimization (GDRO), which goals to reduce the maximal empirical danger throughout $m$ distinct teams. We formulate empirical GDRO as a $textit{two-level}$ finite-sum convex-concave minimax optimization drawback and develop an algorithm known as ALEG to profit from its particular construction. ALEG is a double-looped stochastic primal-dual algorithm that comes with variance discount strategies right into a modified mirror prox routine. To take advantage of the two-level finite-sum construction, we suggest a easy group sampling technique to assemble the stochastic gradient with a smaller Lipschitz fixed after which carry out variance discount for all teams. Theoretical evaluation exhibits that ALEG achieves $varepsilon$-accuracy inside a computation complexity of $mathcal{O}left(frac{msqrt{bar{n}ln{m}}}{varepsilon}proper)$, the place $bar n$ is the typical variety of samples amongst $m$ teams. Notably, our strategy outperforms the state-of-the-art methodology by an element of $sqrt{m}$. Based mostly on ALEG, we additional develop a two-stage optimization algorithm known as ALEM to take care of the empirical Minimax Extra Danger Optimization (MERO) drawback. The computation complexity of ALEM almost matches that of ALEG, surpassing the charges of present strategies.