notesum.ai

Published at November 6

cs.LG

cs.AI

Released Date: November 6, 2024

Authors: Alexandre Galashov¹, Michalis K. Titsias¹, András György¹, Clare Lyle¹, Razvan Pascanu¹, Yee Whye Teh², Maneesh Sahani³

Aff.: ¹Google DeepMind; ²University of Oxford; ³UCL Gatsby

Method	Comp. cost	Memory
SGD	$O(S)$	$O(P)$
Soft resets $\gamma$ per layer	$O(2S)$	$O(L+2P)$
Soft resets $\gamma$ per param.	$O(2S)$	$O(3P)$
Soft resets $\gamma$ per layer + proximal ( $K_{\theta}=10$ iters)	$O(20S)$	$O(L+2P)$
Soft resets $\gamma$ per param. + proximal ( $K_{\theta}$ iters)	$O(20S)$	$O(3P)$
Bayesian Soft Reset Proximal ( $K_{\theta}$ iters) $\gamma$ per layer	$O(30S)$	$P(L+3P)$
Bayesian Soft Reset Proximal ( $K_{\theta}$ iters) $\gamma$ per param.	$O(30S)$	$P(4)$