notesum.ai

Published at November 25

Lion Cub: Minimizing Communication Overhead in Distributed Lion

cs.DC

Released Date: November 25, 2024

Authors: Satoki Ishikawa¹, Tal Ben-Nun², Brian Van Essen², Rio Yokota¹, Nikoli Dryden²

Aff.: ¹Institute of Science Tokyo; ²Lawrence Livermore National Laboratory

Arxiv: http://arxiv.org/abs/2411.16462v1

Refer to caption

Implementation	Latency cost	Bandwidth cost
Parameter server (naïve)	$2(P-1)\alpha$	$2PNb\beta$
Parameter server (efficient)	$2\log_{2}(P)\alpha$	$3\frac{P-1}{P}Nb\beta$
Direct allreduce	$2\log_{2}(P)\alpha$	$2\frac{P-1}{P}N(\log_{2}(P)+1)\beta$
1-bit compressed allreduce	$(P-1+\log_{2}(P))\alpha$	$(1+\frac{P-1}{P})N\beta$