notesum.ai
Published at November 25Lion Cub: Minimizing Communication Overhead in Distributed Lion
cs.DC
Released Date: November 25, 2024
Authors: Satoki Ishikawa1, Tal Ben-Nun2, Brian Van Essen2, Rio Yokota1, Nikoli Dryden2
Aff.: 1Institute of Science Tokyo; 2Lawrence Livermore National Laboratory

| Implementation | Latency cost | Bandwidth cost |
| Parameter server (naïve) | ||
| Parameter server (efficient) | ||
| Direct allreduce | ||
| 1-bit compressed allreduce |