notesum.ai

Published at November 2

cs.DC

cs.AI

cs.LG

Released Date: November 2, 2024

Authors: Ege Erdil¹, David Schneider-Joseph²

Aff.: ¹EpochAI; ²David Schneider-Joseph

	Network bandwidth per gradient step	Slices along…	Communications can coincide with…
Data parallelism	$2N_{\text{p}}(N_{\text{DP}}-1)$	$b$	Nothing
Tensor parallelism	$\approx 8bL\sqrt{{d_{\text{ff}}}{d_{\text{model}}}N_{\text{TP}}}$ (for large $N_{\text{TP}}$ )	${d_{\text{model}}},{d_{\text{ff}}}$	Nothing
Pipeline parallelism	$2b{d_{\text{model}}}(N_{\text{PP}}\cdot i-1)$	$b,L$	Expert parallelism
Expert parallelism	$2b{d_{\text{model}}}(L-1)\cdot(N_{\text{EP}}-1)/N_{\text{EP}}$	$E$	Pipeline parallelism