notesum.ai

Published at December 6

Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference

cs.AI

Released Date: December 6, 2024

Authors: Qingyuan Li¹, Bo Zhang, Liang Ye, Yifan Zhang, Wei Wu, Yerui Sun, Lin Ma, Yuchen Xie

Aff.: ¹Meituan

Arxiv: http://arxiv.org/pdf/2412.04964v1

Refer to caption

Method	Ring All-Reduce	Flash All-Reduce
Total Volume	$2M(N-1)/N$	$2M(N-1)/N$
Reduce Step	$N-1$	$1$
Reduce-Scatter	$M/N$	$M(N-1)/N$
Gather Step	$N-1$	$1$
All-Gather	$M/N$	$M(N-1)/N$
QDQ Step	$N$	$2$