notesum.ai

Published at November 4

xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism

cs.DC

cs.AI

Released Date: November 4, 2024

Authors: Jiarui Fang¹, Jinzhe Pan², Xibo Sun¹, Aoyu Li¹, Jiannan Wang³

Aff.: ¹Tencent; ²Tencent & Huazhong University of Science and Technology; ³Tencent & The University of Hong Kong

Arxiv: http://arxiv.org/abs/2411.01738v1

Refer to caption

Method	Communication		Memory Cost
Method	Cost	Overlap	Model	KV Activations
Tensor Parallelism	$4O(p\times hs)L$	✗	$\frac{1}{N}P$	$\frac{1}{N}KV$
DistriFusion	$2O(p\times hs)L$	✓	$P$	$(KV)L$
SP-Ring	$2O(p\times hs)L$	✓	$P$	$\frac{1}{N}KV$
SP-Ulysses	$\frac{4}{N}O(p\times hs)L$	✗	$P$	$\frac{1}{N}KV$
PipeFusion	2 $O(p\times hs)$	✓	$\frac{1}{N}P$	$\frac{1}{N}(KV)L$