notesum.ai

Published at October 20

Lossless KV Cache Compression to 2%

cs.CV

cs.AI

cs.MA

Released Date: October 20, 2024

Authors: Zhen Yang¹, J. N. Han¹, Kan Wu¹, Ruobing Xie¹, An Wang², Xingwu Sun¹, Zhanhui Kang¹

Aff.: ¹Tencent Hunyuan; ²Tencent Hunyuan, Tokyo Institute of Technology

Arxiv: https://arxiv.org/abs/2410.15252v1

Refer to caption

Method	KV Cache Memory
MHA	$\mathcal{O}\left(BSLND_{h}\right)$
GQA	$\mathcal{O}\left(BSLGD_{h}\right)$
MLA	$\mathcal{O}\left(BSLC\right)$
CLA	$\mathcal{O}\left(BSLND_{h}/F\right)$
CLLA	$\mathcal{O}\left(BSLC/F\right)$
CLLA-quant	$\mathcal{O}\left(BSLC/4F\right)$