notesum.ai

Published at December 4

ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

cs.LG

cs.AI

cs.PF

Released Date: December 4, 2024

Authors: Guangda Liu¹, Chengwei Li, Jieru Zhao, Chenqi Zhang, Minyi Guo

Aff.: ¹Shanghai Jiao Tong University

Arxiv: http://arxiv.org/pdf/2412.03213v1

Refer to caption

	256	512	1024	2048
Quest	35.63	40.83	43.23	45.59
InfiniGen	43.69	45.04	45.13	45.14
ClusterKV (ours)	46.69	48.02	48.34	48.7
Full KV	49.01