notesum.ai

Published at December 3

cs.CL

Released Date: December 3, 2024

Authors: Da Ma¹, Lu Chen¹, Situo Zhang¹, Yuxun Miao¹, Su Zhu², Zhi Chen³, Hongshen Xu¹, Hanqi Li¹, Shuai Fan², Lei Pan², Kai Yu¹

Aff.: ¹X-LANCE Lab, Shanghai Jiao Tong University; ²AISpeech Co., Ltd.; ³ByteDance

Model	Window	LongBench						LEval
\cdashline3-12[1pt/1pt]	Window	SQA	MQA	Summ	Few-Shot	Code	Avg.	Closed	QA	Summ	Avg.
LLaMA3-8B-32K	32K	$32.94$	$32.23$	$25.41$	$69.30$	$66.54$	$45.28$	$42.10$	$24.68$	$15.55$	$27.45$
Token-selection-based methods
SnapKV	4K	$31.76$	$31.85$	$21.92$	$68.62$	$66.72$	$44.17$	$39.86$	$23.90$	$13.53$	$25.76$
PyramidKV	4K	$33.34$	$31.51$	$23.76$	$68.91$	$66.36$	$44.78$	$42.10$	$22.63$	$12.96$	$25.90$
Quest	4K	$32.14$	$32.19$	$24.27$	$69.05$	$66.43$	$44.82$	$40.55$	$25.59$	$14.69$	$26.94$
Token-eviction-based methods
LM-Infite	16+4080	$28.83$	$28.95$	$21.74$	$68.12$	$66.54$	$42.84$	$37.32$	$22.80$	$13.91$	$24.68$
StreamingLLM	16+4080	$28.68$	$28.95$	$21.64$	$68.14$	$66.60$	$42.80$	$37.12$	$22.79$	$13.81$	$24.57$
$\text{H}_{2}\text{O}$	96+4000	$29.36$	$29.51$	$22.73$	$68.45$	$66.17$	$43.24$	$37.15$	$23.18$	$13.48$	$24.60$
WA	4K	$8.90$	$3.63$	$9.05$	11.13	$41.08$	$14.76$	$20.95$	$5.57$	$2.79$	$9.77$
WA + CPT	4K	$26.94$	$27.95$	$22.29$	$66.60$	$66.10$	$41.97$	$32.94$	$22.09$	$12.55$	$22.52$
Layer-sharing-based methods
CLA	32K	$24.02$	$22.58$	$22.50$	$60.92$	$59.35$	$37.87$	$19.05$	$13.52$	$11.52$	$14.70$
\cdashline1-12 PoD (ours)	16+4080+28K	$30.97$	$32.43$	$24.82$	$67.30$	$68.26$	$44.75$	$43.59$	$22.95$	$15.00$	$27.18$
PoD+SnapKV (ours)	4K	$30.98$	$32.68$	$22.90$	$66.90$	$67.79$	$44.25$	$43.07$	$22.12$	$14.32$	$26.50$