notesum.ai

Published at November 29

KV Shifting Attention Enhances Language Modeling

cs.CL

Released Date: November 29, 2024

Authors: Mingyu Xu¹, Wei Cheng¹, Bingning Wang¹, Weipeng Chen¹

Aff.: ¹Baichuan Inc.

Arxiv: http://arxiv.org/pdf/2411.19574v1

Refer to caption

Model	Tokens	lambada	winogrande	hellaswag	Arc-E	Arc-C	cmmlu	mmlu	math	average
Vanilla - 2.9B	340B	52.92	52.09	42.70	27.45	25.97	28.51	29.43	0.80	32.48
	420B	52.80	54.85	43.68	28.96	26.02	34.77	30.34	1.20	34.08
	500B	51.66	54.06	44.49	36.20	27.90	38.22	37.26	1.80	36.45
KV Shiting - 2.9B	340B	55.44	53.91	42.87	36.74	30.04	34.51	36.20	2.00	36.46
	420B	51.91	54.78	43.83	36.66	31.91	37.24	34.30	1.80	36.55
	500B	54.51	55.33	44.52	39.02	30.89	40.78	40.88	2.60	38.57
Vanilla - 19B	160B	59.93	48.22	48.25	30.34	24.56	39.12	39.22	1.80	36.43
	180B	58.80	48.07	47.78	31.28	25.99	40.80	39.34	2.60	36.83
	200B	60.88	49.01	47.36	33.25	25.78	42.92	42.68	2.60	38.06
KV shifting - 19B	160B	61.93	48.46	48.28	31.25	25.06	42.10	42.87	2.00	37.74
	180B	60.20	47.67	48.16	32.45	26.55	43.38	40.49	3.00	37.74
	200B	62.35	48.38	48.42	33.28	29.32	42.40	43.29	3.20	38.83