notesum.ai

Published at October 28

EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

cs.LG

cs.AI

Released Date: October 28, 2024

Authors: Shih-Yang Liu, Huck Yang, Chein-Yi Wang, Nai Chit Fung, Hongxu Yin, Charbel Sakr, Saurav Muralidharan, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, Pavlo Molchanov, Min-Hung Chen

Arxiv: https://arxiv.org/abs/2410.21271v1

Model	Sparsity	Compensation Method	Wikitext2	ARC-E	ARC-C	MathQA
LLaMA3-8B	Uncompressed	-	6.13	80.09	50.42	40.10
	50%	-	8.25	72.13	39.84	32.69
		SVD	7.99	73.90	41.38	32.96
		EoRA	7.98 (-0.01)	75.88 (+1.98)	43.60 (+2.22)	34.90 (+1.94)
	60%	-	12.00	63.38	30.54	27.00
		SVD	10.93	64.64	30.97	28.40
		EoRA	10.71 (-0.22)	68.77 (+4.13)	34.98 (+4.01)	31.62 (+3.22)
	2:4	-	12.32	62.75	30.11	26.43
		SVD	11.31	64.89	31.99	26.49
		EoRA	11.07 (-0.24)	68.22 (+3.33)	34.64 (+2.65)	29.91 (+3.42)
LLaMA2-7B	Uncompressed	-	5.47	69.31	39.84	27.67
	50%	-	6.48	64.14	35.92	26.90
		SVD	6.34	63.51	36.26	26.39
		EoRA	6.31 (-0.03)	66.45 (+2.94)	38.22 (+1.96)	27.10 (+0.71)
	60%	-	8.35	59.72	30.11	25.15
		SVD	7.81	61.61	32.42	25.09
		EoRA	7.69 (-0.12)	62.66 (+1.05)	34.12 (+1.70)	25.99 (+0.9)
	2:4	-	8.77	60.47	30.11	24.65
		SVD	8.15	60.98	30.54	24.89
		EoRA	7.97 (-0.18)	63.42 (+2.44)	32.67 (+2.13)	25.59 (+0.70)
LLaMA2-13B	Uncompressed	-	4.88	73.23	45.56	29.91
	50%	-	5.65	68.81	39.24	27.30
		SVD	5.54	69.69	39.59	27.63
		EoRA	5.54	71.63 (+1.94)	41.97 (+2.38)	28.27 (+0.64)
	60%	-	6.93	63.21	33.70	26.86
		SVD	6.59	65.44	34.12	26.06
		EoRA	6.52 (-0.07)	67.25 (+1.81)	37.71 (+3.59)	27.16 (+1.10)
	2:4	-	7.10	66.32	34.30	25.92
		SVD	6.82	66.28	33.61	25.12
		EoRA	6.75 (-0.07)	68.47 (+2.19)	37.54 (+3.93)	27.53 (+2.41)