notesum.ai

Published at November 12

cs.LG

cs.AI

Released Date: November 12, 2024

Authors: Yilun Liu¹, Yunpu Ma², Shuo Chen², Zifeng Ding³, Bailan He², Zhen Han², Volker Tresp²

Aff.: ¹Technical University of Munich; ²Ludwig Maximilian University of Munich; ³Ludwig Maximilian University of Munich and University of Cambridge

LLM	Arch.	Strategy	# Act.	% Act.	CR	AR
	LoRA₄	$\bm{W}_{q},\bm{W}_{v}$ @Attn	0.52M	0.041	57.15	28.42
	LoRA₁₆	PERFT-R (Top1/2)	0.59M	0.046	66.66	31.91
	LoRA₈	PERFT-R (Top2/2)	0.59M	0.046	66.98	31.18
OLMoE 1B-7B (Top8/64)	LoRA₁₆	$\bm{W}_{q},\bm{W}_{v}$ @Attn	2.10M	0.164	62.86	29.71
	LoRA₄	PERFT-E (Top8/64)	2.10M	0.164	69.42	31.30
	LoRA₃₂	PERFT-R (Top1/4)	2.23M	0.174	67.32	32.29
	LoRA₆₄	$\bm{W}_{q},\bm{W}_{v}$ @Attn	8.39M	0.654	67.95	28.82
	LoRA₁₆	PERFT-E (Top8/64)	8.39M	0.654	69.29	29.08
	LoRA₁₆	PERFT-R (Top8/8)	8.65M	0.675	68.81	31.65
Mixtral 13B-47B (Top2/8)	LoRA₈	$\bm{W}_{q},\bm{W}_{v}$ @Attn	3.41M	0.026	85.02	64.72
	LoRA₈	PERFT-R (Top2/2)	4.46M	0.035	86.23	69.03
	LoRA₈	PERFT-R (Top2/8)	5.24M	0.046	85.68	68.14