notesum.ai

Published at November 29

Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning

cs.CL

cs.AI

cs.LG

Released Date: November 29, 2024

Authors: Kaustubh Ponkshe¹, Raghav Singhal¹, Eduard Gorbunov¹, Alexey Tumanov², Samuel Horvath¹, Praneeth Vepakomma³

Aff.: ¹Affiliation 1; ²Affiliation 2; ³Affiliation 1 and 3

Arxiv: http://arxiv.org/pdf/2411.19557v1

Refer to caption

Model	Method	Rank	# Params	Accuracy ( $\uparrow$ )
Model	Method	Rank	# Params	BoolQ	PIQA	SIQA	HellaS.	WinoG.	ARC-e	ARC-c	OBQA	Avg.
Llama-3.2 3B	Full FT	-	$3.21$ B	$70.43$	$85.64$	$80.45$	$91.92$	$85.02$	$88.52$	$75.29$	$81.88$	$82.39$
	LoRA	$32$	$48.63$ M	$70.03$	$\mathbf{85.20}$	$79.12$	$90.71$	$82.24$	$86.91$	$74.32$	$\mathbf{81.87}$	$81.44$
	LoRA-XS	$32$	$0.20$ M	$65.01$	$82.87$	$76.17$	$87.32$	$80.12$	$84.78$	$70.31$	$75.71$	$77.79$
	LoRA-XS	$64$	$0.80$ M	$66.53$	$83.12$	$77.98$	$88.53$	$81.76$	$85.15$	$72.04$	$77.14$	$79.03$
	LoRA-XS	$96$	$1.81$ M	$67.28$	$83.35$	$78.66$	$88.99$	$82.08$	$85.18$	$72.61$	$78.88$	$79.63$
	LoRA-SB	$32$	$0.20$ M	$66.33$	$84.06$	$78.91$	$89.04$	$81.37$	$86.62$	$72.44$	$76.97$	$79.47$
	LoRA-SB	$64$	$0.80$ M	$68.35$	$84.55$	$79.94$	$\mathbf{91.68}$	$83.03$	$87.84$	$74.83$	$80.12$	$81.29$
	LoRA-SB	$96$	$1.81$ M	$\mathbf{70.34}$	$84.76$	$\mathbf{80.19}$	$91.62$	$\mathbf{84.61}$	$\mathbf{87.92}$	$\mathbf{74.74}$	$81.20$	$\mathbf{81.92}$