notesum.ai

Published at December 9

BoRA: Bi-dimensional Weight-Decomposed Low-Rank Adaptation

cs.CL

Released Date: December 9, 2024

Authors: Qiushi Wang¹, Yuchen Fan, Junwei Bao, Hongfei Jiang, Yang Song

Aff.: ¹Zuoyebang Education Technology Co., Ltd.

Arxiv: http://arxiv.org/pdf/2412.06441v1

Refer to caption

Model	Method	Param(%)	MT-Bench
Llama-2-7b	LoRA	2.32	6.16
	DoRA	2.33	6.38
	rsLoRA	2.32	6.24
	AsyLoRA*	1.26	6.08
	AsyLoRA	2.47	6.20
	PiSSA	2.32	5.58
	MoRA	2.32	5.98
	BoRA	2.35	6.76
Mistral-7b-v0.1	LoRA	2.26	6.75
	DoRA	2.28	6.68
	BoRA	2.30	6.81
Llama-3-8b	LoRA	2.05	6.88
	DoRA	2.06	6.85
	BoRA	2.08	6.99