notesum.ai

Published at November 21

Separable Mixture of Low-Rank Adaptation for Continual Visual Instruction Tuning

cs.CV

cs.AI

Released Date: November 21, 2024

Authors: Ziqi Wang¹, Chang Che¹, Qi Wang², Yangyang Li³, Zenglin Shi¹, Meng Wang¹

Aff.: ¹Hefei University of Technology; ²Tsinghua University; ³Academy of Cyber

Arxiv: http://arxiv.org/abs/2411.13949v1

Refer to caption

	Method	Accuracy on Each Task						Overall Results
	Method	ScienceQA	TextVQA	Flickr30k	ImageNet*	GQA	VQAv2	AP $\uparrow$	MAP $\uparrow$	BWT $\uparrow$	MIF $\uparrow$
Single-type	Multitask	83.49	61.93	169.21	96.53	60.07	65.80	89.51	-	-	98.38
	Zero-shot	52.72	2.95	52.64	22.10	2.73	0.65	22.30	-	-	17.84
	DirLoRA^†	83.75	60.66	164.20	96.71	58.55	64.93	88.13	-	0.00	98.41
	SeqLoRA [14]	55.31	50.22	33.89	22.73	50.52	64.37	46.21	57.41	-48.10	78.35
	MoeLoRA [3]	55.01	48.87	32.04	22.00	50.03	63.64	45.27	56.16	-48.05	79.97
	EWC [19]	57.04	50.02	32.96	22.85	50.16	64.54	46.26	56.19	-49.71	78.90
	Replay [2]	75.61	47.58	31.97	35.84	48.51	58.67	49.70	69.78	-22.71	82.06
	DoRA [25]	51.26	46.36	36.41	28.24	45.29	56.87	44.07	65.03	-31.12	78.59
	SMoLoRA(Ours)	77.36	58.29	151.99	95.35	51.96	65.71	83.44	84.85	-3.23	97.79
Multi-type	Multitask	82.83	61.27	175.97	96.44	59.67	65.80	90.33	-	-	97.93
	Zero-shot	51.85	5.11	44.05	20.34	2.37	1.16	20.81	-	-	19.45
	DirLoRA^†	83.85	60.51	164.66	96.71	57.93	64.90	88.09	-	0.00	98.31
	SeqLoRA [14]	59.21	50.80	20.99	20.30	49.98	64.41	44.28	53.75	-48.73	79.47
	MoeLoRA [3]	58.09	53.30	22.82	22.61	51.80	65.15	45.63	54.88	-49.94	78.19
	EWC [19]	53.60	49.07	20.38	20.48	50.11	64.63	43.10	53.94	-52.47	78.18
	Replay [2]	66.06	47.78	24.21	25.66	46.53	58.59	44.81	66.68	-26.88	80.38
	DoRA [25]	52.03	47.37	27.97	26.18	46.05	57.33	42.82	56.24	-34.11	78.30
	SMoLoRA(Ours)	80.50	58.30	146.63	94.28	52.42	65.96	83.02	85.05	-6.50	98.12