notesum.ai

Published at October 23

Closed-form merging of parameter-efficient modules for Federated Continual Learning

cs.CL

cs.AI

cs.LG

Released Date: October 23, 2024

Authors: Riccardo Salami¹, Pietro Buzzega¹, Matteo Mosconi¹, Jacopo Bonato², Luigi Sabetta², Simone Calderara¹

Aff.: ¹AImageLab - University of Modena and Reggio Emilia, Modena, Italy; ²Leonardo Labs, Roma, Italy

Arxiv: https://arxiv.org/abs/2410.17961v1

Refer to caption

	CIFAR-100			ImageNet-R
Joint	$91.59$			$86.31$
Partition	$\beta=0.5$	$\beta=0.1$	$\beta=0.05$	$\beta=0.5$	$\beta=0.1$	$\beta=0.05$
EWC (Kirkpatrick et al., 2017)	$71.31$	$71.75$	$63.48$	$56.77$	$55.67$	$53.90$
LwF (Li & Hoiem, 2017)	$54.18$	$55.12$	$42.44$	$49.78$	$45.03$	$41.60$
DER++ (Buzzega et al., 2020)	$67.90$	$61.97$	$55.79$	$55.83$	$50.07$	$45.52$
L2P (Wang et al., 2022b)	$\underline{76.30}$	$52.82$	$45.41$	$57.58$	$46.82$	$37.57$
CODA-P (Smith et al., 2023)	$62.00$	$22.21$	$16.12$	$48.02$	$29.68$	$21.52$
FisherAvg (Matena & Raffel, 2022)	$70.43$	$70.10$	$67.87$	$57.05$	$54.05$	$51.55$
RegMean (Jin et al., 2023)	$71.07$	$64.56$	$61.47$	$61.05$	$57.32$	$49.22$
CCVR (Luo et al., 2021)	$73.56$	$\underline{78.04}$	$\underline{79.47}$	$\underline{68.68}$	$\underline{69.55}$	$\bm{71.83}$
TARGET (Zhang et al., 2023b)	$69.36$	$69.18$	$64.23$	$57.32$	$58.75$	$55.70$
PILoRA (Guo et al., 2024)	$75.34$	$73.27$	$68.34$	$53.03$	$51.40$	$47.33$
LoRM (ours)	$\bm{83.19}$	$\bm{82.05}$	$\bm{81.21}$	$\bm{75.62}$	$\bm{70.83}$	$\underline{70.18}$