notesum.ai

Published at November 17

F$^3$OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

cs.CV

cs.AI

Released Date: November 17, 2024

Authors: Pramit Saha¹, Felix Wagner¹, Divyanshu Mishra¹, Can Peng¹, Anshul Thakur¹, David Clifton², Konstantinos Kamnitsas³, J. Alison Noble¹

Aff.: ¹University of Oxford; ²University of Oxford; Oxford-Suzhou Centre for Advanced Research; ³University of Oxford; Imperial College London; University of Birmingham

Arxiv: http://arxiv.org/abs/2411.11912v1

Method	MBit	GFLOP	Param(M)	SLAKE	VM2019	VM2020	VM2021	VR	Overall
Full	3915.2	156.5	122.35	77.45	67.25	15.06	22.00	41.52	46.92
Adap (all)	28.8	85.2	0.90	72.82	64.45	11.56	21.00	38.35	43.17
FedDAT	86.1	86.3	2.69	72.79	63.58	12.46	23.00	38.91	43.93
LN	5.8	84.7	0.18	69.53	62.22	10.59	22.00	36.89	41.30
LoRA	19.5	84.6	0.61	57.73	60.18	4.59	15.00	31.16	35.09
Bias	3.5	84.7	0.11	68.25	55.61	11.17	17.00	35.47	39.54
PromptFL	19.2	85.0	0.60	65.63	57.56	4.78	15.00	40.26	36.65
$F^{3}OCUS$	9.7	80.6	0.30	74.69	60.05	12.84	24.00	42.30	42.78