notesum.ai

Published at October 23

Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation

cs.CL

cs.AI

Released Date: October 23, 2024

Authors: Muquan Li¹, Dongyang Zhang¹, Tao He¹, Xiurui Xie¹, Yuan-Fang Li², Ke Qin¹

Aff.: ¹University of Electronic Science and Technology of China; ²Monash University

Arxiv: https://arxiv.org/abs/2410.17606v1

Refer to caption

			Test accuracy (%)
Dataset	Teacher	Student	T.	S.	DAFL	ZSKT	ADI	DFQ	CMI	SpaceshipNet	DDA
	resnet-34	resnet-18	95.70	95.20	92.22	93.32	93.26	94.61	94.84	95.39	95.64
	vgg-11	resnet-18	92.25	95.20	81.10	89.46	90.36	90.84	91.13	92.27	93.02
	wrn-40-2	wrn-16-1	94.87	91.12	65.71	83.74	83.04	86.14	90.01	90.38	90.92
	wrn-40-2	wrn-40-1	94.87	93.94	81.33	86.07	86.85	91.69	92.78	93.56	93.63
CIFAR-10	wrn-40-2	wrn-16-2	94.87	93.95	81.55	89.66	89.72	92.01	92.52	93.25	93.51
	resnet-34	resnet-18	78.05	77.10	74.47	67.74	61.32	77.01	77.04	77.41	77.56
	vgg-11	resnet-18	71.32	77.10	57.29	34.72	54.13	68.32	70.56	71.41	72.04
	wrn-40-2	wrn-16-1	75.83	65.31	22.50	30.15	53.77	54.77	57.91	58.06	58.96
	wrn-40-2	wrn-40-1	75.83	72.19	34.66	29.73	61.33	61.92	68.88	68.78	69.31
CIFAR-100	wrn-40-2	wrn-16-2	75.83	73.56	40.00	28.44	61.34	59.01	68.75	69.95	70.27
Tiny-ImageNet	resnet-34	resnet-18	66.44	64.87	N/A	N/A	N/A	63.73	64.01	64.04	64.13