notesum.ai

Published at November 18

Distill the Best, Ignore the Rest: Improving Dataset Distillation with Loss-Value-Based Pruning

cs.CV

cs.AI

Released Date: November 18, 2024

Authors: Brian B. Moser¹, Federico Raue¹, Tobias C. Nauen¹, Stanislav Frolov¹, Andreas Dengel²

Aff.: ¹German Research Center for Artificial Intelligence; ²German Research Center for Artificial Intelligence, University of Kaiserslautern-Landau

Arxiv: http://arxiv.org/abs/2411.12115v1

Distillation Space	Algorithm	ImNet-A	ImNet-B	ImNet-C	ImNet-D	ImNet-E	ImNette	ImWoof	ImNet-Birds	ImNet-Fruits	ImNet-Cats
	MTT	33.4 $\pm$ 1.5	34.0 $\pm$ 3.4	31.4 $\pm$ 3.4	27.7 $\pm$ 2.7	24.9 $\pm$ 1.8	24.1 $\pm$ 1.8	16.0 $\pm$ 1.2	25.5 $\pm$ 3.0	18.3 $\pm$ 2.3	18.7 $\pm$ 1.5
pixel space	DC	38.7 $\pm$ 4.2	38.7 $\pm$ 1.0	33.3 $\pm$ 1.9	26.4 $\pm$ 1.1	27.4 $\pm$ 0.9	28.2 $\pm$ 1.4	17.4 $\pm$ 1.2	28.5 $\pm$ 1.4	20.4 $\pm$ 1.5	19.8 $\pm$ 0.9
	DM	27.2 $\pm$ 1.2	24.4 $\pm$ 1.1	23.0 $\pm$ 1.4	18.4 $\pm$ 0.7	17.7 $\pm$ 0.9	20.6 $\pm$ 0.7	14.5 $\pm$ 0.9	17.8 $\pm$ 0.8	14.5 $\pm$ 1.1	14.0 $\pm$ 1.1
	MTT	39.9 $\pm$ 1.2	39.4 $\pm$ 1.3	34.9 $\pm$ 1.1	30.4 $\pm$ 1.5	29.0 $\pm$ 1.1	30.4 $\pm$ 1.5	17.1 $\pm$ 1.1	28.2 $\pm$ 1.1	21.1 $\pm$ 1.2	19.6 $\pm$ 1.2
GLaD	DC	41.8 $\pm$ 1.7	42.1 $\pm$ 1.2	35.8 $\pm$ 1.4	28.0 $\pm$ 0.8	29.3 $\pm$ 1.3	31.0 $\pm$ 1.6	17.8 $\pm$ 1.1	29.1 $\pm$ 1.0	22.3 $\pm$ 1.6	21.2 $\pm$ 1.4
	DM	31.6 $\pm$ 1.4	31.3 $\pm$ 3.9	26.9 $\pm$ 1.2	21.5 $\pm$ 1.0	20.4 $\pm$ 0.8	21.9 $\pm$ 1.1	15.2 $\pm$ 0.9	18.2 $\pm$ 1.0	20.4 $\pm$ 1.6	16.1 $\pm$ 0.7
	MTT	40.4 $\pm$ 1.6	42.4 $\pm$ 1.3	35.4 $\pm$ 1.1	30.5 $\pm$ 1.3	30.0 $\pm$ 1.1	30.5 $\pm$ 1.0	20.3 $\pm$ 1.3	27.7 $\pm$ 1.6	21.8 $\pm$ 1.1	21.2 $\pm$ 0.8
		+0.5+0.4	+3.0+0.0	+0.6+0.0	+0.1-0.2	+1.0+0.0	+0.1-0.5	+3.2+0.2	-0.5-0.5	+0.7-0.1	+0.6-0.4
GLaD	DC	42.8 $\pm$ 1.9	44.6 $\pm$ 1.0	37.6 $\pm$ 2.0	29.5 $\pm$ 1.7	32.8 $\pm$ 1.0	33.6 $\pm$ 1.0	19.3 $\pm$ 1.2	30.9 $\pm$ 0.9	23.3 $\pm$ 1.1	22.4 $\pm$ 0.8
(pruned)		+1.0+0.5	+2.5-0.2	+2.2+0.9	+1.5+0.9	+3.5-0.3	+2.6-0.6	+1.5-0.1	+1.8-0.1	+1.0-0.5	+1.2-0.6
	DM	36.8 $\pm$ 1.6	36.5 $\pm$ 1.6	30.8 $\pm$ 1.2	23.1 $\pm$ 1.4	22.8 $\pm$ 2.0	24.9 $\pm$ 0.8	16.1 $\pm$ 0.9	21.4 $\pm$ 0.9	22.7 $\pm$ 1.1	16.3 $\pm$ 1.0
		+5.2+0.2	+5.2-2.3	+3.9+0.0	+1.6+0.4	+2.4+1.2	+3.0-0.3	+0.9+0.0	+3.2-0.1	+2.3-0.5	+0.2+0.3
	MTT	40.9 $\pm$ 1.1	41.6 $\pm$ 1.7	34.1 $\pm$ 1.7	31.5 $\pm$ 1.2	30.1 $\pm$ 1.3	32.0 $\pm$ 1.3	19.9 $\pm$ 1.2	30.4 $\pm$ 1.5	21.4 $\pm$ 1.1	22.1 $\pm$ 1.0
LD3M	DC	42.3 $\pm$ 1.3	42.0 $\pm$ 1.1	37.1 $\pm$ 1.8	29.7 $\pm$ 1.3	31.4 $\pm$ 1.1	32.9 $\pm$ 1.2	18.9 $\pm$ 0.6	30.2 $\pm$ 1.4	22.6 $\pm$ 1.3	21.7 $\pm$ 0.8
	DM	35.8 $\pm$ 1.1	34.1 $\pm$ 1.0	30.3 $\pm$ 1.2	24.7 $\pm$ 1.0	24.5 $\pm$ 0.9	26.8 $\pm$ 1.7	18.1 $\pm$ 0.7	23.0 $\pm$ 1.8	24.5 $\pm$ 1.9	17.0 $\pm$ 1.1
	MTT	41.2 $\pm$ 1.5	42.4 $\pm$ 1.2	36.0 $\pm$ 1.4	31.7 $\pm$ 1.4	29.9 $\pm$ 0.9	33.8 $\pm$ 1.6	19.9 $\pm$ 1.2	29.5 $\pm$ 1.1	23.3 $\pm$ 1.2	22.3 $\pm$ 0.9
		+0.3+0.5	+0.8-0.5	+1.9-0.3	0.2+0.2	-0.2-0.4	+1.8+0.3	+0.0+0.0	-0.9-0.4	+1.9+0.1	+0.2-0.1
LD3M	DC	42.8 $\pm$ 1.8	44.0 $\pm$ 1.4	37.8 $\pm$ 1.1	29.7 $\pm$ 0.9	33.1 $\pm$ 1.3	34.4 $\pm$ 1.3	19.4 $\pm$ 0.9	31.6 $\pm$ 1.3	24.0 $\pm$ 1.0	22.5 $\pm$ 1.1
(pruned)		+0.5+0.5	+2.0+0.3	+0.7-0.7	+0.0-0.4	+1.7+0.2	+1.5+0.1	+0.5+0.3	+1.4-0.1	+1.4-0.3	+0.8+0.3
	DM	38.8 $\pm$ 1.2	34.9 $\pm$ 1.4	29.7 $\pm$ 1.6	26.0 $\pm$ 1.6	27.2 $\pm$ 0.9	29.6 $\pm$ 1.3	18.4 $\pm$ 1.0	24.5 $\pm$ 1.3	24.5 $\pm$ 1.4	17.3 $\pm$ 1.2
		+3.0+0.1	+0.8+0.4	-0.5+0.4	+1.3+0.6	+2.7+0.0	+2.8-0.4	+0.3+0.3	+1.5-0.5	+0.0-0.5	+0.3+0.1