notesum.ai

Published at November 22

Adaptive Group Robust Ensemble Knowledge Distillation

Released Date: November 22, 2024

Authors: Patrik Kenfack¹, Ulrich Aïvodji¹, Samira Ebrahimi Kahou²

Aff.: ¹ÉTS Montréal, Mila; ²University of Calgary, Mila, Canada CIFAR AI Chair

Arxiv: http://arxiv.org/abs/2411.14984v1

Refer to caption

Models	Debiased	KD	Waterbirds		CelebA		CivilComment
Models	Debiased	KD	Average	WGA	Average	WGA	Average	WGA
Teacher	✗	✗	$85.7_{\pm 1.80}$	$65.6_{\pm 5.75}$	$95.4_{\pm 0.07}$	$37.5_{\pm 2.27}$	$90.0_{\pm 0.25}$	$75.9_{\pm 1.15}$
Teacher	✓	✗	$94.2_{\pm 0.53}$	$90.9_{\pm 1.00}$	$93.7_{\pm 2.44}$	$90.1_{\pm 1.68}$	$85.9_{\pm 0.49}$	$77.9_{\pm 0.49}$
Deep Ensemble	✗	✗	$84.4_{\pm 0.00}$	$59.3_{\pm 0.44}$	$95.6_{\pm 0.02}$	$37.2_{\pm 0.00}$	$90.6_{\pm 0.02}$	$76.1_{\pm 0.10}$
Deep Ensemble	✓	✗	$93.5_{\pm 0.17}$	$90.0_{\pm 0.56}$	$92.4_{\pm 0.18}$	$90.3_{\pm 0.55}$	$85.9_{\pm 0.17}$	$76.6_{\pm 0.09}$
One-hot	✗	✗	$83.8_{\pm 0.97}$	$54.1_{\pm 2.21}$	$95.5_{\pm 0.07}$	$32.3_{\pm 2.66}$	$90.1_{\pm 0.21}$	$75.5_{\pm 0.84}$
One-hot	✓	✗	$91.4_{\pm{1.31}}$	$86.7_{\pm 1.85}$	$92.3_{\pm 0.39}$	$88.9_{\pm 2.90}$	$85.7_{\pm 0.47}$	$76.5_{\pm 1.09}$
Random	✗	✓	$80.0_{\pm 0.37}$	$39.6_{\pm 3.63}$	$95.2_{\pm 0.03}$	$27.2_{\pm 0.00}$	$90.8_{\pm{0.21}}$	$\underline{75.2}_{\pm 0.95}$
Random	✓	✓	$89.6_{\pm 0.71}$	$77.3_{\pm 2.23}$	$\textbf{92.5}_{\pm 0.29}$	$85.1_{\pm 0.84}$	$\textbf{91.0}_{\pm 0.18}$	$74.2_{\pm 1.04}$
AVER	✗	✓	$79.2_{\pm 0.80}$	$46.4_{\pm 2.39}$	$\underline{95.5}_{\pm 0.05}$	$28.8_{\pm 0.78}$	$\underline{90.9}_{\pm 0.03}$	$74.7_{\pm 1.16}$
AVER	✓	✓	$90.8_{\pm 2.44}$	$82.9_{\pm 1.23}$	$92.4_{\pm 0.25}$	$83.4_{\pm 0.45}$	$90.8_{\pm 0.07}$	$75.0_{\pm 0.73}$
AE-KD	✗	✓	$81.7_{\pm 0.80}$	$46.9_{\pm 7.57}$	$95.3_{\pm 0.06}$	$30.5_{\pm 1.88}$	$90.8_{\pm 0.54}$	$73.1_{\pm 3.05}$
AE-KD	✓	✓	$90.9_{\pm 1.72}$	$85.0_{\pm 1.23}$	$92.3_{\pm 0.26}$	$87.5_{\pm 1.17}$	$90.7_{\pm 0.23}$	$74.8_{\pm 1.11}$
AGRE-KD (Ours)	✗	✓	$\underline{82.2}_{\pm 1.37}$	$\underline{55.0}_{\pm 5.47}$	$95.4_{\pm 0.04}$	$\underline{37.6}_{\pm 0.78}$	$89.3_{\pm 3.65}$	$74.7_{\pm 3.00}$
AGRE-KD (Ours)	✓	✓	$\textbf{91.3}_{\pm 0.49}$	$\textbf{87.9}_{\pm 1.23}$	$91.7_{\pm 0.20}$	$\textbf{91.9}_{\pm 0.71}$	$90.2_{\pm 0.49}$	$\textbf{75.9}_{\pm 1.75}$