notesum.ai

Published at November 8

cs.CL

cs.AI

Released Date: November 8, 2024

Authors: Esther Gan¹, Yiran Zhao¹, Liying Cheng², Yancan Mao¹, Anirudh Goyal³, Kenji Kawaguchi¹, Min-Yen Kan¹, Michael Shieh¹

Aff.: ¹National University of Singapore; ²Singapore University of Technology and Design; ³Google DeepMind

Dataset	Model (#Params)	Ori.	Avg-ATA	ATA-1	ATA-2	ATA-4	ATA-8
GSM8K	Gemma-2B (2.5B)	$15.1$	$8.1$ ( $\downarrow 7.0$ )	$11.2$	$9.4$	$7.1$	$4.6$
	Llama2-7B (6.7B)	$27.3$	$16.7$ ( $\downarrow 10.6$ )	$21.8$	$19.7$	$14.7$	$10.6$
	Mistral-7B (7.2B)	$43.7$	$30.1$ ( $\downarrow 13.6$ )	$38.6$	$35.4$	$27.1$	$19.2$
	Gemma-7B (8.5B)	$39.9$	$32.1$ ( $\downarrow 7.8$ )	$38.7$	$36.8$	$29.8$	$23.1$
BBH	Gemma-2B (2.5B)	$29.6$	$20.8$ ( $\downarrow 8.8$ )	$24.7$	$21.9$	$20.2$	$16.4$
	Llama2-7B (6.7B)	$35.7$	$28.1$ ( $\downarrow 7.6$ )	$32.2$	$30.1$	$26.8$	$23.3$
	Mistral-7B (7.2B)	$50.0$	$40.9$ ( $\downarrow 9.1$ )	$46.8$	$43.1$	$39.1$	$34.6$
	Gemma-7B (8.5B)	$42.4$	$35.9$ ( $\downarrow 6.5$ )	$40.6$	$38.1$	$33.5$	$31.3$
MMLU	Gemma-2B (2.5B)	$34.1$	$27.5$ ( $\downarrow 6.6$ )	$30.3$	$29.7$	$27.5$	$22.6$
	Llama2-7B (6.7B)	$35.1$	$29.5$ ( $\downarrow 5.6$ )	$31.6$	$30.2$	$28.9$	$27.5$
	Mistral-7B (7.2B)	$54.6$	$47.0$ ( $\downarrow 7.6$ )	$51.1$	$49.3$	$44.8$	$42.7$
	Gemma-7B (8.5B)	$53.5$	$47.8$ ( $\downarrow 5.7$ )	$51.7$	$50.1$	$47.6$	$41.8$