notesum.ai

Published at November 8

Aioli: A Unified Optimization Framework for Language Model Data Mixing

cs.LG

cs.AI

cs.CL

stat.ML

Released Date: November 8, 2024

Authors: Mayee F. Chen¹, Michael Y. Hu², Nicholas Lourie³, Kyunghyun Cho⁴, Christopher Ré¹

Aff.: ¹Department of Computer Science, Stanford University; ²Center for Data Science, New York University; ³Computer Science Department, New York University; ⁴Computer Science Department, New York University, Prescient Design, Genentech

Arxiv: http://arxiv.org/abs/2411.05735v1

Method	A/SE	GH/C4	B/SE	A/B/SE	CC/GH/W	SlimPajama	# $<$ stratified	# extra runs
Stratified	$16.532$	$35.991$	$47.192$	$35.114$	$41.583$	$26.426$	-	0
GS	$-0.399$	$-0.407$	$-0.645$	$-0.247$	$0.298$	$0.176$	4	10
DML	$-0.241$	$-0.110$	$-0.644$	$-0.599$	$0.242$	$0.175$	4	10
Skill-It	$-0.326$	$0.551$	$-0.728$	$-0.568$	$-0.195$	$-0.184$	5	$m$
DoReMi	$-0.307$	$5.303$	$-0.217$	$-0.393$	$6.898$	$0.123$	3	2
DoGE	$0.419$	$0.184$	$-0.678$	$1.843$	$0.604$	$0.809$	1	1
Aioli	$-0.205$	$-0.340$	$-0.439$	$-0.096$	$-0.196$	$-0.240$	6	0