notesum.ai

Published at November 7

Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning

cs.LG

cs.AI

cs.CL

Released Date: November 7, 2024

Authors: Joey Hong¹, Anca Dragan¹, Sergey Levine¹

Aff.: ¹University of California, Berkeley

Arxiv: http://arxiv.org/abs/2411.05193v1

Refer to caption

	language games			alfworld
Method	Chess	Wordle	20Q	Pick	Examine	Clean	Heat	Cool	Pick2
ReAct	$0$	$-4.96$	$-13.2$	$\mathbf{45}$	$19$	$17$	$7$	$12$	$\mathbf{24}$
SFT	$0.11$	$-3.81$	$-17.3$	$38$	$15$	$0$	$11$	$0$	$18$
ILQL	$0.09$	$-2.08$	$-14.2$	$28$	$7$	$0$	$5$	$2$	$15$
Q-SFT (ours)	$\mathbf{0.15}$	$\mathbf{-2.11}$	$\mathbf{-13.1}$	$39$	$\mathbf{21}$	$\mathbf{19}$	$\mathbf{14}$	$\mathbf{18}$	$21$