notesum.ai

Published at October 29

Fourier Head: Helping Large Language Models Learn Complex Probability Distributions

cs.LG

cs.AI

cs.CL

stat.ML

Released Date: October 29, 2024

Authors: Nate Gillman¹, Daksh Aggarwal¹, Michael Freeman¹, Saurabh Singh², Chen Sun³

Aff.: ¹Brown University; ²Google DeepMind; ³Brown University, Google DeepMind

Arxiv: https://arxiv.org/abs/2410.22269v1

[Uncaptioned image]

Decision Transformer Model	Normalized Return $(\uparrow)$	Smoothness $(\downarrow)$
Linear	2.53 $\pm$ 0.63	0.48 $\pm$ 0.14
Fourier-8	2.78 $\pm$ 0.47	0.22 $\pm$ 0.04
Fourier-14	3.70 $\pm$ 0.47	0.39 $\pm$ 0.11