notesum.ai

Published at November 11

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning

cs.AI

cs.CL

cs.LG

Released Date: November 11, 2024

Authors: Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas

Arxiv: http://arxiv.org/abs/2411.07279v1

Refer to caption

Program Synthesizer	Fine-tuned LM	TTT Method	Score (pass@2)
X	Ours	X	$18.25$ %
X	Ours	Ours	$47.125$ %
X	BARC	Ours	$53$ %
BARC	Ours	Ours	$58.5$ %
BARC	BARC	Ours	$61.875$ %
Avg. Human			$60.2$ %
Best Human			$97.8$ %
BARC (ensemble)			$54.375$ %
BARC (no synthesizer)			$39.25$ %
Claude - Few-shot prompting			$21$ %
GPT-4.0 - Few-shot prompting			$9$ %