notesum.ai

Published at December 9

GameArena: Evaluating LLM Reasoning through Live Computer Games

cs.AI

cs.CL

Released Date: December 9, 2024

Authors: Lanxiang Hu¹, Qiyu Li¹, Anze Xie¹, Nan Jiang¹, Ion Stoica, Haojian Jin¹, Hao Zhang¹

Aff.: ¹University of California, San Diego

Arxiv: http://arxiv.org/pdf/2412.06394v1

Refer to caption

	Akinator		Taboo		Bluffing
Model	Avg. Win Rate	Avg. # Round	Avg. Win Rate	Avg. # Round	Avg. Win Rate	Avg. # Round
claude-3-5-sonnet-20240620	0.55_{$\pm$ 0.11}	16.61_{$\pm$ 1.75}	0.61_{$\pm$ 0.18}	3.36_{$\pm$ 0.88}	0.67_{$\pm$ 0.13}	6.00_{$\pm$ 0.00}
gpt-4o-2024-08-06	0.49_{$\pm$ 0.13}	16.36_{$\pm$ 0.86}	0.67_{$\pm$ 0.11}	3.19_{$\pm$ 0.34}	0.58_{$\pm$ 0.13}	5.92_{$\pm$ 0.18}
gemini-1.5-pro	0.51_{$\pm$ 0.17}	16.57_{$\pm$ 1.49}	0.61_{$\pm$ 0.04}	3.74_{$\pm$ 0.45}	0.60_{$\pm$ 0.18}	5.96_{$\pm$ 0.10}
llama-3.1-405b	0.44_{$\pm$ 0.04}	17.15_{$\pm$ 0.66}	0.62_{$\pm$ 0.18}	3.08_{$\pm$ 0.18}	0.44_{$\pm$ 0.22}	5.90_{$\pm$ 0.27}
mistral-large-latest	0.02_{$\pm$ 0.04}	19.99_{$\pm$ 0.02}	0.66_{$\pm$ 0.13}	3.43_{$\pm$ 0.57}	0.0_{$\pm$ 0.00}	6.00_{$\pm$ 0.00}