notesum.ai

Published at November 20

BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

cs.AI

Released Date: November 20, 2024

Authors: Davide Paglieri¹, Bartłomiej Cupiał², Samuel Coward³, Ulyana Piterbarg⁴, Maciej Wolczyk², Akbir Khan¹, Eduardo Pignatelli¹, Łukasz Kuciński, Lerrel Pinto⁴, Rob Fergus⁴, Jakob Nicolaus Foerster³, Jack Parker-Holder¹, Tim Rocktäschel

Aff.: ¹AI Centre, University College London; ²IDEAS NCBR; ³University of Oxford; ⁴New York University

Arxiv: http://arxiv.org/abs/2411.13543v1

Model	Average Progress (%)
gpt-4o	32.34 $\pm$ 1.49
claude-3.5-sonnet	29.98 $\pm$ 1.98
llama-3.1-70b-it	27.88 $\pm$ 1.43
llama-3.2-90B-it	23.66 $\pm$ 1.09
gemini-1.5-pro	21.00 $\pm$ 1.18
gpt-4o-mini	17.36 $\pm$ 1.35
llama-3.1-8b-it	14.14 $\pm$ 1.51
llama-3.2-11B-it	13.54 $\pm$ 1.05
gemini-1.5-flash	9.73 $\pm$ 0.77
llama-3.2-3B-it	8.47 $\pm$ 1.12
llama-3.2-1B-it	6.32 $\pm$ 1.00