notesum.ai

Published at November 29

Can Large Language Models Reason about the Region Connection Calculus?

cs.CL

Released Date: November 29, 2024

Authors: Anthony G Cohn¹, Robert E Blackwell²

Aff.: ¹School of Computing, University of Leeds, UK; ²The Alan Turing Institute, UK

Arxiv: http://arxiv.org/pdf/2411.19589v1

Refer to caption

Model	1epon	1anon	2epon	2anon	3epon	3anon	Overall
claude-3-5-sonnet-20240620	0.69 $\pm$ 0.008	0.58 $\pm$ 0.012	0.56 $\pm$ 0.011	0.54 $\pm$ 0.008	0.61 $\pm$ 0.023	0.61 $\pm$ 0.026	0.60 $\pm$ 0.016
azure-gpt-4-turbo-2024-04-09	0.50 $\pm$ 0.014	0.45 $\pm$ 0.013	0.50 $\pm$ 0.014	0.49 $\pm$ 0.014	0.67 $\pm$ 0.029	0.57 $\pm$ 0.036	0.53 $\pm$ 0.022
gemini-15-pro	0.51 $\pm$ 0.013	0.49 $\pm$ 0.011	0.51 $\pm$ 0.010	0.46 $\pm$ 0.007	0.52 $\pm$ 0.021	0.42 $\pm$ 0.021	0.49 $\pm$ 0.015
azure-gpt-4o-2024-05-13	0.47 $\pm$ 0.013	0.43 $\pm$ 0.012	0.50 $\pm$ 0.012	0.50 $\pm$ 0.014	0.57 $\pm$ 0.039	0.41 $\pm$ 0.033	0.48 $\pm$ 0.024
azureai-llama-3-70b-instruct	0.43 $\pm$ 0.006	0.40 $\pm$ 0.010	0.40 $\pm$ 0.004	0.40 $\pm$ 0.005	0.43 $\pm$ 0.018	0.18 $\pm$ 0.021	0.37 $\pm$ 0.013
azure-gpt-35-turbo-0125	0.33 $\pm$ 0.009	0.25 $\pm$ 0.011	0.44 $\pm$ 0.015	0.36 $\pm$ 0.011	0.39 $\pm$ 0.034	0.30 $\pm$ 0.030	0.35 $\pm$ 0.021
Overall	0.49 $\pm$ 0.011	0.43 $\pm$ 0.012	0.49 $\pm$ 0.012	0.46 $\pm$ 0.011	0.53 $\pm$ 0.028	0.42 $\pm$ 0.028	0.47 $\pm$ 0.057
Guess rate	0.31 $\pm$ 0.012	0.31 $\pm$ 0.012	0.13 $\pm$ 0.017	0.13 $\pm$ 0.017	0.26 $\pm$ 0.025	0.26 $\pm$ 0.025	0.23 $\pm$ 0.033