notesum.ai

Published at November 8

cs.CL

cs.AI

Released Date: November 8, 2024

Authors: Nicole Meister¹, Carlos Guestrin¹, Tatsunori Hashimoto¹

Aff.: ¹Stanford University

Model	$\mathcal{A}(Y,\hat{Y}_{\mathcal{S},\mathcal{O}})$
GPT-4 (V)	0.204 $\pm$ 0.004
Anthropic Opus (V)	0.219 $\pm$ 0.005
Llama 3 70B (V)	0.226 $\pm$ 0.004
Anthropic Haiku (V)	0.235 $\pm$ 0.005
GPT-4 (Seq)	0.237 $\pm$ 0.004
Humans (V)	0.247 $\pm$ 0.004
GPT-3.5-Turbo (V)	0.259 $\pm$ 0.005
GPT-4 (TS-Log-p)	0.260 $\pm$ 0.004
GPT-3.5-Turbo (Seq)	0.278 $\pm$ 0.005
Anthropic Haiku (Seq)	0.287 $\pm$ 0.006
GPT-3.5-Turbo (TS-Log-p)	0.290 $\pm$ 0.005
Llama 3 70B (Seq)	0.320 $\pm$ 0.006
Anthropic Opus (Seq)	0.337 $\pm$ 0.006
GPT-3.5-Turbo (Log-p)	0.462 $\pm$ 0.007
Llama 3 70B (TS-Log-p)	0.460 $\pm$ 0.007
Llama 3 70B (Log-p)	0.515 $\pm$ 0.006
GPT-4 (Log-p)	0.582 $\pm$ 0.006
Discretization Error (Seq)	0.126 $\pm$ 0.006
Uniform	0.302 $\pm$ 0.005