notesum.ai

Published at October 30

cs.CL

cs.AI

cs.LG

I.2.7

Released Date: October 30, 2024

Authors: Søren Vejlgaard Holm¹, Lars Kai Hansen¹, Martin Carsten Nielsen²

Aff.: ¹Technical University of Denmark, Anker Engelunds Vej 1, 2800 Kongens Lyngby, Denmark; ²Alvenir, Applebys Plads 7, 1411 København K, Denmark

Benchmark	D	$\sigma_{\text{F1}}$	$\sigma_{\text{F2}}$	#F ${}_{\text{Kai.}}$	#F ${}_{\text{PA}}$
Danoliterate	$54\times 8$	$95\%$	$5\%$	1	1
ScandEval Da.	$147\times 8$	$91\%$	$9\%$	1	2
ScandEval Full	$147\times 24$	$80\%$	$13\%$	2	2
HELM Lite	$78\times 10$	$76\%$	$21\%$	2	2
OpenLLM	$1602\times 6$	$95\%$	$5\%$	1	2