notesum.ai

Published at December 9

cs.CL

Released Date: December 9, 2024

Authors: David Samuel¹, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, Andrey Kutuzov

Aff.: ¹University of Oslo, Language Technology Group

Benchmark	Language	NorMistral-11B	NorwAI-Mistral-7B	NorwAI-Llama2-7B	normistral-7b-warm	NorGPT-3B	Viking-7B	Viking-13B	Mistral-Nemo-12B
_{Reading comprehension}
Belebele _(0-shot)	Bokmål	56.7	33.4	38.0	37.4	26.8	27.6	28.2	62.8
NorQuAD _(1-shot)	Bokmål	76.7	63.0	39.2	64.8	3.0	48.4	57.1	76.5
_{Sentiment analysis}
NoReC _{(sentence-level; 16-shot)}	Bokmål	90.5	88.6	86.0	84.9	49.7	77.9	79.2	86.9
NoReC _{(document-level; 1-shot)}	Bokmål	91.2	81.2	79.2	82.9	51.5	80.4	86.8	89.2
_{Commonsense reasoning}
NorCommonsenseQA _(0-shot)	Bokmål	59.2	52.4	48.1	48.9	35.7	43.7	49.7	44.0
NorCommonsenseQA _(0-shot)	Nynorsk	51.6	43.2	37.9	43.2	29.5	39.0	40.0	33.7
_{World knowledge}
NRK-Quiz-QA _(0-shot)	Bokmål	63.7	55.2	52.3	57.9	33.1	44.2	51.0	47.4
NRK-Quiz-QA _(0-shot)	Nynorsk	71.9	65.2	64.3	65.9	37.3	51.1	54.8	47.2
NorOpenBookQA _(16-shot)	Bokmål	68.2	39.3	38.7	37.7	30.2	35.7	37.3	74.4
NorOpenBookQA _(16-shot)	Nynorsk	60.0	36.7	34.4	33.3	38.9	22.2	26.6	73.3
_{Summarization}
NorSumm _(0-shot)	Bokmål	40.0	12.1	4.8	17.4	28.6	30.6	33.8	39.5
NorSumm _(0-shot)	Nynorsk	30.6	10.6	3.1	9.9	22.4	25.8	27.4	29.4
_{Grammatical error correction}
ASK-GEC _(16-shot)	Bokmål	52.6	53.2	51.4	48.7	1.8	51.1	52.4	43.9
_{Language identification}
Scandinavian LID _(16-shot)	$\begin{subarray}{c}\text{Bokm\r{a}l, Nynorsk,}\\ \text{Danish, Swedish}\end{subarray}$	98.6	96.2	93.4	77.5	40.3	68.9	65.6	59.5
_Translation
Tatoeba _{(from English; 16-shot)}	Bokmål	58.8	58.7	57.9	57.2	1.8	59.7	60.0	49.6
Tatoeba _{(from English; 16-shot)}	Nynorsk	48.0	47.4	47.4	44.7	2.6	45.6	45.6	35.7
Tatoeba _{(from English; 16-shot)}	Northern Sámi	45.5	24.8	26.6	14.9	0.0	4.6	9.8	3.4