notesum.ai

Published at November 21

cs.CL

Released Date: November 21, 2024

Authors: Iacopo Ghinassi¹, Leonardo Catalano², Tommaso Colella³

Aff.: ¹Queen Mary University of London; ²Independent Researcher; ³University of Pisa

Model	Consistency $\uparrow$	Coherence $\uparrow$	Fluency $\uparrow$	Relevance $\uparrow$	Average $\uparrow$
Qwen 0.5B	4.52*	4.33*	4.41*	4.06*	4.33*
Qwen 1.8B	4.89	4.83	4.88	4.79	4.85
Qwen 4B	4.75*	4.84	4.91	4.56*	4.77
Qwen 7B	4.84	4.94	4.9	4.74	4.86
Gemma 2B	4.86	4.86	4.96	4.71	4.85
Gemma 7B	4.85	4.94	4.99	4.81	4.9
Phi 3	4.84	4.92	4.94	4.74	4.86
Llama 3	4.82	4.84	4.91	4.74	4.83
Mistral	4.78*	4.84	4.95	4.6	4.79
ChatGPT	4.94	4.96	4.98	4.79	4.91
GPT4	4.83	4.89	4.96	4.81	4.89
MPNet	4.44*	3.03*	3.45*	4.15*	3.77*