notesum.ai

Published at October 21

cs.SE

cs.AI

cs.LO

cs.PL

Released Date: October 21, 2024

Authors: Quang Hieu Pham¹, Hoang Ngo¹, Anh Tuan Luu², Dat Quoc Nguyen¹

Aff.: ¹VinAI Research, Vietnam; ²Nanyang Technological University, Singapore

Model	#Param	SimQA	W/oS	W/S
Gemma 1.1	7B	91.8	12.7	34.4
Mistral	7B	96.5	44.5	54.7
Qwen1.5 Chat	7B	91.9	22.0	50.2
Llama 3	8B	98.2	72.4	71.5
Qwen1.5 Chat	14B	95.5	9.90	51.9
Qwen1.5 Chat	32B	95.5	21.0	60.5
Command R	35B	92.5	69.9	-
Mixtral 8x7B	40B	97.1	53.7	64.2
Llama 3	70B	97.8	83.8	86.1
Qwen1.5 Chat	72B	95.9	28.2	66.8
gpt-3.5-turbo	-	97.9	36.0	58.9