notesum.ai

Published at November 13

VALTEST: Automated Validation of Language Model Generated Test Cases

cs.SE

cs.AI

Released Date: November 13, 2024

Authors: Hamed Taherkhani¹, Hadi Hemmati¹

Aff.: ¹York University, Canada

Arxiv: http://arxiv.org/abs/2411.08254v1

Dataset	LLM	Base				VALTEST
Dataset	LLM	#Tests	VR	LC	MS	#Tests	%Tests	VR	LC	MS
HE	GPT-4o	$3157$	0.83	$0.969$	$0.86$	$2352$	$0.74$	0.925(+ $9.5$ %)	$0.966$	0.84(- $2$ %)
HE	GPT-3.5 Turbo	$2707$	0.74	$0.967$	$0.83$	$1553$	$0.57$	0.892(+ $15.2$ %)	$0.963$	0.79(- $4$ %)
HE	LLaMA 3.1 8B	$2471$	0.63	$0.946$	$0.80$	$598$	$0.24$	0.756(+ $12.6$ %)	$0.933$	0.63(- $17$ %)
HE	Average	2778	0.733	0.961	0.83	1501	0.54	0.858(+ $12.5$ %)	0.954	0.75(- $8$ %)
LeetCode	GPT-4o	$9047$	0.75	$0.983$	$0.852$	$5251$	$0.58$	0.946(+ $19.6$ %)	$0.983$	0.845(- $0.7$ %)
LeetCode	GPT-3.5 Turbo	$8230$	0.63	$0.981$	$0.862$	$3250$	$0.40$	0.870(+ $24$ %)	$0.978$	0.86(- $0.2$ %)
LeetCode	LLaMA 3.1 8B	$4999$	0.46	$0.968$	$0.785$	$1515$	$0.30$	0.690(+ $23$ %)	$0.958$	0.744(- $4.1$ %)
LeetCode	Average	7425	0.613	0.977	0.833	3339	0.46	0.835(+ $22.2$ %)	0.973	0.816(- $1.9$ %)
MBPP	GPT-4o	$7730$	0.71	$0.969$	$0.82$	$2555$	$0.33$	0.796(+ $8.6$ %)	$0.960$	0.79(- $3$ %)
MBPP	GPT-3.5 Turbo	$5923$	0.60	$0.966$	$0.78$	$2088$	$0.35$	0.667(+ $6.7$ %)	$0.954$	0.72(- $6$ %)
MBPP	LLaMA 3.1 8B	$4461$	0.53	$0.959$	$0.72$	$1876$	$0.42$	0.592(+ $6.2$ %)	$0.949$	0.65(- $7$ %)
MBPP	Average	6038	0.613	0.965	0.773	2173	0.37	0.685(+ $7.2$ %)	0.954	0.72(- $5.3$ %)