notesum.ai

Published at November 17

cs.CL

cs.AI

Released Date: November 17, 2024

Authors: Bin Xu¹, Yiguan Lin¹, Yinghao Li¹, Yang Gao¹

Aff.: ¹School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China

Model	MBPP	MBPP+	MBPP+ pass@10	Human-Eval	Human-Eval+	Human-Eval+ pass@10	Average Increment
gemma-2-2b
Instruct	34.42	43.39	48.41	39.76	33.05	37.2	+0.00
CoT	34.90*	43.7	47.9	41.89*	35.37*	39.02*	+1.09*
SRA-MCTS	33.92	45.37*	49.21*	40.73	34.88	37.2	+0.85
Meta-Llama-3.1-8B
Instruct	51.94	45.37	49.21	62.74*	58.90*	67.68	+0.00
CoT	52.94	60.50*	65.08	62.32	58.35	66.46	+4.97
SRA-MCTS	54.52*	59.97	66.14*	62.19	57.87	68.29*	+5.52*
Qwen2.5-14B
Instruct	56.42	61.48	70.37	80.37	76.52*	76.83*	+0.00
CoT	58.12	63.97*	70.37	78.66	73.84	74.39	-0.44
SRA-MCTS	61.02*	61.16	83.60*	85.37*	75.00	75.61	+3.30*