notesum.ai

Published at December 10

The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model

cs.CL

Released Date: December 10, 2024

Authors: Jiawei Chen¹, Wentao Chen, Jing Su, Jingjing Xu, Hongyu Lin, Mengjie Ren, Yaojie Lu, Xianpei Han, Le Sun

Aff.: ¹Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China

Arxiv: http://arxiv.org/pdf/2412.07298v1

Refer to caption

1B+ Models
	HumanEval						MBXP
	Python	PHP	C#	Go	C++	AVE	Python	PHP	C#	Go	C++	AVE
StarCoder-1B (Li et al., 2023b)	15.17	13.04	15.19	8.44	14.63	13.29	29.80	18.56	23.29	58.93	26.18	31.35
StarCoder-3B (Li et al., 2023b)	21.46	21.12	16.46	14.29	24.39	19.54	39.80	30.13	39.27	74.78	35.91	43.98
DeepseekCoder-1.3B (Guo et al., 2024)	\ul29.88	23.60	\ul27.22	\ul20.13	31.10	\ul26.39	52.40	\ul45.41	48.40	\ul78.35	55.11	55.93
Original-1.3B (ours)	28.66	\ul24.22	23.42	21.43	\ul27.44	25.03	41.20	33.41	\ul43,38	79.24	45.39	49.81
Optimized-1.3B (ours)	32.93	26.08	29.75	16.88	\ul27.44	26.61	\ul45.40	46.29	40.64	77.90	\ul49.63	\ul51.97
6B+ Models
StarCoder-7B (Li et al., 2023b)	28.37	24.22	27.85	17.53	25.61	24.72	46.20	42.36	45.66	71.88	43.64	49.95
StarCoder2-7B (Lozhkov et al., 2024)	35.40	32.30	39.24	22.08	34.76	32.76	53.13	55.02	54.79	82.37	55.61	60.18
CodeLlama-7B (Rozière et al., 2024)	33.50	24.22	34.81	20.13	27.44	28.02	49.80	43.89	48.40	72.32	48.63	52.61
DeepseekCoder-6.7B (Guo et al., 2024)	\ul46.95	37.89	47.47	31.82	44.51	\ul41.73	65.80	66.16	66.67	89.78	67.58	71.20
Original-6.7B (ours)	\ul46.95	\ul38.51	39.24	26.62	40.24	38.31	59.00	57.86	61.19	\ul83.04	60.60	64.34
Optimized-6.7B (ours)	48.17	41.61	\ul44.94	\ul31.17	\ul43.90	41.96	\ul63.60	\ul59.83	\ul61.42	81.03	\ul64.84	\ul66.14