notesum.ai

Published at November 19

CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval

cs.SE

cs.AI

Released Date: November 19, 2024

Authors: Ye Liu¹, Rui Meng¹, Shafiq Jot¹, Silvio Savarese¹, Caiming Xiong¹, Yingbo Zhou¹, Semih Yavuz¹

Aff.: ¹Salesforce AI Research

Arxiv: http://arxiv.org/abs/2411.12644v1

Refer to caption

	Text-to-Code			Code-to-Text	Code-to-Code			Hybrid Code
Model	Apps	CosQA	Text2SQL	CSN	CSN-CCR	CodeTrans		StackOverFlow	CodeFeedBack		Avg
	Apps	CosQA	Text2SQL	CSN	CSN-CCR	-Contest	-DL	QA	-ST	-MT
Baselines
Contriever (110M)	5.14	14.21	45.46	34.72	35.74	44.16	24.21	66.05	55.11	39.23	36.40
E5-base (110M)	11.52	32.59	52.31	67.99	56.87	62.50	21.87	86.86	74.52	41.99	50.90
BGE-Base (110M)	4.05	32.76	45.59	69.60	45.56	38.50	21.71	73.55	64.99	31.42	42.77
GTE-Base (110M)	3.24	30.24	46.19	43.35	35.50	33.81	28.80	55.19	55.19	28.48	36.75
UniXcoder (123M)	1.36	25.14	50.45	60.20	58.36	41.82	31.03	44.67	36.02	24.21	37.33
BGE-M3 (567M)	7.37	22.73	48.76	43.23	47.55	47.86	31.16	51.04	49.94	33.46	39.31
E5-Mistral (7B)	21.33	31.27	65.98	54.25	65.27	82.55	33.24	91.54	72.71	33.65	55.18
OpenAI-Ada-002	8.70	28.88	58.32	74.21	69.13	53.34	26.04	72.40	47.12	17.74	45.59
Voyage-Code-002	26.52	29.79	69.26	81.79	73.45	72.77	27.28	67.68	65.35	28.74	56.26
General Training
CodeXEmbed_400M	48.57	34.05	58.96	72.53	80.15	75.67	34.85	89.51	78.87	45.75	61.89
CodeXEmbed_2B	74.99	36.31	59.00	73.50	85.77	86.63	33.17	90.54	81.15	53.08	67.41
CodeXEmbed_7B	85.22	33.27	64.57	78.84	86.77	90.64	32.31	94.25	80.93	57.83	70.46
In-domain Training
CodeXEmbed_400M	45.91	41.28	61.29	81.23	93.74	82.72	40.81	92.35	83.36	61.51	68.42
CodeXEmbed_2B	76.86	40.47	78.42	87.87	97.66	90.30	38.57	94.47	86.36	65.51	75.65
CodeXEmbed_7B	85.38	42.47	78.94	89.67	97.95	94.45	40.46	96.33	87.53	68.83	78.20