notesum.ai

Published at December 5

Retrieval-Augmented Machine Translation with Unstructured Knowledge

cs.CL

cs.AI

Released Date: December 5, 2024

Authors: Jiaan Wang¹, Fandong Meng¹, Yingxue Zhang¹, Jie Zhou¹

Aff.: ¹Pattern Recognition Center, WeChat AI, Tencent Inc

Arxiv: http://arxiv.org/pdf/2412.04342v1

		Zero-Shot LLMs				SFT LLMs				SFT+CSC LLMs
		BLEU	COMET	GRB	GRF	BLEU	COMET	GRB	GRF	BLEU	COMET	GRB	GRF
w/ Empty Document
1	Qwen2.5-7B	50.11	84.63	85.48	86.97	54.64	86.72	88.50	89.07	56.82	87.74	91.16	91.81
2	Qwen2.5-14B	51.34	84.71	86.20	87.65	55.43	86.93	89.12	89.42	57.59	87.88	91.62	92.04
3	LLama-3-8B	40.94	77.54	80.39	78.08	54.35	86.60	88.06	88.24	56.09	87.60	90.30	91.12
4	Mistral-7B	38.12	76.40	79.75	77.60	53.58	86.32	87.69	88.08	54.67	87.16	90.06	90.48
w/ Noisy Document
5	Qwen2.5-7B	48.62	83.36	84.57	86.05	54.66	86.73	88.50	89.11	56.48	87.66	91.09	91.75
6	Qwen2.5-14B	49.40	83.66	84.90	86.49	55.47	86.97	89.17	89.38	57.74	87.91	91.64	92.09
7	LLama-3-8B	26.33	74.64	78.20	78.92	54.25	86.59	87.99	88.17	56.12	87.60	90.26	91.09
8	Mistral-7B	24.29	73.86	77.69	78.04	53.49	86.30	87.72	88.11	54.46	87.14	89.95	90.45
w/ Golden English Document
9	Qwen2.5-7B	49.55	84.29	85.05	86.48	57.10	87.72	90.57	91.34	58.68	88.72	92.48	93.14
10	Qwen2.5-14B	50.58	84.47	85.47	87.10	56.95	87.90	91.10	91.76	59.43	88.85	92.71	93.26
11	LLama-3-8B	29.44	75.29	80.23	80.85	55.84	87.58	89.74	90.35	58.00	88.59	91.39	92.03
12	Mistral-7B	26.56	74.75	79.75	80.27	54.90	87.34	89.49	89.82	56.63	88.14	91.14	91.87
w/ Golden Chinese Document
13	Qwen2.5-7B	49.93	84.40	85.39	86.72	57.25	87.07	91.19	91.50	59.97	89.10	93.06	93.57
14	Qwen2.5-14B	50.65	84.68	85.72	86.95	58.09	87.23	91.67	91.95	60.47	89.21	93.29	93.73
15	LLama-3-8B	35.74	76.39	80.82	81.04	57.09	87.04	90.20	90.81	59.47	88.99	92.84	93.15
16	Mistral-7B	34.86	75.21	79.82	80.43	56.36	87.76	89.93	90.33	58.78	88.75	92.58	93.02
w/ Golden German Document
17	Qwen2.5-7B	44.17	83.15	84.25	85.07	55.59	87.20	88.93	89.57	58.68	88.74	92.39	93.07
18	Qwen2.5-14B	45.20	82.89	84.82	85.64	56.28	87.46	89.36	89.71	59.41	88.89	92.65	93.13
19	LLama-3-8B	35.27	74.83	78.46	78.93	55.13	87.08	88.62	89.13	57.72	88.56	91.27	91.73
20	Mistral-7B	34.98	73.29	77.32	76.45	54.34	86.81	88.28	88.79	56.62	88.14	91.02	91.67