notesum.ai

Published at November 22

KBAda: Efficient Self Adaptation on Specific Knowledge Bases

cs.CL

cs.AI

Released Date: November 22, 2024

Authors: Zheni Zeng¹, Yuxuan Chen¹, Shi Yu¹, Yukun Yan¹, Zhenghao Liu², Shuo Wang¹, Xu Han¹, Zhiyuan Liu¹, Maosong Sun¹

Aff.: ¹Tsinghua University; ²Northeastern University

Arxiv: http://arxiv.org/abs/2411.14790v1

Refer to caption

Methods	LooGLE					ASQA					JEC-QA
	F1	BLEU	ROUGE	BERT	LLM	Match	BLEU	ROUGE	BERT	LLM	Single	Multi	Total
GPT series
GPT-3.5	35.42	13.11	32.67	80.99	78.08	26.79	3.19	25.37	86.61	51.66	14.49	17.92	16.32
wo QE	35.27	12.97	32.51	80.94	77.91	27.40	3.13	24.92	86.71	51.52	13.84	19.15	16.68
GPT-4o	40.20	14.93	36.07	81.70	82.93	32.18	2.87	27.69	87.14	67.76	21.95	26.42	24.33
wo QE	40.21	15.18	36.06	81.71	83.29	32.15	2.94	27.49	87.10	67.88	20.11	27.36	23.98
MiniCPM-2.4B
Original	30.92	10.73	29.27	80.70	64.76	11.91	1.67	21.84	82.30	22.92	39.24	13,87	25.69
wo QE	30.31	10.50	28.62	80.37	64.72	12.37	1.72	22.36	82.90	22.42	38.38	14.06	25.39
Ours	54.09	18.32	49.75	86.48	75.19	15.68	2.67	24.59	85.41	24.81	49.95	9.94	28.91
$\Delta$	(+23.17)	(+7.59)	(+20.48)	(+5.78)	(+10.43)	(+3.77)	(+1.00)	(+2.75)	(+3.11)	(+1.89)	(+10.71)	(-3.93)	(+3.22)
wo QE	53.76	18.55	49.61	86.23	73.19	16.12	2.75	24.65	85.48	25.69	49.41	10.92	29.16
LLaMA3.1-8B-Instruction
Original	40.46	15.91	36.31	81.57	77.15	20.21	2.94	23.24	84.93	37.28	22.70	24.66	23.73
wo QE	39.94	15.78	36.14	81.50	77.08	20.03	2.94	23.09	85.14	35.64	22.92	24.07	23.53
Ours	62.07	21.73	57.34	88.63	80.16	25.23	3.43	23.59	86.29	42.44	34.59	14.13	23.83
$\Delta$	(+21.61)	(+5.82)	(+21.03)	(+6.06)	(+2.85)	(+5.02)	(+0.49)	(+1.35)	(+1.36)	(+5.16)	(+11.89)	(-10.53)	(+0.10)
wo QE	61.79	21.60	57.09	88.55	79.96	25.56	3.23	23.27	86.89	41.31	34.16	14.42	23.78