notesum.ai

Published at November 6

Fine-Grained Guidance for Retrievers: Leveraging LLMs' Feedback in Retrieval-Augmented Generation

cs.IR

cs.AI

Released Date: November 6, 2024

Authors: Yuhang Liu¹, Xueyu Hu¹, Shengyu Zhang¹, Jingyuan Chen¹, Fan Wu², Fei Wu¹

Aff.: ¹Zhejiang University; ²Shanghai Jiao Tong University

Arxiv: http://arxiv.org/abs/2411.03957v1

GPT-3.5-Turbo
Method	MMLU					NQ	PQA	HoPo	FEV.	All Avg.
Method	Hum.	Soc.	STEM	Other	All	NQ	PQA	HoPo	FEV.	All Avg.
No retrieval	52.9	76.6	53.1	75.7	63.4	48.1	44.3	33.6	82.1	54.3
Contriever	55.1	76.3	54.5	74.5	64.2	48.8	45.6	39.0	89.4	57.4
AAR ${}_{\text{Contriever}}$	54.3	78.5	52.5	77.1	64.4	49.0	46.3	36.9	89.6	57.2
BGE	52.9	78.2	54.0	76.5	64.1	50.3	43.9	39.5	89.3	57.4
SBERT	54.1	77.9	52.8	77.4	64.5	49.4	50.1	38.7	88.5	58.2
FiGRet ${}_{\text{Contriever}}$ (Ours)	55.4	76.9	54.5	77.1	65.0	49.6	48.0	39.9	90.6	58.6
FiGRet ${}_{\text{BGE}}$ (Ours)	55.8	79.8	54.3	76.5	65.5	50.4	45.7	40.0	90.3	58.4
FiGRet ${}_{\text{SBERT}}$ (Ours)	55.8	77.2	55.2	76.8	65.4	49.9	50.1	39.1	88.7	58.6
Llama-3-8B-Instruct
No retrieval	52.9	74.4	51.6	73.3	62.0	33.1	26.1	25.9	79.1	45.2
Contriever	52.9	76.3	52.5	73.3	62.5	41.3	41.7	36.0	84.5	53.2
AAR ${}_{\text{Contriever}}$	52.9	77.2	54.0	73.9	63.2	42.1	42.3	35.3	85.2	53.6
BGE	54.4	76.9	52.8	73.6	63.3	44.1	36.1	35.9	86.1	53.1
SBERT	53.9	76.6	54.6	73.3	63.4	41.7	46.0	35.7	86.2	54.6
FiGRet ${}_{\text{Contriever}}$ (Ours)	53.5	77.5	53.1	74.8	63.4	43.0	44.3	36.9	86.5	54.8
FiGRet ${}_{\text{BGE}}$ (Ours)	53.9	76.3	53.4	75.1	63.6	45.3	41.4	37.5	87.8	55.1
FiGRet ${}_{\text{SBERT}}$ (Ours)	54.6	76.3	54.0	74.2	63.7	42.8	46.3	36.1	86.2	55.0
Claude-3-Haiku
No retrieval	59.5	82.6	59.4	78.3	68.8	27.6	31.7	26.9	70.4	45.1
Contriever	61.6	82.0	60.0	79.8	70.0	35.7	41.3	33.0	90.0	54.0
AAR ${}_{\text{Contriever}}$	62.6	83.5	59.1	79.5	70.2	36.1	42.1	32.7	90.2	54.3
BGE	61.4	82.0	58.5	77.4	69.0	38.1	37.5	33.0	89.6	53.4
SBERT	62.0	81.0	58.8	79.2	69.4	35.9	46.2	32.7	89.5	54.7
FiGRet ${}_{\text{Contriever}}$ (Ours)	62.9	82.0	60.3	80.1	70.5	36.5	44.2	33.8	90.4	55.1
FiGRet ${}_{\text{BGE}}$ (Ours)	63.3	82.9	57.9	78.9	69.9	40.0	42.2	35.7	90.0	55.6
FiGRet ${}_{\text{SBERT}}$ (Ours)	63.7	84.2	59.4	77.7	70.5	37.1	46.6	33.0	90.1	55.5