notesum.ai

Published at October 28

cs.AI

cs.LG

Released Date: October 28, 2024

Authors: Nour Jedidi¹, Yung-Sung Chuang², Leslie Shing¹, James Glass²

Aff.: ¹MIT Lincoln Laboratory; ²Massachusetts Institute of Technology

		High Resource		Low Resource (BEIR)
Model	Init. Retrieval	DL19	DL20	News	Covid	FiQA	SciFact	DBPedia	NFCorpus	Robust04	BEIR (Avg.)
BM25	N/A	50.6	48.0	39.5	59.5	23.6	67.9	31.8	32.2	40.7	42.2
Contriever	N/A	44.5	42.1	34.8	27.3	24.5	64.9	29.2	31.7	31.6	34.9
Hybrid (BM25 + Contriever)	N/A	52.9	50.9	42.2	52.9	28.4	71.6	34.8	33.7	43.0	43.8
$\textnormal{Contriever}_{\textnormal{AvgPRF}}$
PRF-Depth: 3	Hybrid (3)	52.1	46.4	43.4	48.1	22.8	59.9	31.9	32.3	38.6	39.6
PRF-Depth: 20	Hybrid (20)	49.2	46.2	39.9	51.3	12.0	36.9	28.6	21.7	39.2	32.8
Zero-Shot Dense Retrieval
PromptReps (Llama3-8B-I)	N/A	-	-	-	59.5	27.1	52.7	31.5	29.6	-	-
HyDE (Mistral-7B-Instruct)	N/A	57.8	53.9	44.0	56.9	21.6	65.1	35.3	27.7	41.5	41.7
HyDE ${}_{\text{PRF}}$ (Mistral-7B-Instruct)	Hybrid (20)	63.5	62.0	46.9	59.1	28.3	64.5	35.2	35.0	45.6	44.9

ReDE-RF (Ours)
Default: Contriever	Hybrid (20)	60.3	59.4	46.1	65.6	28.2	67.4	37.0	34.8	49.8	47.0
Default: HyDE ${}_{\text{PRF}}$	Hybrid (20)	62.8	60.4	47.1	65.6	29.3	66.9	37.6	35.5	51.7	47.7
Supervised Dense Retrieval
DPR	N/A	62.2	65.3	16.1	33.2	11.2	31.8	26.3	18.9	25.2	23.2
ANCE	N/A	64.5	64.6	38.2	65.4	29.5	50.7	28.1	23.7	39.2	39.3
$\textnormal{Contriever}^{\textnormal{FT}}$	N/A	62.1	63.2	42.8	59.6	32.9	67.7	41.3	32.8	47.3	46.3