notesum.ai

Published at November 29

Towards Understanding Retrieval Accuracy and Prompt Quality in RAG Systems

cs.SE

cs.AI

Released Date: November 29, 2024

Authors: Shengming Zhao, Yuheng Huang, Jiayang Song, Zhijie Wang, Chengcheng Wan, Lei Ma

Arxiv: http://arxiv.org/pdf/2411.19463v1

Refer to caption

Scenario	Conala	DS1000	PNE	Avg	Improv.
Oracle*	0.381	0.356	0.784	0.507	0.0%
Oracle, repeat	0.357	0.376	0.772	0.502	-1.1%
Oracle + Random	0.393	0.369	0.802	0.521	2.8%
Oracle + Diff	0.381	0.388	0.814	0.528	4.1%
Oracle + Dummy	0.381	0.387	0.814	0.527	4%
Oracle + Ellipsis	0.345	0.373	0.784	0.501	-1.2%
Distracting*	0.226	0.264	0.653	0.381	0.0%
Distracting, repeat	0.274	0.319	0.689	0.427	12.2%
Distracting + Random	0.202	0.327	0.713	0.414	8.7%
Distracting + Diff	0.250	0.321	0.719	0.43	12.9%
Distracting + Dummy	0.238	0.349	0.695	0.427	12.2%
Distracting + Ellipsis	0.179	0.292	0.677	0.383	0.4%
Retrieved Top-5*	0.286	0.339	0.719	0.448	0.0%
Retrieved Top-5, repeat	0.298	0.295	0.713	0.435	-2.8%
Retrieved Top-5 + Random	0.262	0.321	0.743	0.442	-1.3%
Retrieved Top-5 + Diff	0.310	0.343	0.778	0.477	6.5%
Retrieved Top-5 + Dummy	0.298	0.317	0.766	0.46	2.8%
Retrieved Top-5 + Ellipsis	0.298	0.312	0.772	0.461	2.8%
None*	0.226	0.367	0.790	0.461	0.0%
None + Random	0.286	0.323	0.76	0.456	-1%
None + Diff	0.321	0.357	0.790	0.489	6.1%
None + Dummy	0.298	0.343	0.760	0.467	1.3%
None + Ellipsis	0.262	0.332	0.764	0.453	-1.8%
Random*	0.262	0.351	0.754	0.456	1.8%
Random, repeat	0.357	0.371	0.737	0.488	8.9%
Diff*	0.333	0.399	0.790	0.507	13.2%
Diff, repeat	0.345	0.378	0.772	0.502	12.1%
Dummy*	0.310	0.356	0.772	0.479	6.9%
Dummy, repeat	0.310	0.369	0.737	0.472	5.4%
Ellipsis*	—	—	—	—
Ellipsis, repeat	0.274	0.368	0.784	0.475	6.0%