notesum.ai

Published at November 21

Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective

cs.CL

Released Date: November 21, 2024

Authors: Shenglai Zeng¹, Jiankun Zhang¹, Bingheng Li¹, Yuping Lin¹, Tianqi Zheng², Dante Everaert², Hanqing Lu², Hui Liu¹, Hui Liu¹, Yue Xing¹, Monica Xiao Cheng², Jiliang Tang¹

Aff.: ¹Michigan State University; ²Amazon.com

Arxiv: http://arxiv.org/abs/2411.14572v1

Retrieval Type	Model	NQ		PopQA
Retrieval Type	Model	Noisy Acc (%)	Clean Acc(%)	Noisy Acc (%)	Clean Acc(%)
No-retrieval	LLaMA_2-7B-ChatTouvron et al. (2023)	73.17%	29.03%	71.20%	19.60%
	LLaMA_{3-8B-Instruct}AI@Meta (2024)	80.86%	32.73%	74.16%	22.45%
	Mistral_7B-InstructJiang et al. (2023a)	97.21%	20.10%	98.02%	15.58%
	Alpaca_7BTaori et al. (2023)	72.61%	23.94%	71.84%	13.07%
	Vicuna_7BZheng et al. (2023)	73.16%	26.64%	74.56%	19.43%
Unfiltered	LLaMA_2-7B-chat	34.66%	26.96%	60.91%	45.90%
	LLaMA_{3-8B-Instruct}	48.12%	33.59%	51.27%	40.54%
	Mistral_7B-instruct	28.97%	24.35%	55.96%	48.58%
	Alpaca_7B	37.12%	29.80%	62.65%	53.10%
	Vicuna_7B	36.12%	28.28%	54.35%	49.75%
Filtered	Direct filtering	30.08%	24.32%	54.05%	46.31%
	ICL filtering	29.90%	23.95%	55.28%	47.02%
	CoT filtering	30.19%	24.18%	56.03%	46.95%
	Self-RAG_Llama-2	39.10%	30.27%	65.17%	52.08%
	Self-RAG_Mistral	32.30%	26.07%	60.65%	50.57%
	Rep-PCA(Mistral)	70.73%	29.81%	73.63%	56.16%
	Rep-Con(Mistral)	72.53%	32.39%	72.62%	57.62%
	Rep-PCA(Llama-2)	67.93%	31.32%	66.78%	53.97%
	Rep-Con(Llama-2)	69.95%	33.64%	67.59%	54.26%
	Rep-PCA(Llama-3)	67.81%	35.32%	71.16%	50.18%
	Rep-Con(Llama-3)	69.81%	36.75%	72.16%	52.26%