notesum.ai

Published at October 24

DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations

cs.LG

cs.AI

Released Date: October 24, 2024

Authors: Aryo Pradipta Gema¹, Chen Jin, Ahmed Abdulaal², Tom Diethe², Philip Teare¹, Beatrice Alex¹, Pasquale Minervini¹, Amrutha Saseendran³

Aff.: ¹University of Edinburgh, United Kingdom; ²University College London, United Kingdom; ³AstraZeneca, United Kingdom

Arxiv: https://arxiv.org/abs/2410.18860v1

Refer to caption

Model	XSum			MemoTrap		IFEval		NQ-Open	NQ-Swap
Model	ROUGE-L $\uparrow$	BERTScore-F1 $\uparrow$	factKB $\uparrow$	Macro Acc $\uparrow$	Micro Acc $\uparrow$	Prompt Acc $\uparrow$	Instruct Acc $\uparrow$	EM $\uparrow$	EM $\uparrow$
Llama3-8b-Instruct	19.90	67.23	47.61	65.86	64.40	70.24	78.30	69.68	60.62
+ ITI (Li et al., 2024b)	13.25	59.96	34.35	62.65	58.96	52.31	63.19	56.16	51.08
+ CAD (Shi et al., 2024)	18.82	67.20	67.16	-	-	-	-	69.83	74.21
+ DoLA (low) (Chuang et al., 2023)	19.82	67.19	47.21	65.27	63.69	69.69	78.18	69.68	60.77
+ DoLA (high) (Chuang et al., 2023)	19.92	67.34	48.49	64.85	63.17	70.24	78.66	69.49	60.98
+ AD (Chen et al., 2024)	19.79	67.31	48.49	65.38	64.28	67.65	76.26	68.93	60.51
+ DeCoRe ${}_{\text{static}}$	19.87	67.83	64.07	69.53	69.20	69.13	78.06	70.62	64.43
+ DeCoRe ${}_{\text{entropy}}$	19.45	67.69	66.10	74.14	74.87	68.39	76.38	70.66	66.08
Llama3-70b-Instruct	22.41	69.77	61.32	68.47	66.52	77.45	84.41	71.07	76.11
+ ITI (Li et al., 2024b)	21.64	69.46	61.33	71.24	68.73	76.71	83.69	71.90	74.76
+ CD (Li et al., 2023)	22.71	69.99	54.73	69.27	67.55	71.72	79.74	65.80	68.37
+ CAD (Shi et al., 2024)	21.45	69.28	65.61	-	-	-	-	71.83	84.70
+ DoLA (low) (Chuang et al., 2023)	22.46	69.80	61.11	67.99	65.93	77.08	84.29	71.07	75.98
+ DoLA (high) (Chuang et al., 2023)	22.43	69.93	59.99	67.92	65.81	78.00	84.65	70.40	75.26
+ AD (Chen et al., 2024)	22.49	69.91	60.57	67.51	66.44	76.89	84.41	71.15	74.02
+ DeCoRe ${}_{\text{static}}$	21.94	69.35	64.88	71.96	71.41	78.56	84.89	72.51	79.06
+ DeCoRe ${}_{\text{entropy}}$	21.93	69.40	65.49	74.07	73.65	78.56	84.89	72.66	79.79
+ DeCoRe ${}_{\text{entropy-lite}}$	22.28	69.34	59.57	72.11	70.58	61.37	71.46	71.26	75.90