notesum.ai

Published at October 18

Enhancing Large Language Models' Situated Faithfulness to External Contexts

cs.CL

cs.AI

Released Date: October 18, 2024

Authors: Yukun Huang¹, Sanxing Chen¹, Hongyi Cai¹, Bhuwan Dhingra¹

Aff.: ¹Duke University

Arxiv: https://arxiv.org/abs/2410.14675v1

GPT-4o-m	RedditQA			FreshQA			ClashEval			TriviaQA			PopQA			NaturalQA			Total
Method	TR	FA	OV	TR	FA	OV	TR	FA	OV	TR	FA	OV	TR	FA	OV	TR	FA	OV	OV
Closed-book			80.7			53.0			20.0			81.7			56.7			62.0	59.0
DIA	96.0	12.5	54.3	96.3	2.3	49.3	85.3	12.0	48.7	96.0	12.3	54.2	97.7	11.0	54.4	88.7	10.3	49.5	51.7
TACS (LR)	93.8	16.5	55.2	86.3	4.6	45.5	76.3	14.3	45.3	92.3	16.0	54.2	76.3	14.3	45.3	86.0	15.7	50.9	49.4
Rule-based Confidence Reasoning (Evaluation-based)
ContextEval	90.3	53.4	71.9	77.3	35.7	56.5	86.7	11.7	49.2	92.3	47.4	69.9	91.0	53.0	\ul72.0	86.6	36.3	61.5	63.5
InternalEval	88.6	77.2	82.9	83.3	30.0	56.7	67.0	17.0	42.0	88.7	71.7	80.2	78.0	48.3	63.2	72.3	57.3	64.8	65.0
Rule-based Confidence Reasoning (Confidence-based)
TPC	92.0	25.6	58.8	88.7	22.7	55.7	82.7	20.3	\ul51.5	94.7	39.7	67.2	95.0	17.7	56.4	87.0	34.3	60.7	58.4
ContextConf	93.8	13.1	53.5	82.3	16.3	49.3	80.0	16.7	48.4	93.0	19.7	56.4	94.0	14.6	54.2	85.0	30.6	57.8	53.2
InternalConf	82.3	79.5	80.9	83.6	38.3	\ul61.0	64.6	20.6	42.6	89.6	77.0	83.3	84.0	50.0	67.0	84.0	49.3	66.7	66.9
Self-guided Confidence Reasoning
ImplicitSCR	91.5	79.0	85.3	93.7	26.3	60.0	89.3	25.3	57.3	96.3	48.0	72.2	97.0	48.3	72.7	82.0	50.0	\ul66.0	\ul68.9
ExplicitSCR	92.0	73.9	\ul83.0	82.7	47.0	64.9	82.0	17.0	49.5	92.3	71.3	\ul81.8	93.0	51.0	\ul72.0	80.3	51.0	65.7	69.5
GPT-4o	RedditQA			FreshQA			ClashEval			TriviaQA			PopQA			NaturalQA			Total
Method	TR	FA	OV	TR	FA	OV	TR	FA	OV	TR	FA	OV	TR	FA	OV	TR	FA	OV	OV
Closed-book			84.1			63.3			39.0			90.7			81.0			66.7	70.8
DIA	94.9	12.5	53.7	93.7	5.7	49.7	84.7	24.7	54.7	96.0	23.3	59.7	97.3	12.3	54.8	88.7	12.7	50.7	53.9
TACS (LR)	96.0	17.0	56.5	81.3	9.3	45.3	82.7	25.7	54.2	96.0	26.7	61.4	96.7	21.3	59.0	88.3	20.7	54.5	55.1
Rule-based Confidence Reasoning (Evaluation Based)
ContextEval	92.6	66.5	79.6	87.7	44.0	65.9	81.3	23.0	52.2	94.3	71.3	82.8	96.7	73.3	85.0	86.7	51.3	69.0	72.4
InternalEval	87.5	81.3	84.4	81.3	40.0	60.7	76.7	30.0	53.4	95.0	85.7	\ul90.4	88.7	70.3	79.5	74.3	60.3	67.3	72.6
Rule-based Confidence Reasoning (Confidence Based)
TPC	94.3	71.6	83.0	87.0	39.7	63.4	77.0	39.0	58.0	94.3	75.3	84.8	96.7	66.7	81.7	84.0	50.3	67.2	73.0
ContextConf	93.8	60.2	77.0	78.7	37.0	57.9	70.7	30.0	50.4	94.6	43.0	68.8	95.7	26.3	61.0	78.7	36.7	57.7	62.1
InternalConf	89.8	78.9	84.4	83.7	50.0	66.9	61.6	40.6	51.1	93.0	88.0	90.5	88.7	78.7	83.7	78.6	59.6	69.1	74.3
Self-guided Confidence Reasoning
ImplicitSCR	92.6	75.6	84.1	82.0	54.0	\ul68.0	86.0	44.3	65.2	94.7	85.7	90.2	96.0	76.7	86.4	83.3	66.7	75.0	78.1
EplicitSCR	93.8	77.2	85.5	85.3	55.3	70.3	83.0	33.3	58.2	94.7	84.3	89.5	92.7	78.0	\ul85.4	78.7	60.0	\ul69.4	\ul76.4