notesum.ai

Published at October 22

Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination

cs.LG

cs.AI

Released Date: October 22, 2024

Authors: Jerry Huang¹, Prasanna Parthasarathi², Mehdi Rezagholizadeh², Boxing Chen², Sarath Chandar³

Aff.: ¹Mila - Quebec AI Institute, Université de Montréal; ²Huawei Noah's Ark Lab; ³Mila - Quebec AI Institute, Polytechnique Montréal

Arxiv: https://arxiv.org/abs/2410.17477v1

Model Name	Hallu. Detection	Instr. Following	Closed-Book QA	Reading Comp.	Sum.	Fact-Checking	Faithfulness	Factuality
Attention-Only Models
Gemma-2B	49.65 ( $\uparrow$ 11.51)	30.25 ( $\uparrow$ 13.84)	38.20 ( $\downarrow$ -6.92)	38.96 ( $\downarrow$ -13.96)	24.93 ( $\downarrow$ -4.53)	54.14 ( $\uparrow$ 1.60)	35.85 ( $\uparrow$ 1.92)	32.56 ( $\downarrow$ -4.56)
Gemma-7B	53.81 ( $\downarrow$ -5.67)	31.94 ( $\uparrow$ 6.79)	27.43 ( $\downarrow$ -4.13)	31.35 ( $\downarrow$ -5.83)	20.63 ( $\uparrow$ 1.04)	43.28 ( $\downarrow$ -5.22)	36.48 ( $\uparrow$ 2.71)	44.50 ( $\downarrow$ -6.04)
Gemma2-2B	58.33 ( $\uparrow$ 0.64)	26.73 ( $\uparrow$ 10.90)	29.59 ( $\uparrow$ 4.83)	32.95 ( $\downarrow$ -5.26)	16.04 ( $\downarrow$ -0.47)	39.57 ( $\uparrow$ 18.96)	35.61 ( $\uparrow$ 0.65)	34.96 ( $\downarrow$ -0.98)
Gemma2-9B	65.64 ( $\uparrow$ 6.31)	23.76 ( $\uparrow$ 30.07)	39.36 ( $\uparrow$ 3.13)	39.74 ( $\downarrow$ -0.70)	21.38 ( $\downarrow$ -3.35)	62.06 ( $\uparrow$ 6.47)	40.56 ( $\uparrow$ 7.55)	46.85 ( $\uparrow$ 3.13)
Gemma2-27B	62.03 ( $\uparrow$ 12.43)	26.62 ( $\uparrow$ 33.58)	47.19 ( $\uparrow$ 2.86)	43.04 ( $\uparrow$ 5.02)	28.92 ( $\downarrow$ -0.74)	68.28 ( $\uparrow$ 1.00)	41.54 ( $\uparrow$ 13.02)	53.92 ( $\uparrow$ 2.36)
LLaMA2-7B	53.63 ( $\uparrow$ 4.30)	28.94 ( $\uparrow$ 8.99)	37.64 ( $\downarrow$ -1.89)	27.58 ( $\uparrow$ 3.74)	25.19 ( $\uparrow$ 0.69)	51.38 ( $\uparrow$ 5.88)	35.14 ( $\uparrow$ 4.01)	42.51 ( $\downarrow$ -0.12)
LLaMA2-13B	67.80 ( $\downarrow$ -3.68)	26.57 ( $\uparrow$ 9.72)	39.65 ( $\downarrow$ -0.65)	32.17 ( $\downarrow$ -3.64)	27.33 ( $\downarrow$ -0.23)	62.35 ( $\downarrow$ -1.64)	40.83 ( $\uparrow$ 0.88)	46.27 ( $\downarrow$ -0.49)
LLaMA2-70B	62.34 ( $\uparrow$ 10.78)	30.24 ( $\uparrow$ 17.11)	47.85 ( $\downarrow$ -2.53)	39.48 ( $\downarrow$ -8.21)	28.00 ( $\downarrow$ -0.55)	66.63 ( $\downarrow$ -1.45)	41.74 ( $\uparrow$ 5.82)	53.69 ( $\downarrow$ -2.34)
LLaMA3-8B	60.76 ( $\uparrow$ 10.12)	22.06 ( $\uparrow$ 26.76)	41.55 ( $\uparrow$ 1.30)	33.52 ( $\uparrow$ 3.62)	26.62 ( $\downarrow$ -1.60)	60.84 ( $\uparrow$ 4.82)	37.60 ( $\uparrow$ 10.09)	48.14 ( $\uparrow$ 1.60)
LLaMA3-70B	71.78 ( $\uparrow$ 8.69)	21.59 ( $\uparrow$ 25.60)	48.07 ( $\uparrow$ 3.43)	46.38 ( $\downarrow$ -7.73)	28.91 ( $\downarrow$ -1.68)	69.57 ( $\uparrow$ 0.99)	45.92 ( $\uparrow$ 6.23)	54.80 ( $\uparrow$ 2.81)
Mistral-7B	60.48 ( $\uparrow$ 3.36)	28.39 ( $\uparrow$ 16.85)	41.24 ( $\uparrow$ 5.21)	32.03 ( $\uparrow$ 1.92)	26.77 ( $\downarrow$ -0.77)	58.59 ( $\uparrow$ 6.78)	38.47 ( $\uparrow$ 4.67)	47.42 ( $\uparrow$ 4.17)
Mixtral-8x7B	73.51 ( $\downarrow$ -0.15)	26.84 ( $\uparrow$ 17.60)	48.99 ( $\uparrow$ 4.33)	40.66 ( $\downarrow$ -3.90)	27.91 ( $\downarrow$ -1.07)	68.35 ( $\uparrow$ 0.49)	46.16 ( $\uparrow$ 3.89)	55.15 ( $\uparrow$ 3.50)
Falcon-7B	52.40 ( $\downarrow$ -2.90)	25.54 ( $\uparrow$ 12.62)	33.03 ( $\downarrow$ -4.42)	29.18 ( $\downarrow$ -3.64)	20.70 ( $\downarrow$ -1.00)	46.91 ( $\downarrow$ -7.71)	34.91 ( $\downarrow$ -0.71)	36.92 ( $\downarrow$ -3.75)
Recurrent and Hybrid Models
RecurrentGemma-2B	52.88 ( $\uparrow$ 2.33)	33.43 ( $\uparrow$ 3.12)	27.36 ( $\downarrow$ -1.85)	30.66 ( $\downarrow$ -1.77)	18.41 ( $\uparrow$ 2.86)	42.97 ( $\uparrow$ 3.63)	36.55 ( $\downarrow$ -0.14)	31.46 ( $\uparrow$ 1.39)
RecurrentGemma-9B	55.75 ( $\downarrow$ -1.67)	31.67 ( $\uparrow$ 12.96)	36.79 ( $\uparrow$ 2.03)	36.57 ( $\downarrow$ -6.66)	22.99 ( $\uparrow$ 1.84)	51.25 ( $\uparrow$ 13.36)	37.61 ( $\downarrow$ -0.17)	42.96 ( $\uparrow$ 3.28)
Jamba	57.66 ( $\downarrow$ -2.90)	43.36 ( $\downarrow$ -5.76)	39.50 ( $\downarrow$ -0.98)	33.00 ( $\downarrow$ -5.88)	23.72 ( $\downarrow$ -15.48)	59.88 ( $\downarrow$ -2.60)	39.80 ( $\downarrow$ -6.91)	45.95 ( $\downarrow$ -0.91)
FalconMamba-7B	55.80 ( $\uparrow$ 0.73)	42.97 ( $\downarrow$ -1.19)	39.94 ( $\downarrow$ -0.43)	23.76 ( $\downarrow$ -0.85)	23.89 ( $\downarrow$ -0.09)	61.85 ( $\downarrow$ -1.80)	35.92 ( $\downarrow$ -0.26)	47.02 ( $\downarrow$ -0.38)