notesum.ai

Published at October 21

Reducing annotator bias by belief elicitation

cs.CL

cs.AI

Released Date: October 21, 2024

Authors: Terne Sasha Thorn Jakobsen¹, Andreas Bjerre-Nielsen¹, Robert Böhm²

Aff.: ¹Copenhagen Center for Social Data Science, University of Copenhagen; ²Department of Psychology, University of Vienna

Arxiv: https://arxiv.org/abs/2410.15726v1

Refer to caption

		M $\pm$ SD	Mdn[Quartiles]	Within $p$	M $\pm$ SD	Mdn[Quartiles]	Within $p$	Between $p$
		Experiment 1
		Democrats (n=641)			Republicans (n=619)
D	Judgement	0.72 $\pm$ 0.26	0.78[0.51;0.95]	4.45 $\times 10^{-39}$	0.62 $\pm$ 0.29	0.63[0.46;0.88]	2.34 $\times 10^{-10}$	5.58 $\times 10^{-10}$
D	Belief	0.59 $\pm$ 0.14	0.56[0.50;0.69]	4.45 $\times 10^{-39}$	0.56 $\pm$ 0.15	0.54[0.49;0.65]	2.34 $\times 10^{-10}$	$1.42\times 10^{-3}$
R	Judgement	0.44 $\pm$ 0.35	0.43[0.12;0.77]	6.87 $\times 10^{-4}$	0.56 $\pm$ 0.31	0.57[0.34;0.85]	1.69 $\times 10^{-7}$	2.52 $\times 10^{-10}$
R	Belief	0.48 $\pm$ 0.15	0.50[0.41;0.55]	6.87 $\times 10^{-4}$	0.51 $\pm$ 0.16	0.51[0.43;0.58]	1.69 $\times 10^{-7}$	$1.55\times 10^{-3}$
		Experiment 2
		Democrats (n=169)			Republicans (n=161)
D	Judgement	0.73 $\pm$ 0.23	0.78[0.57;0.91]	4.19 $\times 10^{-16}$	0.66 $\pm$ 0.27	0.73[0.50;0.89]	1.66 $\times 10^{-5}$	0.022
D	Belief	0.56 $\pm$ 0.11	0.55[0.50;0.62]	4.19 $\times 10^{-16}$	0.58 $\pm$ 0.12	0.56[0.50;0.62]	1.66 $\times 10^{-5}$	0.83
R	Judgement	0.51 $\pm$ 0.31	0.50[0.26;0.78]	0.48	0.56 $\pm$ 0.34	0.60[0.25;0.89]	0.026	0.046
R	Belief	0.52 $\pm$ 0.12	0.51[0.46;0.56]	0.48	0.51 $\pm$ 0.11	0.50[0.46;0.55]	0.026	0.87