notesum.ai

Published at December 5

Hostility Detection in UK Politics: A Dataset on Online Abuse Targeting MPs

cs.CL

Released Date: December 5, 2024

Authors: Mugdha Pandya¹, Mali Jin¹, Kalina Bontcheva¹, Diana Maynard¹

Aff.: ¹University of Sheffield

Arxiv: http://arxiv.org/pdf/2412.04046v1

Flat Classification
Model	Accuracy	Precision	Recall	F1
Set 1
BERT	61.93 $\pm$ 2.03	31.42 $\pm$ 2.14	28.31 $\pm$ 2.36	28.18 $\pm$ 2.91
RoBERTa	63.13 $\pm$ 0.35	24.40 $\pm$ 0.32	26.03 $\pm$ 0.20	20.79 $\pm$ 0.21
RoBERTa-Hate	64.01 $\pm$ 1.41	45.81 $\pm$ 6.34	34.74 $\pm$ 3.90	36.67 $\pm$ 4.54
Set 2
BERT	65.93 $\pm$ 1.68	37.12 $\pm$ 1.62	34.77 $\pm$ 2.79	35.07 $\pm$ 2.72
RoBERTa	67.26 $\pm$ 1.05	33.01 $\pm$ 5.96	32.25 $\pm$ 3.95	32.04 $\pm$ 4.68
RoBERTa-Hate	69.64 $\pm$ 1.24	50.82 $\pm$ 7.61	41.66 $\pm$ 4.90	43.31 $\pm$ 5.65
LLaMA	55.48 $\pm$ 2.29	42.01 $\pm$ 5.38	46.60 $\pm$ 8.18	38.09 $\pm$ 5.69
LLaMA w/ Def.	61.33 $\pm$ 1.41	48.61 $\pm$ 3.58	52.39 $\pm$ 3.68	47.02 $\pm$ 3.25
GPT	65.57 $\pm$ 1.77	52.64 $\pm$ 4.21	55.02 $\pm$ 5.21	52.65 $\pm$ 4.45
GPT w/ Def.	65.03 $\pm$ 1.71	53.70 $\pm$ 2.71	56.89 $\pm$ 2.69	54.74 $\pm$ 2.22
Set 3
BERT	65.03 $\pm$ 2.31	19.45 $\pm$ 3.80	18.43 $\pm$ 2.03	18.23 $\pm$ 2.66
RoBERTa	66.66 $\pm$ 1.72	20.30 $\pm$ 3.68	20.43 $\pm$ 3.44	20.08 $\pm$ 3.64
RoBERTa-Hate	68.70 $\pm$ 2.64	28.55 $\pm$ 8.12	23.51 $\pm$ 3.70	23.98 $\pm$ 4.49
Hierarchical Classification
Set 1
BERT	60.78 $\pm$ 1.00	27.44 $\pm$ 5.76	25.60 $\pm$ 2.01	24.79 $\pm$ 2.14
RoBERTa	61.99 $\pm$ 1.32	37.66 $\pm$ 9.18	27.53 $\pm$ 2.08	28.87 $\pm$ 2.44
RoBERTa-Hate	62.47 $\pm$ 2.29	38.77 $\pm$ 5.79	28.42 $\pm$ 1.58	31.21 $\pm$ 2.39
Set 2
BERT	66.30 $\pm$ 4.52	32.42 $\pm$ 2.08	28.41 $\pm$ 2.95	29.09 $\pm$ 3.08
RoBERTa	66.14 $\pm$ 1.70	40.77 $\pm$ 8.44	30.47 $\pm$ 6.38	32.85 $\pm$ 7.03
RoBERTa-Hate	68.10 $\pm$ 1.57	39.93 $\pm$ 4.37	32.18 $\pm$ 4.57	33.81 $\pm$ 4.63
LLaMA	64.79 $\pm$ 1.97	54.62 $\pm$ 3.75	51.77 $\pm$ 3.83	52.15 $\pm$ 3.65
LLaMA w/ Def.	64.70 $\pm$ 2.37	53.11 $\pm$ 11.04	53.98 $\pm$ 3.67	54.16 $\pm$ 4.43
GPT	54.19 $\pm$ 2.77	55.61 $\pm$ 5.11	54.29 $\pm$ 5.79	50.53 $\pm$ 5.08
GPT w/ Def.	64.43 $\pm$ 1.52	54.15 $\pm$ 3.42	60.02 $\pm$ 3.11	55.98 $\pm$ 3.08
Set 3
BERT	66.30 $\pm$ 4.32	21.53 $\pm$ 2.29	19.14 $\pm$ 1.51	19.49 $\pm$ 1.60
RoBERTa	65.84 $\pm$ 2.24	30.52 $\pm$ 8.89	23.01 $\pm$ 6.86	23.60 $\pm$ 6.55
RoBERTa-Hate	67.80 $\pm$ 2.07	26.00 $\pm$ 2.28	25.09 $\pm$ 3.29	24.22 $\pm$ 2.96