notesum.ai

Published at October 20

LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content

cs.CL

cs.AI

cs.DC

cs.LG

Released Date: October 20, 2024

Authors: Mohamed Bayan Kmainasi¹, Ali Ezzat Shahroor², Maram Hasanain², Sahinur Rahman Laskar³, Naeemul Hassan⁴, Firoj Alam²

Aff.: ¹Qatar University, Qatar; ²Qatar Computing Research Institute, Qatar; ³UPES, India; ⁴University of Maryland, USA

Arxiv: https://arxiv.org/abs/2410.15308v1

Task	Dataset	Metric	SOTA	Llama	Ours	$\Delta$	Task	Dataset	Metric	SOTA	Llama	Ours	$\Delta$
Arabic							English
Att.worthiness	CT22Attentionworthy	W-F1	0.412	0.158	0.544	0.132	Checkworthiness	CT24_T1	F1_Pos	0.753	0.404	0.877	0.124
Checkworthiness	CT24_T1	F1_Pos	0.569	0.404	0.877	0.308	Claim	claim-detection	Mi-F1	–	0.545	0.915	–
Claim	CT22Claim	Acc	0.703	0.581	0.778	0.075	Cyberbullying	Cyberbullying	Acc	0.907^∗	0.175	0.847	-0.060
Cyberbullying	ArCyc_CB	Acc	0.863^∗	0.766	0.753	-0.110	Emotion	emotion	Ma-F1	0.790^∗	0.353	0.878	0.088
Emotion	Emotional-Tone	W-F1	0.658^∗	0.358	0.748	0.090	Factuality	News_dataset	Acc	0.920^∗	0.654	0.946	0.026
Emotion	NewsHeadline	Acc	1.000^∗	0.406	0.551	-0.449	Factuality	Politifact	W-F1	0.490^∗	0.121	0.290	-0.200
Factuality	Arafacts	Mi-F1	0.850^∗	0.210	0.534	-0.316	News Cat.	CNN_News_Articles	Acc	0.940	0.644	0.915	-0.025
Factuality	COVID19Factuality	W-F1	0.831	0.492	0.781	-0.050	News Cat.	News_Category	Ma-F1	0.769^∗	0.970	0.505	-0.264
Harmfulness	CT22Harmful	F1_Pos	0.557	0.507	0.508	-0.049	News Genre	SemEval23T3-ST1	Mi-F1	0.815	0.687	0.241	-0.574
Hate Speech	annotated-hatetweets-4	W-F1	0.630	0.257	0.549	-0.081	News Sum.	xlsum	R-2	0.152	0.074	0.141	-0.010
Hate Speech	OSACT4SubtaskB	Mi-F1	0.950	0.819	0.802	-0.148	Offensive	Offensive_Hateful	Mi-F1	–	0.692	0.805	–
News Cat.	ASND	Ma-F1	0.770^∗	0.587	0.938	0.168	Offensive	offensive_language	Mi-F1	0.994	0.646	0.884	-0.110
News Cat.	SANADAkhbarona	Acc	0.940	0.784	0.922	-0.018	Offensive & Hate	hate-offensive-speech	Acc	0.945	0.602	0.924	-0.021
News Cat.	SANADAlArabiya	Acc	0.974	0.893	0.986	0.012	Propaganda	QProp	Ma-F1	0.667	0.759	0.851	0.184
News Cat.	SANADAlkhaleej	Acc	0.986	0.865	0.967	-0.019	Sarcasm	News-Headlines	Acc	0.897^∗	0.668	0.956	0.059
News Cat.	UltimateDataset	Ma-F1	0.970	0.376	0.883	-0.087	Sentiment	NewsMTSC	Ma-F1	0.817	0.628	0.627	-0.190
News Credibility	NewsCredibility	Acc	0.899^∗	0.455	0.494	-0.405	Subjectivity	CT24_T2	Ma-F1	0.744	0.535	0.508	-0.236
News Sum.	xlsum	R-2	0.137	0.034	0.075	-0.063	Hindi
Offensive	ArCyc_OFF	Ma-F1	0.878^∗	0.489	0.652	-0.226	Factuality	fake-news	Mi-F1	–	0.759	0.713	–
Offensive	OSACT4SubtaskA	Ma-F1	0.905	0.782	0.899	-0.006	Hate Speech	hate-speech-detection	Mi-F1	0.639^∗	0.750	0.994	0.355
Propaganda	ArPro	Mi-F1	0.767	0.597	0.762	-0.005	Hate Speech	Hindi-Hostility	W-F1	0.841^∗	0.469	0.720	-0.121
Sarcasm	ArSarcasm-v2	F1_Pos	0.584	0.477	0.307	-0.277	NLI	NLI_dataset	W-F1	0.646	0.633	0.655	0.009
Sentiment	ar_reviews_100k	F1_Pos	–	0.343	0.665	–	News Sum.	xlsum	R-2	0.136	0.078	0.117	-0.019
Sentiment	ArSAS	Acc	0.920^∗	0.603	0.795	-0.125	Offensive	Offensive Speech	Mi-F1	0.723	0.621	0.847	0.124
Stance	stance	Ma-F1	0.767	0.608	0.936	0.169	Cyberbullying	MC-Hinglish1.0	Acc	0.609	0.233	0.587	-0.022
Stance	Mawqif-Arabic-Stance	Ma-F1	0.789	0.764	0.867	0.079	Sentiment	Sentiment Analysis	Acc	0.697	0.552	0.669	-0.028
Subjectivity	ThatiAR	F1_Pos	0.800	0.562	0.207	-0.593