notesum.ai

Published at October 21

Augmenting Legal Decision Support Systems with LLM-based NLI for Analyzing Social Media Evidence

cs.CY

cs.AI

cs.LG

Released Date: October 21, 2024

Authors: Ram Mohan Rao Kadiyala¹, Siddartha Pullakhandam², Kanwal Mehreen³, Subhasya Tippareddy⁴, Ashay Srivastava¹

Aff.: ¹University of Maryland; ²University of Wisconsin; ³Traversaal.ai; ⁴University of South Florida

Arxiv: https://arxiv.org/abs/2410.15990v1

Refer to caption

LLM Used	Trained on	Alignment approach	A	P	R	F1
GEMMA-2-27B	NLLP*	None	0.857	0.871	0.894	0.871
GEMMA-2-27B	NLLP	None	0.857	0.859	0.891	0.865
Mistral-8x7B	NLLP*	None	0.869	0.877	0.902	0.881
QWEN-2-7B	NLLP*	None	0.833	0.828	0.868	0.839
QWEN-2-7B	NLLP	None	0.821	0.852	0.869	0.842
Phi-3-Medium	NLLP*	None	0.821	0.853	0.813	0.820
OpenHermes-13B	NLLP*	None	0.774	0.820	0.832	0.803
GEMMA-2-27B	SNLI, NLLP*	None	0.869	0.866	0.899	0.874
GEMMA-2-27B	SNLI, NLLP	None	0.821	0.828	0.862	0.831
GEMMA-2-27B	SNLI, NLLP*	ORPO Random	0.845	0.852	0.882	0.855
GEMMA-2-27B	NLLP*	ORPO Multiple	0.833	0.842	0.860	0.840
GEMMA-2-27B	SNLI, NLLP*	ORPO Preferred	0.869	0.885	0.902	0.887
Mistral-NEMO	NLLP*	ORPO Multiple	0.869	0.867	0.890	0.877
Phi-3-Medium	NLLP*	ORPO Multiple	0.845	0.872	0.833	0.838
Zephyr-7B	NLLP*	ORPO Multiple	0.810	0.838	0.858	0.832
Phi-3-Medium‘	NLLP‘	ORPO Multiple‘	0.845‘	0.884‘	0.844‘	0.853‘
baseline	-	-	-	-	-	0.807