notesum.ai

Published at October 30

Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers

cs.SE

cs.CL

cs.CR

Released Date: October 30, 2024

Authors: Lam Nguyen Tung¹, Steven Cho², Xiaoning Du¹, Neelofar Neelofar¹, Valerio Terragni², Stefano Ruberto³, Aldeida Aleti¹

Aff.: ¹Monash University, Australia; ²University of Auckland, New Zealand; ³JRC European Commission, Italy

Arxiv: http://arxiv.org/abs/2410.22663v1

Refer to caption

Dataset		Data Statistics			Model Under Test		Number of top words	Importance Type
Dataset		Trust	Untrust	Total	Model Type	Accuracy	Number of top words	Importance Type
Dong’s (Dong, 2018)	movie	311	47	358	MLP	0.832	10, 20	importance
Dong’s (Dong, 2018)	20news	311	47	358	MLP	0.939	10, 20	equivalent
Garg et al.’s (Garg et al., 2022)	CAMS	1,206	739	1,945	mentalbert-base-uncased	0.397	10	importance equivalent
Mathew et al.’s (Mathew et al., 2021)	HateXplain	3,002	304	3,306	bert-base-uncased	0.797	10	importance equivalent
Ours	amazon_polarity	226	19	245	roberta-base-cased	0.960	5, 10, 20	importance different
	ag_news				bert-base-uncased	0.934
	rotten_tomatoes				distilbert-base-uncased	0.841
	yahoo_answers_topics				bert-base-uncased	0.750
	imdb				distilbert-base-uncased	0.928
	emotion				distilbert-base-uncased	0.926