notesum.ai

Published at November 27

Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment

cs.MM

Released Date: November 27, 2024

Authors: Shima Mohammadi¹, João Ascenso

Aff.: ¹Institution information not provided

Arxiv: http://arxiv.org/abs/2411.18372v1

Refer to caption

Method	Metric	Trial percentage
Method	Metric	$2.5\%$	$5\%$	$7.5\%$	$10\%$	$15\%$	$20\%$	$30\%$	$40\%$	$50\%$
LBPS-EIC	PLCC	0.89	0.90	0.92	0.93	0.94	0.96	0.97	0.98	0.99
LBPS-EIC	SROCC	0.89	0.90	0.90	0.91	0.92	0.93	0.94	0.96	0.97
ASAP	PLCC	0.55	0.71	0.79	0.82	0.87	0.89	0.92	0.94	0.95
ASAP	SROCC	0.73	0.85	0.90	0.92	0.94	0.96	0.97	0.98	0.98
Crowd-BT	PLCC	0.53	0.67	0.74	0.79	0.83	0.86	0.89	0.90	0.91
Crowd-BT	SROCC	0.64	0.78	0.85	0.89	0.93	0.94	0.95	0.96	0.96
Hybrid-MST	PLCC	0.67	0.78	0.83	0.86	0.90	0.92	0.94	0.95	0.95
Hybrid-MST	SROCC	0.77	0.87	0.90	0.92	0.95	0.96	0.97	0.98	0.98
PS-PC	PLCC	-	-	-	0.80	-	0.84	0.86	0.89	0.89
PS-PC	SROCC	-	-	-	0.44	-	0.49	0.53	0.60	0.62
Random	PLCC	0.30	0.42	0.49	0.55	0.61	0.66	0.74	0.78	0.83
Random	SROCC	0.31	0.48	0.6	0.67	0.75	0.81	0.87	0.91	0.93