notesum.ai

Published at October 30

A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction

cs.LG

cs.AI

Released Date: October 30, 2024

Authors: Qidong Yang¹, Weicheng Zhu², Joseph Keslin³, Laure Zanna², Tim G. J. Rudner², Carlos Fernandez-Granda²

Aff.: ¹MIT; ²New York University; ³University of Illinois Urbana-Champaign

Arxiv: http://arxiv.org/abs/2410.23272v1

Refer to caption

Scenario	Regularization	ECE $(\downarrow)$	AUC $(\uparrow)$	CE $(\downarrow)$	BS $(\downarrow)$
Seaquest	✗	0.0435 $\pm$ 0.0004	0.8671 $\pm$ 0.0035	1.0577 $\pm$ 0.0241	0.1247 $\pm$ 0.0012
	time-dependent	0.0277 $\pm$ 0.0023	0.8678 $\pm$ 0.0028	0.6705 $\pm$ 0.0285	0.1144 $\pm$ 0.0007
	constant	0.0365 $\pm$ 0.0002	0.8625 $\pm$ 0.0020	0.8173 $\pm$ 0.0068	0.1177 $\pm$ 0.0008
River Raid	✗	0.0583 $\pm$ 0.0016	0.6453 $\pm$ 0.0009	1.2034 $\pm$ 0.0281	0.1750 $\pm$ 0.0016
	time-dependent	0.0388 $\pm$ 0.0013	0.6346 $\pm$ 0.0035	0.8585 $\pm$ 0.0132	0.1671 $\pm$ 0.0012
	constant	0.0474 $\pm$ 0.0004	0.6280 $\pm$ 0.0020	1.0274 $\pm$ 0.0158	0.1686 $\pm$ 0.0005
Bank Heist	✗	0.0559 $\pm$ 0.0032	0.6938 $\pm$ 0.0028	1.1874 $\pm$ 0.0540	0.2340 $\pm$ 0.0020
	time-dependent	0.0148 $\pm$ 0.0014	0.6782 $\pm$ 0.0016	0.7647 $\pm$ 0.0130	0.2166 $\pm$ 0.0005
	constant	0.0399 $\pm$ 0.0016	0.6928 $\pm$ 0.0046	0.8894 $\pm$ 0.0112	0.2211 $\pm$ 0.0007
H.E.R.O.	✗	0.0947 $\pm$ 0.0014	0.6785 $\pm$ 0.0061	1.1310 $\pm$ 0.0225	0.1261 $\pm$ 0.0009
	time-dependent	0.0481 $\pm$ 0.0034	0.7159 $\pm$ 0.0105	0.6940 $\pm$ 0.0246	0.1170 $\pm$ 0.0008
	constant	0.0352 $\pm$ 0.0001	0.7041 $\pm$ 0.0102	0.7218 $\pm$ 0.0391	0.1212 $\pm$ 0.0011
Road Runner	✗	0.0779 $\pm$ 0.0035	0.6913 $\pm$ 0.0100	1.1586 $\pm$ 0.0291	0.1575 $\pm$ 0.0034
	time-dependent	0.0204 $\pm$ 0.0012	0.6823 $\pm$ 0.0084	0.5255 $\pm$ 0.0077	0.1382 $\pm$ 0.0003
	constant	0.0303 $\pm$ 0.0027	0.6898 $\pm$ 0.0140	0.6275 $\pm$ 0.0250	0.1394 $\pm$ 0.0010
FaceMed	✗	0.1503 $\pm$ 0.0048	0.7534 $\pm$ 0.0079	1.6932 $\pm$ 0.0188	0.3464 $\pm$ 0.0012
	time-dependent	0.0757 $\pm$ 0.0068	0.7614 $\pm$ 0.0024	0.9085 $\pm$ 0.0303	0.3328 $\pm$ 0.0008
	constant	0.0974 $\pm$ 0.0045	0.7613 $\pm$ 0.0028	1.0071 $\pm$ 0.0499	0.3356 $\pm$ 0.0030