notesum.ai

Published at November 25

Interpreting Language Reward Models via Contrastive Explanations

cs.AI

Released Date: November 25, 2024

Authors: Junqi Jiang¹, Tom Bewley², Saumitra Mishra², Freddy Lecue², Manuela Veloso²

Aff.: ¹Imperial College London; ²J.P. Morgan AI Research

Arxiv: http://arxiv.org/abs/2411.16502v1

Refer to caption

Dataset	Method	Chosen		Rejected		Both
Dataset	Method	CF cov.	SF cov.	CF cov.	SF cov.	CF cov.	SF cov.
harmless	PJ	0.74 $\pm$ .149	0.94 $\pm$ .039	0.40 $\pm$ .016	0.95 $\pm$ .064	0.26 $\pm$ .080	0.89 $\pm$ .061
	RP	0.64 $\pm$ .118	0.93 $\pm$ .051	0.38 $\pm$ .159	0.92 $\pm$ .082	0.22 $\pm$ .100	0.86 $\pm$ .072
	OURS	0.76 $\pm$ .086	0.97 $\pm$ .031	0.85 $\pm$ .109	0.91 $\pm$ .059	0.69 $\pm$ .111	0.90 $\pm$ .061
helpful	PJ	0.97 $\pm$ .035	0.87 $\pm$ .042	0.14 $\pm$ .097	0.99 $\pm$ .008	0.13 $\pm$ .088	0.87 $\pm$ .041
	RP	0.84 $\pm$ .066	0.81 $\pm$ .081	0.23 $\pm$ .095	0.99 $\pm$ .008	0.21 $\pm$ .088	0.81 $\pm$ .079
	OURS	0.81 $\pm$ .042	0.99 $\pm$ .020	0.98 $\pm$ .028	0.75 $\pm$ .081	0.80 $\pm$ .047	0.74 $\pm$ .067
hs2	RP	0.86 $\pm$ .035	0.54 $\pm$ .060	0.05 $\pm$ .038	0.99 $\pm$ .008	0.04 $\pm$ .035	0.54 $\pm$ .060
hs2	OURS	0.83 $\pm$ .069	0.86 $\pm$ .160	0.39 $\pm$ .145	0.95 $\pm$ .047	0.33 $\pm$ .144	0.84 $\pm$ .149