notesum.ai

Published at November 2

Rule Based Rewards for Language Model Safety

cs.AI

Released Date: November 2, 2024

Authors: Tong Mu¹, Alec Helyar¹, Johannes Heidecke¹, Joshua Achiam¹, Andrea Vallone¹, Ian Kivlichan¹, Molly Lin¹, Alex Beutel¹, John Schulman¹, Lilian Weng¹

Aff.: ¹OpenAI

Arxiv: http://arxiv.org/abs/2411.01111v1

	Human Evaluation			Internal Automated
	Not-Unsafe	Not-Overref	F1-Score*	Not-Unsafe	Not-Overref	F1-Score*
Helpful-PPO	93.64 $\pm$ 1.3%	98.13 $\pm$ 0.8%	95.8 $\pm$ 0.8%	86.98 $\pm$ 1.6%	97.84 $\pm$ 0.7%	92.1 $\pm$ 0.9%
Human-PPO	100.00 $\pm$ 0.0%	84.70 $\pm$ 2.2%	91.7 $\pm$ 1.3%	99.04 $\pm$ 0.4%	84.40 $\pm$ 1.8%	91.1 $\pm$ 1.1%
RBR-PPO	97.27 $\pm$ 0.9%	97.01 $\pm$ 1.0%	97.1 $\pm$ 0.7%	93.95 $\pm$ 1.1%	94.95 $\pm$ 1.0%	94.4 $\pm$ 0.7%