notesum.ai

Published at October 21

A Comprehensive Survey of Datasets, Theories, Variants, and Applications in Direct Preference Optimization

cs.SD

cs.AI

eess.AS

Released Date: October 21, 2024

Authors: Wenyi Xiao¹, Zechuan Wang¹, Leilei Gan¹, Shuai Zhao², Wanggui He³, Luu Anh Tuan², Long Chen³, Hao Jiang³, Zhou Zhao¹, Fei Wu¹

Aff.: ¹Zhejiang University, China; ²Nanyang Technological University, Singapore; ³Alibaba Group, China

Arxiv: https://arxiv.org/abs/2410.15595v1

Method	Objective
RQ0: why DPO?
PPO (Ouyang et al., 2022a)	$r_{\phi}(x,y)-\beta\mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta}(y\mid x)\\|\pi_{% \text{ref }}(y\mid x)\right]$
DPO (Rafailov et al., 2023)	$-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}(y_{w}\|% x)}-\beta\log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}\right)$
RQ1: Generalization Ability of RM?
IPO (Azar et al., 2023)	$\left(\beta\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}(y_{w}\|x)}-\beta% \log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}-\frac{1}{2}\right)% ^{2}$
RQ2: Rewarding Signals?
RRHF (Yuan et al., 2023)	$\sum_{r_{i}<r_{j}}\max(0,p_{i}-p_{j})-\sum_{t}\log P_{\pi}(y_{i^{\prime},t}\|x,% y_{i^{\prime},<t})$
	$\text{where}\,\,p_{i}=\frac{\sum_{t}\log P_{\pi}(y_{i,t}\|x,y_{i,<t})}{\\|y_{i}% \\|}\text{,}\,\,i^{\prime}=\arg\max_{i}{r_{i}}$
SPIN (Chen et al., 2024a)	$\ell\left(\lambda\log\frac{p_{\theta}(y\|x)}{p_{\theta_{t}}(y\|x)}-\lambda\log% \frac{p_{\theta}(y^{\prime}\|x)}{p_{\theta_{t}}(y^{\prime}\|x)}\right)$
	$\text{where}\,\,\ell(t):=\log(1+\exp(-t))\text{,}\,\,y\sim p_{\text{data}}(% \cdot\|x)\text{,}\,\,y^{\prime}\sim p_{\theta_{t}}(\cdot\|x)$
Step-DPO (Lai et al., 2024)	$-\log\sigma(\beta\log\frac{\pi_{\theta}(s_{w}\|x,s_{1\sim k-1})}{\pi_{ref}(s_{w% }\|x,s_{1\sim k-1})}-\beta\log\frac{\pi_{\theta}(s_{l}\|x,s_{1\sim k-1})}{\pi_{% ref}(s_{l}\|x,s_{1\sim k-1})})$
T-DPO (Zeng et al., 2024)	$-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}(y_{w}\|% x)}-\beta\log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}\right)$
	$-\beta\left(D_{\text{SeqKL}}(x,y_{l};\pi_{\text{ref}}\|\|\pi_{\theta})-D_{\text{% SeqKL}}(x,y_{w};\pi_{\text{ref}}\|\|\pi_{\theta})\right))$
KTO (Ethayarajh et al., 2024)	$-\lambda_{w}\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}% (y_{w}\|x)}-z_{\text{ref}}\right)-\lambda_{l}\sigma\left(z_{\text{ref}}-\beta% \log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}\right),\,$
	$\text{where}\,\,z_{\text{ref}}=\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\beta% \text{KL}\left(\pi_{\theta}(y\|x)\|\|\pi_{\text{ref}}(y\|x)\right)\right]$
RQ3: $\beta$ Coeficient and Reference Model?
$\beta$ -DPO (Wu et al., 2024a)	$-\log\sigma\left(\beta_{i}\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}(y_% {w}\|x)}-\beta_{i}\log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}\right)$
	$\text{where}\,\,\beta_{i}=[1+\alpha(M_{i}-M_{0})]\beta_{0}\text{,}$
	$\text{and}\,\,M=\beta_{0}\log\left(\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text% {ref}}(y_{w}\mid x)}\right)-\beta_{0}\log\left(\frac{\pi_{\theta}(y_{l}\mid x)% }{\pi_{\text{ref}}(y_{l}\mid x)}\right)$
CPO (Xu et al., 2024b)	$-\log p_{\theta}(y_{w}\|x)-\log\sigma\left(\beta\log\pi_{\theta}(y_{w}\|x)-\beta% \log\pi_{\theta}(y_{l}\|x)\right)$
ORPO (Hong et al., 2024)	$-\log p_{\theta}(y_{w}\|x)-\lambda\log\sigma\left(\log\frac{p_{\theta}(y_{w}\|x)% }{1-p_{\theta}(y_{w}\|x)}-\log\frac{p_{\theta}(y_{l}\|x)}{1-p_{\theta}(y_{l}\|x)}% \right),\,$
	$\text{where}\,\,p_{\theta}(y\|x)=\exp\left(\frac{1}{\|y\|}\log\pi_{\theta}(y\|x)\right)$
RQ4: Training Stratege of DPO?
OPtune (Chen et al., 2024b)	$-R(x,y_{w},y_{l})\cdot\log\sigma\left(\beta_{1}\log\frac{\pi_{t+1}(y_{w}\|x)}{% \pi_{t}(y_{w}\|x)}-\beta_{1}\log\frac{\pi_{t+1}(y_{l}\|x)}{\pi_{t}(y_{l}\|x)}\right)$
	$\text{where}\,\,R(x,y_{w},y_{l})=\sigma\left[\beta_{2}(r(x,y_{w})-r(x,y_{l}))\right]$
IRPO (Pang et al., 2024)	$-\log\sigma\left(\beta\log\frac{\pi_{\theta}(c_{w},y_{w}\|x_{i})}{\pi_{t}(c_{w}% ,y_{w}\|x_{i})}-\beta\log\frac{\pi_{\theta}(c_{l},y_{l}\|x_{i})}{\pi_{t}(c_{l},y% _{l}\|x_{i})}\right)-\alpha\frac{\log\pi_{\theta}(c_{w},y_{w}\|x_{i})}{\|c_{w}\|+\|% y_{w}\|}$
RQ5: Reward Hacking?
R-DPO (Park et al., 2024)	$-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}(y_{w}\|% x)}-\beta\log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}-\left(% \alpha\|y_{w}\|-\alpha\|y_{l}\|\right)\right)$
LD-DPO (Liu et al., 2024d)	$-\log\sigma\left(\beta_{i}\log\frac{\pi_{\theta}(y_{w}\|x)}{\pi_{\text{ref}}(y_% {w}\|x)}-\beta_{i}\log\frac{\pi_{\theta}(y_{l}\|x)}{\pi_{\text{ref}}(y_{l}\|x)}\right)$
	$\text{where}\,\,{\pi}_{\theta}(y\mid x)=\prod_{i=1}^{l}p^{\alpha}(y_{i}\mid x,% y_{<i})\prod_{i=1}^{l_{p}}p^{1-\alpha}(y_{i}\mid x,y_{<i})$
SimPO (Meng et al., 2024)	$-\log\sigma\left(\frac{\beta}{\|y_{w}\|}\log\pi_{\theta}(y_{w}\|x)-\frac{\beta}{\|% y_{l}\|}\log\pi_{\theta}(y_{l}\|x)-\gamma\right)$
RQ6: Alignment Tax?
SPO (Lou et al., 2024a)	$-\log\sigma\left(\xi_{2}\phi_{2}(x,y_{w},y_{l})-\xi_{1}\phi_{1}(x,y_{w},y_{l})\right)$
	$\text{where}\,\,\forall i\in\{1,2\}\text{,}\,\,\phi_{i}(x,y_{w},y_{l})=\log% \frac{\pi_{i}(y_{l}\|x)}{\pi_{i-1}(y_{w}\|x)}-\log\frac{\pi_{i}(y_{w}\|x)}{\pi_{i% -1}(y_{l}\|x)}$