notesum.ai

Published at November 10

PLM-Based Discrete Diffusion Language Models with Entropy-Adaptive Gibbs Sampling

cs.CL

cs.AI

Released Date: November 10, 2024

Authors: Hyukhun Koh¹, Minha Jhang¹, Dohyung Kim², Sangmook Lee², Kyomin Jung¹

Aff.: ¹IPAI, Seoul National University; ²Dept. of ECE, Seoul National University

Arxiv: http://arxiv.org/abs/2411.06438v1

Refer to caption

	QG
		Text Quality			Diversity
Model	Step	PPL $\downarrow$	MAUVE $\uparrow$	SOME $\uparrow$	VS(ngram) $\uparrow$	VS(emb) $\uparrow$	self-bleu $\downarrow$	distinct-1 $\uparrow$	distinct-2 $\uparrow$
GPT-2	1	124.8	0.141	0.759	4.564	3.130	0.176	0.210	0.629
DiffuSeq	20	395.0	0.149	0.730	1.555	1.274	0.901	0.170	0.564
DINOISER	2000	155.9	0.159	0.776	1.396	1.121	0.944	0.166	0.553
DiffusionBert	2000	513.6	0.150	0.712	3.040	2.209	0.566	0.392	0.759
Diffusion-EAGS	5	80.7	0.121	0.782	4.646	3.538	0.152	0.403	0.798
	QQP
Model	Step	PPL $\downarrow$	MAUVE $\uparrow$	SOME $\uparrow$	VS(ngram) $\uparrow$	VS(emb) $\uparrow$	self-bleu $\downarrow$	distinct-1 $\uparrow$	distinct-2 $\uparrow$
GPT-2	1	66.270	0.112	0.754	3.886	2.566	0.423	0.344	0.787
DiffuSeq	2000	124.247	0.00674	0.709	1.927	1.242	0.813	0.226	0.543
DINOISER	20	79.742	0.0042	0.821	1.421	1.126	0.935	0.264	0.542
DiffusionBert	2000	500.959	0.0709	0.618	4.489	2.836	0.196	0.321	0.761
Diffusion-EAGS	5	48.106	0.683	0.824	4.006	2.390	0.338	0.421	0.832