notesum.ai

Published at November 12

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

cs.CV

cs.AI

cs.CL

Released Date: November 12, 2024

Authors: Yiyang Ma¹, Xingchao Liu², Xiaokang Chen², Wen Liu², Chengyue Wu, Zhiyu Wu², Zizheng Pan², Zhenda Xie², Haowei Zhang², Xingkai yu², Liang Zhao², Yisong Wang³, Jiaying Liu⁴, Chong Ruan²

Aff.: ¹DeepSeek-AI, Peking University; ²DeepSeek-AI; ³DeepSeek-AI, Tsinghua University; ⁴Peking University

Arxiv: http://arxiv.org/abs/2411.07975v1

Exp. ID	Model Setting				Train. Iter.	Evaluation Benchmarks
Exp. ID	REPA	Und. Modules	Gen. Modules	Type	Train. Iter.	POPE $\uparrow$	VQAv2 ${}_{val}\uparrow$	GQA $\uparrow$	FID $\downarrow$	CLIP $\uparrow$
A	$\times$	SigLIP	VAE^†+ConvNeXt	Unified	50,000	82.40	69.62	54.43	19.84	24.94
B	$\checkmark$	Shared VAE^†+ConvNeXt		Unified	50,000	78.13	53.94	44.04	18.05	26.38
C	$\checkmark$	VAE+ConvNeXt	VAE^†+ConvNeXt	Unified	50,000	75.30	55.41	44.44	17.53	26.32
D	$\checkmark$	SigLIP	-	Und. Only	13,000	85.03	69.10	54.23	-	-
E	$\checkmark$	-	VAE^†+ConvNeXt	Gen. Only	37,000	-	-	-	16.69	26.89
F	$\checkmark$	SigLIP	VAE^†+ConvNeXt	Unified	50,000	84.73	69.20	54.83	17.61	26.40