notesum.ai

Published at November 25

DreamRunner: Fine-Grained Storytelling Video Generation with Retrieval-Augmented Motion Adaptation

cs.CV

cs.AI

cs.CL

Released Date: November 25, 2024

Authors: Zun Wang¹, Jialu Li¹, Han Lin¹, Jaehong Yoon¹, Mohit Bansal¹

Aff.: ¹UNC Chapel Hill

Arxiv: http://arxiv.org/abs/2411.16657v1

[Uncaptioned image]

Method	Image		Fine-Grained Text		Full Text		Transition
Method	CLIP	DINO	CLIP	ViCLIP	CLIP	ViCLIP	DINO
VideoDirectorGPT [33]	54.3	9.5	23.7	21.7	22.4	22.5	63.5
VLogger [74]	62.5	41.3	23.5	23.1	22.5	22.2	73.6
DreamRunner (Ours)	70.7 (+13.1%)	55.1 (+33.4%)	24.7 (+5.11%)	23.7 (+2.60%)	24.2 (+7.56%)	24.1 (+8.56%)	93.6 (+27.2%)