notesum.ai

Published at December 5

T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts

cs.CV

cs.AI

Released Date: December 5, 2024

Authors: Ziwei Huang¹, Wanggui He, Quanyu Long, Yandi Wang, Haoyuan Li, Zhelun Yu, Fangxun Shu, Long Chen, Hao Jiang, Leilei Gan

Aff.: ¹Zhejiang University, China

Arxiv: http://arxiv.org/pdf/2412.04300v1

Refer to caption

Model	SKCM	SKCU		MKCC
Model	Concept	Concept	Task	Concept	Task	Composition
Text-to-image Generation
SD v1.5	40.5	52.9	53.9	37.6	13.4	15.1
SD XL	45.8	59.9	65.8	51.7	28.0	35.4
Pixart	26.4	46.2	55.3	35.8	19.8	24.3
Playground	45.6	66.1	62.5	53.8	35.4	44.8
Flux.1 dev	35.6	54.9	58.0	56.9	54.1	63.8
SD 3.5	46.2	64.6	71.2	68.9	59.2	75.5
DALLE-3	55.5	72.4	88.5	71.3	70.2	85.6
Visual-Knowledge Injection
SSR-Encoder	71.8 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,31.3}$	69.0 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,16.1}$	22.8 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,31.1}$	43.1 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,5.5}$	12.5 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,0.9}$	9.5 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,5.6}$
MS-Diffusion	84.8 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,39.0}$	80.4 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,30.5}$	50.8 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,15.0}$	65.5 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,13.8}$	32.9 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,14.9}$	31.0 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,4.4}$
Text-Knowledge Injection
Flux.1 dev*	41.2 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,5.6}$	60.3 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,5.4}$	59.8 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,1.8}$	64.2 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,7.3}$	56.9 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,2.8}$	72.6 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,8.8}$
SD 3.5*	49.7 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,3.5}$	66.7 ${\color[rgb]{0.3828125,0.66796875,0.40234375}\uparrow\,2.1}$	65.8 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,5.4}$	67.9 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,1.0}$	53.6 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,5.6}$	64.7 ${\color[rgb]{0.9765625,0.20703125,0.15625}\downarrow\,10.8}$