notesum.ai

Published at December 6

CompCap: Improving Multimodal Large Language Models with Composite Captions

cs.CV

cs.AI

cs.LG

Released Date: December 6, 2024

Authors: Xiaohui Chen¹, Satya Narayan Shukla², Mahmoud Azab², Aashu Singh², Qifan Wang², David Yang², ShengYun Peng³, Hanchao Yu², Shen Yan², Xuewen Zhang², Baosheng He²

Aff.: ¹Meta, Tufts University; ²Meta; ³Meta, Georgia Tech

Arxiv: http://arxiv.org/pdf/2412.05243v1

Category	Metadata	Image Simulator(s)	Caption Composition	#Samples	Avg. Char.
\faThLarge Collage	Image-Caption & Layout	OpenCV (Bradski, 2000) / PIL (Clark, 2015)	LGC^∗	50K	913
\faNewspaper Image-Text	Image-Caption & Text & Layout	OpenCV / PIL / Augraphy (Groleau et al., 2023)	Text + LGC / Text	37K	221
\faChartBar Chart	(Geo) Tabular data	Plotly (Inc., 2015)	LGC	22K	1468
\faSitemap Diagram	Mermaid diagram code	Mermaid (Knsv, 2024) & Selenium	LGC	3K	2044
\faCode Code	Code snippet	Carbon (Carbon, 2024) & Selenium	Code snippet + LGC	2K	1106
\faTable Table	Tabular data	Matplotlib (Hunter, 2007)	Markdown table + LGC	4K	928