notesum.ai

Published at November 22

Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains

cs.CL

Released Date: November 22, 2024

Authors: Yurii Paniv¹, Artur Kiulian², Dmytro Chaplynskyi³, Mykola Khandoga², Anton Polishko², Tetiana Bas⁴, Guillermo Gabrielli²

Aff.: ¹Ukrainian Catholic University; ²OpenBabylon; ³lang-uk initiative; ⁴Minerva University

Arxiv: http://arxiv.org/abs/2411.14647v1

Model	ZNO val	ZNO test
Gemini Pro	0.680	0.675
Claude 3.5 Sonnet	0.651	0.643
Qwen2-VL-72B	0.496	0.512
GPT-4o	0.416	0.470
Qwen2-VL-7B	0.247	0.264
Baseline	0.224	0.219
Llama-3.2-11B	0.116	0.095
LLaVa-v1.6-mistral-7b	0.071	0.067
Paligemma-3b	0.049	0.058
Pixtral-12b	0.000	0.000