notesum.ai

Published at October 24

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages

cs.LG

cs.AI

cs.NE

Released Date: October 24, 2024

Authors: Artur Kiulian¹, Anton Polishko¹, Mykola Khandoga¹, Yevhen Kostiuk², Guillermo Gabrielli¹, Łukasz Gagała, Fadi Zaraket³, Qusai Abu Obaida⁴, Hrishikesh Garud⁵, Wendy Wing Yee Mak⁶, Dmytro Chaplynskyi¹, Selma Belhadj Amor⁶, Grigol Peradze⁶

Aff.: ¹OpenBabylon; ²OpenBabylon, ARG-Tech, University of Dundee, UK; ³Doha Institute for Graduate Studies; ⁴Arab Center for Research and Policy Studies; ⁵Google; ⁶PolyAgent

Arxiv: https://arxiv.org/abs/2410.18836v1

Model	GCS $\uparrow$	NEWR $\downarrow$	CSWR $\downarrow$
Ukrainian
Vanilla	0.264	0.089	0.515
Tuned	0.388	0.032	0.002
Ours	0.503	0.030	0.001
Arabic
Vanilla	0.040	0.863	0.450
Tuned	0.238	0.079	0.004
Ours	0.548	0.050	0.002