notesum.ai
Published at October 21Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs
cs.CL
cs.AI
Released Date: October 21, 2024
Authors: Yanzhu Guo1, Simone Conia2, Zelin Zhou3, Min Li3, Saloni Potdar3, Henry Xiao3
Aff.: 1Inria Paris; 2Sapienza University of Rome; 3Apple

| Human | Qwen1.5 | Qwen2 | Mistral-v0.3 | Mistral-Nemo | Llama-3 | Llama-3.1 | ||
|---|---|---|---|---|---|---|---|---|
| Model Size (# parameters) | — | 7B | 7B | 7B | 12B | 8B | 8B | |
| Lexical Divergence | 23.07 | 30.36 | 25.31 | 23.30 | 25.12 | 29.00 | 26.79 | |
| English | Syntactic Divergence | 3.53 | 22.19 | 13.67 | 13.56 | 14.77 | 17.72 | 16.80 |
| Lexical Divergence | 25.91 | 41.00 | 37.08 | 39.02 | 34.78 | 36.88 | 33.29 | |
| Chinese | Syntactic Divergence | 2.93 | 23.33 | 20.66 | 17.29 | 12.84 | 15.45 | 10.32 |
| Lexical Divergence | 24.25 | 38.35 | 31.18 | 28.73 | 31.34 | 32.22 | 31.52 | |
| French | Syntactic Divergence | 3.22 | 24.21 | 12.10 | 12.72 | 14.72 | 17.88 | 11.27 |