notesum.ai
Published at November 11What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance
cs.CL
cs.AI
Released Date: November 11, 2024
Authors: Hong Meng Yam1, Nathan J Paek
Aff.: 1Stanford University
| Model | Dataset | BLiMP Supplement | BLiMP Filtered | EWoK | Macroaverage |
|---|---|---|---|---|---|
| GPT2-18M | CHILDES | 52.8 | 58.2 | 50.5 | 53.83 |
| Gutenberg | 55.7 | 62.4 | 50.3 | 56.13 | |
| Mix | 55.9 | 63.7 | 49.7 | 56.43 | |
| TinyStories | 55.2 | 57.5 | 50.7 | 54.47 | |
| GPT2-44M | CHILDES | 55.3 | 57.8 | 51.2 | 54.77 |
| Gutenberg | 57.6 | 63.0 | 50.0 | 56.87 | |
| Mix | 58.2 | 65.6 | 50.4 | 58.07 | |
| TinyStories | 52.8 | 57.1 | 50.4 | 53.43 | |
| GPT2-97M | CHILDES | 49.7 | 60.5 | 49.6 | 53.27 |
| Gutenberg | 59.0 | 65.3 | 51.1 | 58.47 | |
| Mix | 58.0 | 66.0 | 50.6 | 58.20 | |
| TinyStories | 54.6 | 59.1 | 50.3 | 54.67 | |
| Llama-20M | CHILDES | 53.4 | 57.9 | 50.2 | 53.83 |
| Gutenberg | 57.4 | 60.0 | 50.6 | 56.00 | |
| Mix | 56.6 | 62.8 | 50.2 | 56.53 | |
| TinyStories | 46.7 | 51.1 | 49.8 | 49.20 | |
| GPT2-705M | Gutenberg | 59.9 | 66.8 | 50.6 | 59.10 |
| Mix | 56.7 | 66.1 | 50.6 | 57.80 | |
| Llama-360M | Gutenberg | 56.7 | 66.5 | 50.2 | 57.80 |
| Mix | 56.6 | 62.8 | 50.5 | 56.63 |