notesum.ai
Published at November 21Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models
cs.CL
Released Date: November 21, 2024
Authors: Iacopo Ghinassi1, Leonardo Catalano2, Tommaso Colella3
Aff.: 1Queen Mary University of London; 2Independent Researcher; 3University of Pisa

| Model | Consistency | Coherence | Fluency | Relevance | Average |
|---|---|---|---|---|---|
| Qwen 0.5B | 4.52* | 4.33* | 4.41* | 4.06* | 4.33* |
| Qwen 1.8B | 4.89 | 4.83 | 4.88 | 4.79 | 4.85 |
| Qwen 4B | 4.75* | 4.84 | 4.91 | 4.56* | 4.77 |
| Qwen 7B | 4.84 | 4.94 | 4.9 | 4.74 | 4.86 |
| Gemma 2B | 4.86 | 4.86 | 4.96 | 4.71 | 4.85 |
| Gemma 7B | 4.85 | 4.94 | 4.99 | 4.81 | 4.9 |
| Phi 3 | 4.84 | 4.92 | 4.94 | 4.74 | 4.86 |
| Llama 3 | 4.82 | 4.84 | 4.91 | 4.74 | 4.83 |
| Mistral | 4.78* | 4.84 | 4.95 | 4.6 | 4.79 |
| ChatGPT | 4.94 | 4.96 | 4.98 | 4.79 | 4.91 |
| GPT4 | 4.83 | 4.89 | 4.96 | 4.81 | 4.89 |
| MPNet | 4.44* | 3.03* | 3.45* | 4.15* | 3.77* |