notesum.ai
Published at December 9I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
cs.LG
cs.CL
Released Date: December 9, 2024
Authors: Roi Cohen1, Konstantin Dobler1, Eden Biran2, Gerard de Melo1
Aff.: 1HPI / University of Potsdam; 2Tel Aviv University

| LAMA Google-RE | LAMA T-Rex | LAMA SQuAD | TriviaQA | PopQA | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | |
| Mistral-7B-v0.1 | |||||||||||||||
| Mistral-7B-v0.1 + The Pile | |||||||||||||||
| Mistral-7B-v0.1 + Confidence Threshold | |||||||||||||||
| Mistral-7B-v0.1 + P(True) | |||||||||||||||
| Mistral-7B-v0.1 + Semantic Entropy | |||||||||||||||
| Mistral-7B-v0.1 + IDK-tuning on The Pile | |||||||||||||||