notesum.ai
Published at November 29Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability
cs.CL
cs.AI
Released Date: November 29, 2024
Authors: Yujin Han1, Lei Xu2, Sirui Chen3, Difan Zou1, Chaochao Lu2
Aff.: 1The University of Hong Kong; 2Shanghai Artificial Intelligence Laboratory; 3Tongji University

| Dataset | Term | Origin & Intervention Data |
|---|---|---|
| 2-digit Multiplication (Mask) | Origin | What is 50 times 20? A: 1000 |
| TE with | What is <Mask> times 20? A: None | |
| AICE with | What <Mask> 50 times 20? A: 1000 | |
| CommonsenseQA (Rephrase) | Origin | Reading newspaper one of many ways to practice your what? A: literacy |
| TE with | Using newspapers to wrap gifts is one way to practice your what? A: money | |
| AICE with | Using newspapers to read articles is one way to practice your what? A: literacy |