notesum.ai
Published at November 27Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
cs.CY
cs.HC
Released Date: November 27, 2024
Authors: Neil C. C. Brown1, Pierre Weill-Tessier1, Juho Leinonen2, Paul Denny3, Michael Kölling1
Aff.: 1King's College London, UK; 2Aalto University, Finland; 3University of Auckland, New Zealand

| Attribute | Type | Method of extraction |
|---|---|---|
| Word count | Integer | Count number of words, including words in code, but ignoring punctuation, symbols and emoji. |
| Reading level | Real | Flesch-Kincaid Reading Grade Level (Kincaid, 1975). |
| Sentiment | Real | VADER Sentiment Analyzer (Hutto and Gilbert, 2014), compound polarity score. |
| Model | Category | The name of the AI model or the term “Human” to indicate how the hint was generated. |
| Telling | Boolean | Feedback-literacy-inspired category. See subsection 4.7. |
| Guiding | Boolean | Feedback-literacy-inspired category. See subsection 4.7. |
| Developing understanding | Boolean | Feedback-literacy-inspired category. See subsection 4.7. |
| Opening up a new perspective | Boolean | Feedback-literacy-inspired category. See subsection 4.7. |
| Partially incorrect | Boolean | Flag indicating whether a hint was partially incorrect. |