notesum.ai

Published at December 4

You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?

cs.AI

Released Date: December 4, 2024

Authors: Dominic Lohr¹, Hieke Keuning², Natalie Kiesler³

Aff.: ¹Friedrich-Alexander-Universität Erlangen-Nürnberg; ²Utrecht University; ³Nuremberg Tech

Arxiv: http://arxiv.org/pdf/2412.03516v1

Label	Name	Description
simple types
KR^∗	Knowledge of result	a simple indication of ‘correct’ or ‘incorrect’. Keuning et al. define ‘correct’ in the context of a programming task as one or more of the following elements: 1) it passes all tests, 2) it is equal to a model program, and/or 3) it satisfies one or more constraints.
KCR	Knowledge of correct result	A description or an indication of the correct response (e.g. providing the learner a model solution)
KP^∗	Knowledge of performance	Summative feedback on the achieved performance level regarding a set of tasks (e.g. ‘6 out of 10 tasks correct’ or ‘75% correct’.)
elaborate types
KTC^∗	Knowledge about task constraints	Focuses on the task itself. This feedback could be about certain constraints (not using library functions, or enforcing a specific approach, such as recursion), or more general hints on how to approach the task without taking the student’s current program into account.
KC^∗	Knowledge about concepts	Contains explanations on topics and concepts relevant to the exercise, or examples illustrating these concepts.
KM^∗	Knowledge about mistakes	Points out mistakes, by means of showing syntactic or semantic errors detected by a compiler (not exercise-specific), test cases that fail, or solution errors (logical or runtime). Other subcategories focus on style and performance-related issues. This content can be basic, containing a number (e.g. showing the number of failed tests), location or short identifier; or with a detailed description.
KH^∗	Knowledge on how to proceed	Hints on how to correct errors, next-step hints, and hints on how to improve the style of a semantically correct program.
KMC	Knowledge about meta-cognition	Feedback related to problem-solving strategies and the student’s awareness of their own performance.
^∗ used for data analysis