notesum.ai
Published at November 4Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast
cs.SE
cs.AI
cs.LO
cs.PL
Released Date: November 4, 2024
Authors: Marilyn Rego1, Wen Fan1, Xin Hu, Sanya Dod1, Zhaorui Ni1, Danning Xie, Jenna DiVincenzo1, Lin Tan
Aff.: 1Purdue University West Lafayette, USA
| Error Types | Error Name |
|
|
|
|
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Compile Error | Syntax Error | 54.14% | 71.42% | 85.71% | 57.14% | ||||||||
| Semantic Error | 57.14% | 57.14% | 57.14% | 28.57% | |||||||||
| Verification Error | Missing Open & Close Statements | 42.85% | 28.57% | 28.57% | 57.14% | ||||||||
| Incorrect Contracts | 57.14% | 28.57% | 28.57% | 42.85% | |||||||||
| Incorrect Predicate Declaration | 14.28% | 28.57% | 42.85% | 42.85% | |||||||||
| Heap Related Error | 42.85% | 28.57% | 42.85% | 28.57% |