notesum.ai

Published at November 20

Are Large Language Models Memorizing Bug Benchmarks?

cs.SE

cs.AI

Released Date: November 20, 2024

Authors: Daniel Ramos¹, Claudia Mamede¹, Kush Jain¹, Paulo Canelas¹, Catarina Gamboa¹, Claire Le Goues¹

Aff.: ¹Carnegie Mellon University

Arxiv: http://arxiv.org/abs/2411.13323v1

Refer to caption

Model	Tokens (T)	Cutoff Year
Codegen Multi (6B) [18]	0.5	2022
CodeLlama (7B) [19]	2.5	2023
LlaMa 3.1 (8B / 70B) [20]	15.0	2024
StarCoder 2 (7B) [21]	3.5	2024
Gemma 2 (2B / 27B) [22]	2.0 / 13.0	2024
CodeGemma (7B) [23]	6.5	2024
Mistral (7B) [24]	-	-