notesum.ai

Published at December 10

Granite Guardian

cs.CL

Released Date: December 10, 2024

Authors: Inkit Padhi¹, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, Prasanna Sattigeri

Aff.: ¹IBM Research

Arxiv: http://arxiv.org/pdf/2412.07724v1

model	AUC	AUPRC	F1	Recall	Precision
Llama-Guard-7B	0.824	0.803	0.659	0.533	0.861
Llama-Guard-2-8B	0.841	0.822	0.723	0.627	0.852
Llama-Guard-3-1B	0.796	0.775	0.656	0.575	0.765
Llama-Guard-3-8B	0.826	0.819	0.710	0.607	0.857
ShieldGemma-2B	0.748	0.704	0.421	0.277	0.883
ShieldGemma-9B	0.753	0.707	0.404	0.262	0.886
ShieldGemma-27B	0.772	0.718	0.438	0.295	0.849
Granite-Guardian-3.0-2B	0.782	0.746	0.674	0.747	0.614
Granite-Guardian-3.0-8B	0.871	0.846	0.758	0.735	0.781