notesum.ai

Published at November 22

ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation

cs.CV

cs.AI

cs.CL

Released Date: November 22, 2024

Authors: Xiaoman Zhang¹, Hong-Yu Zhou¹, Xiaoli Yang¹, Oishi Banerjee¹, Julián N. Acosta¹, Josh Miller², Ouwen Huang², Pranav Rajpurkar¹

Aff.: ¹Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA; ²Gradient Health, Durham, NC, USA

Arxiv: http://arxiv.org/abs/2411.15122v1

Model	1/RadCliQ-v1 $\uparrow$	BLEU-2 $\uparrow$	BertScore $\uparrow$	SembScore $\uparrow$	RadGraph $\uparrow$	RaTEScore $\uparrow$	GREEN $\uparrow$	1/FineRadScore $\uparrow$
Findings + Impression
MedVersa	0.92 ± 0.02	0.19 ± 0.00	0.43 ± 0.01	0.32 ± 0.01	0.27 ± 0.01	0.55 ± 0.01	0.42 ± 0.01	0.36 ± 0.03
CheXpertPlus_CheX_MIMIC	0.83 ± 0.02	0.17 ± 0.00	0.36 ± 0.00	0.39 ± 0.01	0.20 ± 0.00	0.52 ± 0.00	0.37 ± 0.01	0.36 ± 0.03
CheXpertPlus_MIMIC	0.80 ± 0.02	0.17 ± 0.00	0.35 ± 0.00	0.38 ± 0.01	0.19 ± 0.00	0.51 ± 0.00	0.38 ± 0.01	0.36 ± 0.03
CheXpertPlus_CheX	0.71 ± 0.01	0.13 ± 0.00	0.30 ± 0.00	0.34 ± 0.01	0.17 ± 0.00	0.51 ± 0.00	0.30 ± 0.01	0.35 ± 0.03
RadFM	0.62 ± 0.02	0.08 ± 0.00	0.28 ± 0.00	0.24 ± 0.01	0.11 ± 0.00	0.45 ± 0.00	0.21 ± 0.01	0.35 ± 0.03
GPT4V	0.55 ± 0.01	0.07 ± 0.00	0.20 ± 0.00	0.19 ± 0.01	0.09 ± 0.00	0.43 ± 0.00	0.13 ± 0.01	0.33 ± 0.03
Findings
MedVersa	1.10 ± 0.02	0.21 ± 0.00	0.45 ± 0.01	0.47 ± 0.01	0.27 ± 0.01	0.55 ± 0.01	0.37 ± 0.01	0.36 ± 0.03
CheXpertPlus_CheX_MIMIC	0.81 ± 0.02	0.14 ± 0.00	0.37 ± 0.00	0.38 ± 0.01	0.18 ± 0.01	0.49 ± 0.01	0.30 ± 0.01	0.36 ± 0.03
RaDialog	0.80 ± 0.02	0.13 ± 0.00	0.36 ± 0.00	0.39 ± 0.01	0.17 ± 0.00	0.48 ± 0.00	0.27 ± 0.01	0.36 ± 0.03
CheXpertPlus_MIMIC	0.79 ± 0.02	0.14 ± 0.00	0.36 ± 0.00	0.38 ± 0.01	0.17 ± 0.00	0.48 ± 0.01	0.31 ± 0.01	0.36 ± 0.03
RGRG	0.76 ± 0.02	0.13 ± 0.00	0.35 ± 0.00	0.34 ± 0.01	0.17 ± 0.00	0.49 ± 0.00	0.27 ± 0.01	0.35 ± 0.03
CheXagent	0.74 ± 0.02	0.11 ± 0.00	0.35 ± 0.01	0.35 ± 0.01	0.15 ± 0.00	0.47 ± 0.01	0.26 ± 0.01	0.35 ± 0.03
Cvt2distilgpt2_MIMIC	0.72 ± 0.02	0.13 ± 0.00	0.33 ± 0.01	0.33 ± 0.01	0.15 ± 0.01	0.43 ± 0.01	0.27 ± 0.01	0.36 ± 0.03
CheXpertPlus_CheX	0.70 ± 0.01	0.08 ± 0.00	0.31 ± 0.00	0.33 ± 0.01	0.14 ± 0.00	0.47 ± 0.00	0.23 ± 0.01	0.35 ± 0.03
MAIRA-2	0.69 ± 0.02	0.09 ± 0.00	0.31 ± 0.00	0.34 ± 0.01	0.13 ± 0.00	0.52 ± 0.00	0.22 ± 0.01	0.36 ± 0.03
VLCI_MIMIC	0.68 ± 0.02	0.14 ± 0.00	0.30 ± 0.00	0.30 ± 0.01	0.14 ± 0.00	0.45 ± 0.01	0.26 ± 0.01	0.36 ± 0.03
RadFM	0.65 ± 0.02	0.09 ± 0.00	0.31 ± 0.00	0.26 ± 0.01	0.11 ± 0.00	0.45 ± 0.01	0.18 ± 0.01	0.35 ± 0.03
Cvt2distilgpt2_IU	0.61 ± 0.02	0.06 ± 0.00	0.30 ± 0.00	0.19 ± 0.01	0.10 ± 0.00	0.45 ± 0.01	0.16 ± 0.01	0.35 ± 0.03
VLCI_IU	0.60 ± 0.02	0.07 ± 0.00	0.26 ± 0.00	0.21 ± 0.01	0.11 ± 0.00	0.45 ± 0.01	0.21 ± 0.01	0.35 ± 0.03
GPT4V	0.56 ± 0.01	0.07 ± 0.00	0.21 ± 0.00	0.21 ± 0.01	0.08 ± 0.00	0.42 ± 0.00	0.16 ± 0.01	0.34 ± 0.03
BiomedGPT_IU	0.54 ± 0.01	0.02 ± 0.00	0.19 ± 0.00	0.22 ± 0.01	0.06 ± 0.00	0.36 ± 0.00	0.12 ± 0.01	0.34 ± 0.03
LLM-CXR	0.52 ± 0.01	0.04 ± 0.00	0.18 ± 0.00	0.16 ± 0.01	0.05 ± 0.00	0.34 ± 0.00	0.04 ± 0.00	0.31 ± 0.03