notesum.ai

Published at November 29

SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks

cs.CV

cs.LG

Released Date: November 29, 2024

Authors: Kim-Celine Kahl¹, Selen Erkan¹, Jeremias Traub¹, Carsten T. Lüth, Klaus Maier-Hein², Lena Maier-Hein³, Paul F. Jaeger¹

Aff.: ¹German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Germany; ²German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Germany; ³German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems (IMSY), Germany

Arxiv: http://arxiv.org/pdf/2411.19688v1

# Tokens	Learning Rate	Closed-Ended (Mistral Accuracy)	Open-Ended (Mistral Score)
40	3e-2	0.85 +/- 0.0	2.96 +/- 0.06
40	3e-1	0.85 +/- 0.01	2.95 +/- 0.05
60	3e-2	0.85 +/- 0.01	2.98 +/- 0.04
60	3e-1	0.85 +/- 0.0	2.97 +/- 0.04
80	3e-2	0.81 +/- 0.04	2.96 +/- 0.04
80	3e-1	0.84 +/- 0.01	2.99 +/- 0.02
100	3e-2	0.83 +/- 0.02	2.99 +/- 0.04
100	3e-1	0.85 +/- 0.0	3.0 +/- 0.03