notesum.ai

Published at December 3

Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

cs.AI

cs.LG

Released Date: December 3, 2024

Authors: Kai Sun¹, Siyan Xue, Fuchun Sun, Haoran Sun, Yu Luo, Ling Wang, Siyuan Wang, Na Guo, Lei Liu, Tian Zhao, Xinzhou Wang, Lei Yang, Shuo Jin, Jun Yan, Jiahong Dong

Aff.: ¹School of Biomedical Engineering, Tsinghua University

Arxiv: http://arxiv.org/pdf/2412.02621v1

Dataset Name	Type	Modality	Purpose	Size	Key Features
MC-CXR & SZ-CXR[68]	Chest X-rays	X-ray	Disease detection and classification	138(MC)/662(SZ) images	Focus on diagnostic challenges in chest X-ray interpretation
CBIS-DDSM-CALC & MASS[69]	Mammography images	Mammography	Breast cancer detection research	1K images	Detailed annotations for calcifications and masses
MMR Datasets[70]	Various medical image types	Multimodal	Multimodal medical research	166K images	Multimodal dataset covering multiple specialties
MMR-Colon Pathology[70]	Pathology images	Pathology	Disease diagnosis and treatment planning	107K images	Focuses on colon pathology
MMR-Dermatoscopy[70]	Dermatoscopy images	Dermatoscopy	Skin disease diagnosis	10K images	Contains dermatoscopy images for skin disease diagnosis
MMR-Retinal OCT[70]	Retinal OCT images	Retinal OCT	Eye disease diagnosis	1K images	Retinal OCT scans for diagnosing retinal conditions
MMR-Chest X-ray[70]	Chest X-ray images	X-ray	Lung disease diagnosis	5K images	Chest X-rays with focus on lung-related diseases
MMR-Breast Ultrasound[70]	Ultrasound images	Ultrasound	Breast cancer detection	780 images	Ultrasound images for breast cancer diagnosis
MMR-Blood Cell Microscope[70]	Microscope images	Microscopy	Blood cell analysis	17K images	Blood cell microscope images for various hematological conditions
MMR-Coronal Abdominal CT[70]	Abdominal CT images	CT Imaging	Abdominal disease diagnosis	23K images	Coronal Abdominal CT scans for abdominal pathology
CLIP-Driven Universal Model[71]	CT images	CT Imaging	Tumor segmentation and detection	3K training CT scans, 6K external scans	Model developed from 14 datasets targeting 25 organs and 6 tumor types
AbdomenAtlas-8K[72]	Annotated CT Volumes	CT Imaging	Organ segmentation	8K CT volumes	Focus on 8 key abdominal organs including liver, spleen, kidneys
AbdomenAtlas 1.1	Annotated CT Volumes	CT Imaging	Multi-organ and tumor segmentation	9K CT volumes	Detailed voxel-wise annotations of 25 anatomical structures and 7 tumor types
MedSAM[73]	Medical image mask pairs	Multimodal (10 imaging modes)	Segmentation and cancer detection	1.5M image-mask pairs	Covers 10 imaging modalities and 30+ cancer types
SAM-Med3D[74]	3D medical image segmentation	CT, MRI	3D image segmentation for organs and lesions	21K images, 131K masks	Covers 27 modalities and 240+ target categories
EyeFound[75]	Ophthalmology images	11 clinical imaging modalities	Eye disease research	3M images	Collected from 227 hospitals, includes CFP, FFA, ICGA, OCT, ultrasound, etc.
RETFound[76]	Retinal images (CFP, OCT)	Fundus Imaging, OCT	Eye disease detection	904K CFP, 736k OCT	Focus on fundus and OCT images for eye condition diagnosis
Swin UNETR[77]	CT images	CT Imaging	Pretraining for 3D segmentation	5K CT volumes	Includes chest, abdomen, and head/neck volumes
Disruptive Autoencoders[78]	CT and MRI images	CT, MRI (T1, T2, FLAIR, T1ce)	Pretraining for 3D segmentation and autoencoders	10K 3D volumes	Combines 5 modalities for diverse segmentation tasks