notesum.ai

Published at November 4

DiffuMask-Editor: A Novel Paradigm of Integration Between the Segmentation Diffusion Model and Image Editing to Improve Segmentation Ability

cs.CV

cs.AI

Released Date: November 4, 2024

Authors: Bo Gao¹, Fangxu Xing², Daniel Tang³

Aff.: ¹School of Intelligent Systems Engineering, Sun Yat-sen University; ²Department of Radiology, Harvard Medical School; ³Interdisciplinary Security and Trust Centre (SnT), University of Luxembourg

Arxiv: http://arxiv.org/abs/2411.01819v1

			Semantic Segmentation (IoU) for Selected Classes/%
Train Set	Number	Backbone	aeroplane	bird	boat	bus	car	cat	chair	cow	dog	horse	person	sheep	sofa	mIoU
Train with Pure Real Data
VOC	R: 10.6k (all)	R50	87.5	94.4	70.6	95.5	87.7	92.2	44.0	85.4	89.1	82.1	89.2	80.6	53.6	77.3
VOC	R: 10.6k (all)	Swin-B	97.0	93.7	71.5	91.7	89.6	96.5	57.5	95.9	96.8	94.4	92.5	95.1	65.6	84.3
	R: 5.0k	Swin-B	95.5	87.7	77.1	96.1	91.2	95.2	47.3	90.3	92.8	94.6	90.9	93.7	61.4	83.4
Train with Pure Synthetic Data
DiffuMask	S: 60.0k	R50	80.7	86.7	56.9	81.2	74.2	79.3	14.7	63.4	65.1	64.6	71.0	64.7	27.8	57.4
DiffuMask	S: 60.0k	Swin-B	90.8	92.9	67.4	88.3	82.9	92.5	27.2	92.2	86.0	89.0	76.5	92.2	49.8	70.6
ours	S: 60.0k	R50	82.1	88.3	58.3	83.1	79.0	81.6	17.7	65.4	67.3	65.9	75.0	66.0	29.6	62.5
ours	S: 60.0k	Swin-B	92.1	94.7	69.2	88.2	84.1	92.4	30.4	92.7	87.4	89.1	78.8	92.2	52.0	72.0
Finetune on Real Data
DiffuMask	S: 60.0k + R: 5.0k	R50	85.4	92.8	74.1	92.9	83.7	91.7	38.4	86.5	86.2	82.5	87.5	81.2	39.8	77.6
DiffuMask	S: 60.0k + R: 5.0k	Swin-B	95.6	94.4	72.3	96.9	92.9	96.6	51.5	96.7	95.5	96.1	91.5	96.4	70.2	84.9
ours	S: 60.0k + R: 5.0k	R50	86.5	94.1	73.7	94.3	85.7	91.9	41.3	87.2	89.6	83.0	88.0	80.6	46.8	78.9
ours	S: 60.0k + R: 5.0k	Swin-B	96.2	94.8	73.5	96.9	93.9	96.7	52.3	96.9	95.7	97.2	92.1	96.5	71.1	85.6