notesum.ai

Published at December 9

Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers

cs.CV

cs.AI

cs.LG

Released Date: December 9, 2024

Authors: Johanna Vielhaben¹, Dilyara Bareeva¹, Jim Berend¹, Wojciech Samek², Nils Strodthoff³

Aff.: ¹Fraunhofer Heinrich-Hertz-Institute Berlin; ²Fraunhofer Heinrich-Hertz-Institute Berlin, Technische Universitat Berlin, BIFOLD – Berlin Institute for the Foundations of Learning and Data; ³Carl von Ossietzky Universitat Oldenburg

Arxiv: http://arxiv.org/pdf/2412.06639v1

		FS	CLIP	DINO	MAE
SEQ	PCA	0.45	0.90	1.	0.60
	MCD	0.81	0.90	0.81	0.70
	KMeans	0.90	0.90	0.81	1.
	NLMCD	0.90	1.	0.90	1.
CLS	PCA	0.75	0.72	0.50	0.33
	MCD	0.33	0.16	0.16	0.16
	KMeans	0.83	0.91	0.50	0.66
	NLMCD	1.	1.	0.83	0.75