notesum.ai

Published at October 23

Differentially Private Learning Needs Better Model Initialization and Self-Distillation

cs.CE

Released Date: October 23, 2024

Authors: Ivoline C. Ngong¹, Joseph P. Near¹, Niloofar Mireshghallah²

Aff.: ¹University of Vermont; ²University of Washington

Arxiv: https://arxiv.org/abs/2410.17566v1

Refer to caption

Dataset	Phase 1 generation model	Phase 1 generated examples	Phase 2 fine-tuned model	Phase 3 generated examples	Phase 3 fine-tuned model
MRPC	GPT2	73706	T5-large	70143	T5-large
XSUM	GPT2	151779	T5-large	202707	T5-large
Pubmed	BioGPT	45484	T5-large	234048	T5-large