notesum.ai

Published at November 25

Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis

cs.CV

Released Date: November 25, 2024

Authors: Boming Miao¹, Chunxiao Li¹, Xiaoxiao Wang², Andi Zhang³, Rui Sun⁴, Zizhe Wang⁵, Yao Zhu⁵

Aff.: ¹Beijing Normal University; ²University of Chinese Academy of Sciences; ³The University of Manchester; ⁴The Chinese University of Hong Kong, Shenzhen; ⁵Tsinghua University

Arxiv: http://arxiv.org/abs/2411.16503v1

Activation Function	Mathematical Form	First Order Derivative Properties	Second Order Derivative Properties
ReLU	$g(x)=\max(0,x)$	$g^{\prime}(x)=1\cdot I(x>0)+0\cdot I(x<0)$	$g^{\prime\prime}(x)=0\text{ (except at }x=0)$
Sigmoid	$g(x)=\frac{1}{1+e^{-x}}$	$\left\|g^{\prime}(x)\right\|\leq\frac{1}{4}$	$\left\|g^{\prime\prime}(x)\right\|\leq\frac{1}{4}$
Tanh	$g(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$	$\left\|g^{\prime}(x)\right\|\leq 1$	$\left\|g^{\prime\prime}(x)\right\|\leq 2$
Leaky ReLU	$g(x)=x\cdot I(x>0)+\alpha x\cdot I(x\leq 0)$	$g^{\prime}(x)=1\cdot I(x>0)+\alpha\cdot I(x<0)$	$g^{\prime\prime}(x)=0\text{ (except at }x=0)$
Swish	$g(x)=x\cdot\frac{1}{1+e^{-x}}$	$\left\|g^{\prime}(x)\right\|<2$	$\left\|g^{\prime\prime}(x)\right\|<2$
ELU	$g(x)=x\cdot I(x>0)+\alpha(e^{x}-1)\cdot I(x\leq 0)$	$\left\|g^{\prime}(x)\right\|\leq\max(0,\alpha)$	$\left.g^{\prime\prime}(x)\leq\left\|\alpha\right\|\text{ (except at }x=0\right)$
GELU	$g(x)=x\cdot\Phi(x)$	$\left\|g^{\prime}(x)\right\|\leq 1$	$\left\|g^{\prime\prime}(x)\right\|\leq 0.5$
Mish	$g(x)=x\cdot\tanh\left(\ln\left(1+e^{x}\right)\right)$	$\left\|g^{\prime}(x)\right\|\leq 1.5$	$\left\|g^{\prime\prime}(x)\right\|\leq 2$
Softplus	$g(x)=\ln\left(1+e^{x}\right)$	$\left\|g^{\prime}(x)\right\|\leq 1$	$\left\|g^{\prime\prime}(x)\right\|\leq\frac{1}{4}$
Softmax	$g_{i}(x)=\frac{e^{x_{i}}}{\sum_{j}e^{x_{j}}}$	$\left\|g^{\prime}_{i}(x)\right\|\leq 1$	$\left\|g^{\prime\prime}_{i}(x)\right\|\leq 1$