Amazing Gauss!

$$f(x;0,1)=\frac{1}{\sqrt{2\pi}}e^{-\frac{x^2}{2}}$$

$$X\sim \mathcal{N}(2, 1)\quad \text{vs}\quad X\sim \mathcal{N}(-2, 1)$$

$$\mathcal{N}(0,1),\quad\mathcal{N}(0,0.1),\quad\mathcal{N}(0,4),\quad\mathcal{N}(0,8)$$

Univariate Gaussian Distribution

$$X\sim (\mu, \sigma^2)\quad x\in\Bbb{R}^d$$ $$\Downarrow$$ $$f(x;\mu,\sigma^2)=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2}}$$

Multivariate Gaussian Distribution

$$X\sim (\mu, \Sigma)$$ $$\Downarrow$$ $$f(x;\mu,\Sigma)=\frac{1}{{(2\pi)}^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}{(x-\mu)^T}{\Sigma^{-1}}{(x-\mu)}}$$ $$\mu=\Bbb{E}(x)\quad\Sigma=\Bbb{E}[(x-\mu)(x-\mu)^T]$$

$$X\sim \left(\begin{bmatrix}0\\0\end{bmatrix}, \begin{bmatrix}1 & 0\\0& 1\end{bmatrix}\right)$$

Bivariate Gaussian Distribution

$$X\sim \left(\begin{bmatrix}0\\0\end{bmatrix}, \begin{bmatrix}1 & 0\\0& 1\end{bmatrix}\right)$$ $$\Downarrow$$ $$f(x;\mu,\Sigma)=\frac{1}{{(2\pi)}^{\frac{2}{2}}\left|\begin{matrix}1 & 0\\0& 1\end{matrix}\right|^{\frac{1}{2}}}e^{-\frac{1}{2}{\left(x-\begin{bmatrix}0\\0\end{bmatrix}\right)^T}{\begin{bmatrix}1 & 0\\0& 1\end{bmatrix}}{\left(x-\begin{bmatrix}0\\0\end{bmatrix}\right)}}$$ $$x=\begin{bmatrix}0\\0\end{bmatrix},\quad f\left(\begin{bmatrix}0\\0\end{bmatrix};\begin{bmatrix}0\\0\end{bmatrix},\begin{bmatrix}1 & 0\\0& 1\end{bmatrix}\right)=\frac{1}{2\pi}$$

$$p(x;\mu,\Sigma)=\frac{1}{2\pi}=0.15915$$

What if change $\Sigma?$

What if change $\Sigma?$

What if change $\Sigma?$

How to become thinner?

Question

Find answer from formula

Review

IF

$$x=\begin{bmatrix}x_1\\x_2\end{bmatrix},\quad \mu=\begin{bmatrix}\mu_1\\\mu_2\end{bmatrix},\quad \Sigma=\begin{bmatrix}\sigma_1^2&0\\0&\sigma_2^2\end{bmatrix}$$

THEN

$$\begin{aligned}f(x;\mu,\Sigma)&=\frac{1}{{2\pi}\left|\begin{matrix}\sigma_1^2 & 0\\0& \sigma_2^2\end{matrix}\right|^{\frac{1}{2}}}e^{-\frac{1}{2}{\left(\begin{bmatrix}x_1\\x_2\end{bmatrix}-\begin{bmatrix}\mu_1\\\mu_2\end{bmatrix}\right)^T}{\begin{bmatrix}\frac{1}{\sigma_1^2} & 0\\0& \frac{1}{\sigma_2^2}\end{bmatrix}}{\left(\begin{bmatrix}x_1\\x_2\end{bmatrix}-\begin{bmatrix}\mu_1\\\mu_2\end{bmatrix}\right)}}\\&=\frac{1}{2\pi\sigma_1\sigma_2}e^{-\frac{1}{2\sigma_1^2}(x_1-\mu_1)^2-\frac{1}{2\sigma_2^2}(x_1-\mu_2)^2}\end{aligned}$$

Two Univariate Guassian Distribution

Two Multivariate Guassian Distribution

Data

$$\mu_0=\begin{bmatrix}1\\1\end{bmatrix},\mu_1=\begin{bmatrix}4\\4\end{bmatrix},\Sigma=\begin{bmatrix}1&0\\0&1\end{bmatrix}$$

Distributions for red and blue data points

$$\begin{aligned}&p(y=\text{blue})=\phi\\&p(y=\text{red})=1-\phi\end{aligned}$$

Blue represented by 1，Red represented by 0

$$\begin{aligned}&p(y=1)=\phi\\&p(y=0)=1-\phi\end{aligned}$$

Integerate:

$$\qquad\quad p(y;\phi)=\phi^y(1-\phi)^{(1-y)}$$

Distribution for red data points

$$p(x|y=0)=\frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)}$$

Distribution for red data points

$$p(x|y=1)=\frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)}$$

Suppose we have known all parameters

$$\arg\max p(X,y;\phi,\mu_0,\mu_1,\Sigma)$$

Derivation

$$\begin{aligned}&\arg\max p(X,y;\phi,\mu_0,\mu_1,\Sigma)\\ =&\arg\max\prod_{i=1}^{m}p(x^{(i)},y^{(i)};\phi,\mu_0,\mu_1,\Sigma)\\ =&\arg\max\prod_{i=1}^{m}p(x^{(i)}|y^{(i)};\mu_0,\mu_1,\Sigma)p(y^{(i)};\phi)\\ =&\arg\max\sum_{i=1}^{m}\log p(x^{(i)}|y^{(i)};\mu_0,\mu_1,\Sigma)+\sum_{i=1}^{m}p(y^{(i)};\phi)\end{aligned}$$

How about $\phi?$

$$\begin{aligned}&\frac{\partial\sum_{i=1}^{m} \log p(x^{(i)}|y^{(i)};\mu_0,\mu_1,\Sigma)+\sum_{i=1}^{m} \log p(y^{(i)};\phi)}{\partial \phi}=0\\ \Rightarrow&\frac{\partial\sum_{i=1}^{m}\log p(y^{(i)};\phi)}{\partial \phi}=0\\ \Rightarrow&\frac{\partial\sum_{i=1}^{m}\log \phi^{y^{(i)}}(1-\phi)^{(1-y^{(i)})}}{\partial \phi}=0\\ \Rightarrow&\frac{\partial\sum_{i=1}^{m}{y^{(i)}}\log \phi+{(1-y^{(i)})}\log(1-\phi)}{\partial \phi}=0\\ \Rightarrow&\frac{\partial\sum_{i=1}^{m}{{1}{\{y^{(i)}=1\}}}\log \phi+{1}{\{y^{(i)}=0\}}\log(1-\phi)}{\partial \phi}=0\\ \Rightarrow&\phi=\frac{1}{m}\sum_{i=1}^{m}1\{y^{(i)}=1\}\end{aligned}$$

Find $\mu_0$ and $\mu_1$

$$\begin{aligned}&\frac{\partial\sum_{i=1}^{m} \log p(x^{(i)}|y^{(i)};\mu_0,\mu_1,\Sigma)+\sum_{i=1}^{m} \log p(y^{(i)};\phi)}{\partial \mu_0}=0\\ \Rightarrow&\frac{\partial\sum_{i=1}^{m} \log p(x^{(i)}|y^{(i)};\mu_0,\mu_1,\Sigma)}{\partial \mu_0}=0\\ \Rightarrow&\frac{\partial \sum_{i=1}^{m}\log\frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x^{(i)}-\mu_0)^T\Sigma^{-1}(x^{(i)}-\mu_0)}}{\partial \mu_0}=0\\ \Rightarrow&0+\frac{\partial \sum_{i=1}^{m}{-\frac{1}{2}(x^{(i)}-\mu_0)^T\Sigma^{-1}(x^{(i)}-\mu_0)}}{\partial \mu_0}=0 \end{aligned}$$

Find $\mu_0$ and $\mu_1$ cont.

Given that $\frac{\partial X^TAX}{\partial X}=(A+A^T)X$，令$(x^{(i)}-\mu_0)=X$

$$\begin{aligned}&0+\frac{\partial \sum_{i=1}^{m}{-\frac{1}{2}(x^{(i)}-\mu_0)^T\Sigma^{-1}(x^{(i)}-\mu_0)}}{\partial \mu_0}=0\\ \Rightarrow&{\sum_{i=1}^{m}-\frac{1}{2}((\Sigma^{-1})^T+\Sigma^{-1})(x^{(i)}-\mu_0)\cdot(-1)}=0\\ \Rightarrow& \sum_{i=1}^{m}1\{y^{(i)}=0\}x^{(i)}=\sum_{i=1}^{m}1\{y^{(i)}=0\}\mu_0\\ \Rightarrow&\mu_0=\frac{\sum_{i=1}^{m}1\{y^{(i)}=0\}x^{(i)}}{\sum_{i=1}^{m}1\{y^{(i)}=0\}} \end{aligned}$$

Find $\mu_0$ and $\mu_1$ cont.

$$\mu_0=\frac{\sum_{i=1}^{m}1\{y^{(i)}=0\}x^{(i)}}{\sum_{i=1}^{m}1\{y^{(i)}=0\}}$$ $$\mu_1=\frac{\sum_{i=1}^{m}1\{y^{(i)}=1\}x^{(i)}}{\sum_{i=1}^{m}1\{y^{(i)}=1\}}$$

Find $\Sigma$

$$\begin{aligned}&\frac{\partial\sum_{i=1}^{m} \log p(x^{(i)}|y^{(i)};\mu_0,\mu_1,\Sigma)+\sum_{i=1}^{m} \log p(y^{(i)};\phi)}{\partial \Sigma}=0\\ \Rightarrow&\frac{\partial \sum_{i=1}^{m}\log\frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x^{(i)}-\mu_{y^{(i)}})^T\Sigma^{-1}(x^{(i)}-\mu_{y^{(i)}})}}{\partial \Sigma}=0\\ \Rightarrow&\frac{\partial \sum_{i=1}^{m}-\frac{d}{2}\log2\pi}{\partial \Sigma}+\frac{\partial \sum_{i=1}^{m}-\frac{1}{2}\log|\Sigma|}{\partial \Sigma}\\&+\frac{\partial \sum_{i=1}^{m}{-\frac{1}{2}(x^{(i)}-\mu_{y^{(i)}})^T\Sigma^{-1}(x^{(i)}-\mu_{y^{(i)}})}}{\partial \Sigma}=0\\\\ \Rightarrow&\frac{\partial \sum_{i=1}^{m}-\frac{1}{2}\log|\Sigma|}{\partial \Sigma}+\frac{\partial \sum_{i=1}^{m}{-\frac{1}{2}(x^{(i)}-\mu_{y^{(i)}})^T\Sigma^{-1}(x^{(i)}-\mu_{y^{(i)}})}}{\partial \Sigma}\\ &=0 \end{aligned}$$

Find $\Sigma$ cont.

Given that $\frac{\partial|\Sigma|}{\partial\Sigma}=|\Sigma|\Sigma^{-1},\quad \frac{\partial\Sigma^{-1}}{\partial\Sigma}=-\Sigma^{-2}$

$$\begin{aligned}&\frac{\partial \sum_{i=1}^{m}\log|\Sigma|}{\partial \Sigma}+\frac{\partial \sum_{i=1}^{m}{(x^{(i)}-\mu_{y^{(i)}})^T\Sigma^{-1}(x^{(i)}-\mu_{y^{(i)}})}}{\partial \Sigma}=0\\ \Rightarrow&m\frac{1}{|\Sigma|}|\Sigma|\Sigma^{-1}+\sum_{i=1}^m(x^{(i)}-\mu_{y^{(i)}})^T(x^{(i)}-\mu_{y^{(i)}})(-\Sigma^{-2}))=0\\ \Rightarrow&\Sigma=\frac{1}{m}\sum_{i=1}^{m}(x^{(i)}-\mu_{y^{(i)}})(x^{(i)}-\mu_{y^{(i)}})^T \end{aligned}$$

All parameters

$$\begin{aligned}&\phi=\frac{1}{m}\sum_{i=1}^{m}1\{y^{(i)}=1\}\\\\ &\mu_0=\frac{\sum_{i=1}^{m}1\{y^{(i)}=0\}x^{(i)}}{\sum_{i=1}^{m}1\{y^{(i)}=0\}}\\\\ &\mu_1=\frac{\sum_{i=1}^{m}1\{y^{(i)}=1\}x^{(i)}}{\sum_{i=1}^{m}1\{y^{(i)}=1\}}\\\\ &\Sigma=\frac{1}{m}\sum_{i=1}^{m}(x^{(i)}-\mu_{y^{(i)}})(x^{(i)}-\mu_{y^{(i)}})^T \end{aligned}$$

Estimates

$$\begin{aligned} &\phi=0.5\\\\ &\mu_0=\begin{bmatrix}4.0551\\4.1008\end{bmatrix}\\\\ &\mu_1=\begin{bmatrix}0.85439\\1.03622\end{bmatrix}\\\\ &\Sigma=\begin{bmatrix}1.118822&-0.058976\\-0.058976&1.023049\end{bmatrix} \end{aligned}$$

Distribution Estimation

Predicte

Recall: $\quad p(x|y=0)=\frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)}$

$\phi=0.5\,\mu_0=\begin{bmatrix}4.055\\4.101\end{bmatrix} \Sigma=\begin{bmatrix}1.1188&-0.059\\-0.059&1.023\end{bmatrix}x=\begin{bmatrix}0.88\\3.95\end{bmatrix}$

Find $\quad p\left(x=\begin{bmatrix}0.88\\3.95\end{bmatrix}\Bigg|y=0\right)$

$$\begin{aligned} &\frac{1}{2\pi|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}{\begin{bmatrix}x_1-\mu_1\\x_2-\mu_2\end{bmatrix}^T\Sigma^{-1}\begin{bmatrix}x_1-\mu_1\\x_2-\mu_2\end{bmatrix}}}\\=&\frac{1}{{2\pi}\left|\begin{matrix}1.1188&-0.059\\-0.059&1.023\end{matrix}\right|^{\frac{1}{2}}}e^{-\frac{1}{2}{\begin{bmatrix}-3.175\\-0.151\end{bmatrix}^T\begin{bmatrix}0.896&-0.052\\-0.0520&0.98\end{bmatrix}\begin{bmatrix}-3.175\\-0.151\end{bmatrix}}}\\ =&\frac{1}{2\pi\sqrt{(1.141)}}e^{-\frac{1}{2}\times 9.11} =0.149\times 0.01=0.0015 \end{aligned}$$

Recall: $\quad p(x|y=1)=\frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)}$

$\phi=0.5\,\mu_1=\begin{bmatrix}0.85\\1.036\end{bmatrix}\Sigma=\begin{bmatrix}1.1188&-0.059\\-0.059&1.023\end{bmatrix}x=\begin{bmatrix}0.88\\3.95\end{bmatrix}$

Find $\quad p\left(x=\begin{bmatrix}0.88\\3.95\end{bmatrix}\Bigg| y=1\right)$

$$\begin{aligned} &\frac{1}{2\pi|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}{\begin{bmatrix}x_1-\mu_1\\x_2-\mu_2\end{bmatrix}^T\Sigma^{-1}\begin{bmatrix}x_1-\mu_1\\x_2-\mu_2\end{bmatrix}}}\\=&\frac{1}{{2\pi}\left|\begin{matrix}1.1188&-0.059\\-0.059&1.023\end{matrix}\right|^{\frac{1}{2}}}e^{-\frac{1}{2}{\begin{bmatrix}0.03\\2.91\end{bmatrix}^T\begin{bmatrix}0.896&-0.052\\-0.0520&0.98\end{bmatrix}\begin{bmatrix}0.03\\2.91\end{bmatrix}}}\\ =&\frac{1}{2\pi\sqrt{(1.141)}}e^{-\frac{1}{2}\times 8.336} =0.149\times 0.015=0.0022 \end{aligned}$$

$$\quad p\left(x=\begin{bmatrix}0.88\\3.95\end{bmatrix}\Bigg| y=0\right)=0.0015$$

$$\quad p\left(x=\begin{bmatrix}0.88\\3.95\end{bmatrix}\Bigg| y=1\right)=0.0022$$

Thus, decided as blue. False!

Decision Boundary

$-3.0279x_1-3.1701x_2+15.575=0$

Notes

We can derive Logistic Regression: $p(y=1|x)=\frac{1}{1+e^{-w^Tx}}$

1. If $x|y$ with a Multivariate Gaussian Distribution，then the posterior probability $y|x$ with a Logistic Regression; but the inverse is not true.

2. If $x|y$ with a Gaussian Distribution, then GDA is a good choice; otherwise Logistic Regression is a better one.

3. If $x|y=0$ and $x|y=1$ are both with Poisson Distribution, then $y|x$ with Logistic Regression as well.

Summary

• Univariate Gaussian Distribution
• Multivariate Gaussian Distribution
• Gaussian Discriminant Analysis
• Parameter Estimation and Inference

Reference