> [!definition] > > The normal distribution $N(\mu, \sigma^2)$ is a [[Probability Distribution|probability distribution]] where the probability density is the highest in the middle, and decreases symmetrically in both directions. It has the following [[Probability Distribution Function|probability distribution function]]: > > $ > p(x) = \frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} > $ > > The parameter $\mu$ is its [[Expectation|expected value]], and the parameter $\sigma^2$ is its [[Variance|variance]]. As a result, the function is symmetric across $x = \mu$, has inflection points at $x = \mu \pm \sigma$, and flattens as $\sigma$ increases. > > $ > p(x) = \frac{1}{\sqrt{\det(2\pi C)}} \exp \paren{-\frac{1}{2}(x - \mu)C^{-1}(x-\mu)} > $ > $\gamma_{\mu, \sigma^2}$ denotes the probability distribution of $N(\mu, \sigma^2)$. If $\mu \in \real^n$, then $\gamma_{\mu, C}$ denotes the probability distribution of $N(\mu, C)$ where $C$ is the covariance matrix. > [!theorem] [[Differential Entropy]] > > $ > H(X) = \log_2\paren{\sigma\sqrt{2\pi e}} > $ > *Proof*. Using a substitution with $y = \frac{x - \mu}{\sigma\sqrt{2}}$ > $ > \begin{align*} > H(X) &= -\frac{1}{\sigma \sqrt{2\pi}}\int_{-\infty}^{\infty}\exp\paren{-\frac{1}{2}\paren{\frac{x - \mu}{\sigma}}^2} \\ > &\times\log_2\paren{\frac{1}{\sigma \sqrt{2\pi}}\exp\paren{-\frac{1}{2}\paren{\frac{x - \mu}{\sigma}}^2}}dx \\ > &= \log_2\paren{\sigma\sqrt{2\pi}} + \frac{\log_2(e)}{\sqrt{\pi}}\int_{-\infty}^{\infty}e^{-y^2}y^2dy\\ > &= \log_2\paren{\sigma\sqrt{2\pi}} + \frac{\log_2(e)}{\sqrt{\pi}}\cdot\frac{1}{2}\sqrt{\pi}\\ > &= \log_2\paren{\sigma\sqrt{2\pi}} + \frac{\log_2(e)}{\sqrt{\pi}}\cdot\frac{1}{2}\sqrt{\pi} \\ > &= \log_2\paren{\sigma\sqrt{2\pi e}} > \end{align*} > $ > [!theorem] > > A [[Linear Combination|linear combination]] of [[Probabilistic Independence|independent]] normally distributed random variables is also normally distributed. $\sum{aX} = N(\sum{a\mu}, \sum{a^2\sigma^2})$. > [!definition] > > An entry is considered to be an **outlier** (*outside the bell curve*) if it is outside of three [[Standard Deviation|standard deviations]] of the [[Mean|mean]]. > [!definition] > > $ > \Phi(z) = P(Z \le z) = \int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-\frac{t^2}{2}} > $ > > The [[Random Variable|random variable]] with normal distribution $N(0, 1)$ is denoted by $Z$ and is known as the standard normal distribution. Its *cumulative* probability distribution is denoted as $\Psi (z)$. > > If $X$ is $N(\mu, \sigma^2)$, then its [[Z-Score|z-score]] $\frac{X - \mu}{\sigma} = N(0, 1) = Z$, allowing mapping between the standard normal distribution and any normal distribution. > > $z_\alpha$ is the number such that $P(Z \ge z_\alpha) = \alpha$, meaning that $z_\alpha$ is the $100(1-\alpha)$-th [[Tiles|percentile]] of the standard normal distribution. Since the standard normal distribution is symmetrical, $z_{1-\alpha} = -z_\alpha$. > [!theorem] > > Let $\seq{X_n}$ be a family of independent, identically distributed standard Gaussian random variables. If $S_n = \sum_{k \le n}X_n$ is their partial sums, then > $ > \frac{S_n}{\sqrt{n}} \sim N(\mu, \sigma^2) > $ > Let $L = \limsup_{n \to \infty}\frac{S_n}{\sqrt{n}}$, then $L$ is [[Kolmogorov's 0-1 Law|tail-measurable]] and hence constant almost surely. Let $R \in \real$, then $\bracs{L > R}$ is a tail event > $ > \begin{align*} > \bp\bracs{L > R} &= \bp\bracs{\limsup\frac{S_n}{\sqrt{n}} > R} \\ > &\ge \bp\bracs{\limsup\bracs{\frac{S_n}{\sqrt{n}} > r + 1}} \\ > &\ge \limsup\bp\bracs{\frac{S_n}{\sqrt{n}} > R + 1} \\ > &> 0 > \end{align*} > $ > By pushing the computations to the sequence of distributions, we get that $\bp\bracs{L > R} = 1$. Hence $L = \infty$ almost surely. Similarly, $\liminf \frac{S_n}{\sqrt{n}} = -\infty$ almost surely. Thus the sequence $\frac{S_n}{\sqrt{n}}$ cannot converge. > [!theorem] > > Let $X$ and $Y$ be two independent, identically distributed random variables on the same probability space such that $\ev(X) = 0$ and $\ev(X^2) = 1$. If any of the following are true: > - $(X + Y)/\sqrt{2}$ has the same distribution as $X$. > - $X + Y$ and $X - Y$ are independent. > > then $X$ has $\gamma_{0, 1}$ distribution. > > *Proof*. Let $\mu$ be the distribution of $X$. Let $Z = X + Y$ and $W = X - Y$, then $2X = Z + W$. Taking the characteristic functions, > $ > \wh \mu(2\xi) = (\wh \mu(\xi))^2 \cdot (\wh \mu(\xi) \cdot \wh \mu(-\xi)) = [\wh\mu(\xi)]^3 \cdot \wh \mu(-\xi) > $ > On the flip side, $\wh\mu(-2\xi) = [\wh \mu(-\xi)]^3 \cdot \wh \mu(\xi)$. So > $ > \frac{\wh \mu(2\xi)}{\wh \mu(-2\xi)} = \braks{\frac{\wh \mu(\xi)}{\wh \mu(-\xi)}}^2 > $ > where the denominator cannot be $0$ as otherwise $\wh \mu(0) = 0$ by continuity. Let $g = \wh \mu(\xi)/\wh \mu(-\xi)$, then $g(\xi) = g(\xi/2)^2 = g(\xi/2^n)^{2^n}$. Over a Taylor expansion, > $ > g(\xi) = \braks{1 + 0 + o(1/2^n)}^{2^n} > $ > so $g$ is constant, and $\wh \mu$ is symmetric, > $ > \wh \mu(\xi) = \wh \mu\paren{\frac{\xi}{2}}^4 = \wh \mu\paren{\frac{\xi}{2^n}}^{2^n} > $ > Over a Taylor expansion, > $ > \wh \mu(\xi) = \braks{1 + 0 - \frac{\xi^2}{2n} + o(1/2^{2n})}^{2^{2n}} > $ [^1]: Ideally, $np \ge 10$ and $nq \ge 10$.