3.8. Independent identically distributed random variables

Definition 3.7 (Independent identically distributed (iid) random variables)

3.8.1. Law of the large numbers

3.8.1.1. Weak form

todo

3.8.1.2. Strong form

todo

3.8.2. Central Limit Theorem

Theorem 3.2 (CLT)

Let \(\{ X_k \}_{k=1:n}\) a sequence of iid random variables with average value \(\mathbb{E}[X_k] = \mu\) and finite1 variance \(\mathbb{E}[(X_k-\mu)^2] = \sigma^2 < \infty\), then the sample average

\[\overline{X}_n := \dfrac{1}{n} \sum_{k=1}^n X_k \ ,\]

converges in distribution - or weakly converges - to the normal distribution \(\mathscr{N}\left(\mu, \frac{\sigma^2}{n} \right)\),

\[\overline{X}_n \quad \rightarrow^d \quad \mathscr{N}\left(\mu, \frac{\sigma^2}{n} \right) \ .\]

Proof of CLT

Let \(\{ X_k \}_{k=1:n}\) the sequence of iid random variables. Thus, \(\sum_{k=1}^n X_k\) has expected value \(n \mu\) and variance \(n \sigma^2\). Let

\[Z_n := \dfrac{\sum_{k=1}^n X_k - n \mu}{\sqrt{n \sigma^2}} = \sum_{k=1}^{n} \dfrac{X_k - \mu}{\sqrt{n \sigma^2}} =: \sum_{k=1}^n \dfrac{Y_k}{\sqrt{n}} \ .\]

Expeceted value and variance of variables \(Y_k\) are respectively \(\mathbb{E}[Y_k] = 0\) and \(\mathbb{E}[Y_k^2] = 1\). The characteristic function of \(Z_n\), see Example 3.2 for the linear combination of independent variables, reads

\[\begin{split}\begin{aligned} \varphi_{Z_n}(t) & = \varphi_{Y_1} \left( \dfrac{t}{\sqrt{n}} \right) \dots \varphi_{Y_n} \left( \dfrac{t}{\sqrt{n}} \right) = && (1) \\ & = \left[ \varphi_{Y_1} \left( \dfrac{t}{\sqrt{n}} \right) \right]^n \ , \end{aligned}\end{split}\]

as \((1)\) the variables are not only independent but identically distributed: as they have the same pdf, they also have the same characteristic function. Expanding in Taylor series, see example Example 3.3 for \(\frac{t}{\sqrt{n}} \rightarrow 0\), the approximation of the characteristic function reads (remembering that \(Y_n\) have zero expected value and unit variance),

\[\varphi_{Y_1}\left( \dfrac{t}{\sqrt{n}} \right) \sim 1 - \dfrac{t^2}{2 n} \ ,\]

while

\[\varphi_{Z_n}(t) = \left[ \varphi_{Y_1} \left( \dfrac{t}{\sqrt{n}} \right) \right]^n \sim \left[ 1 - \dfrac{t^2}{2 n} \right]^n \sim e^{- \frac{t^2}{2}} \ ,\]

i.e. it converges to the characteristic function of a normal distribution \(\mathscr{N}(0,1)\), see Example 3.4.

Levy’s continuity theorem completes the proof. todo

---

1

Does the CLT hold for heavy-tailed distributions? See section about Rare Events. work-in-progress