Convergence of Random Variables

1. Introduction

1.1 Convergence of Sequences

Convergence is not a new concept in mathematics. In calculus, we define convergence of a sequence $X_n\to X$ as:

$$\forall ϵ>0,\exists N\in \mathbb{N},\text{ s.t. }\forall n\ge N,|X_n-X|<ϵ.$$

Or in vector form, a sequence of vectors $X_n\to X$ if:

$$\forall ϵ>0,\exists N\in \mathbb{N},\text{ s.t. }\forall n\ge N,\Vert X_n-X\Vert <ϵ.$$

Here, when we talk about convergence, we are talking about limits, and when we talk about limits, we are talking about approximations. Thus we need a distance metric to measure the closeness of two points, such as $|X_n-X|$ or $\Vert X_n-X\Vert$.

1.2 Convergence of Functions

Recall that, in stochastic calculus, $X_n$ is a random variable, and random variables are functions from $\Omega$ to $\mathbb{R}$. Thus, convergence of random variables is essentially the limit of a sequence of functions.

Assume that $\{f_n(x)\}$ is a sequence of functions and $f(x)$ is a function, where all functions map from $A\subseteq \mathbb{R}$ to $B\subseteq \mathbb{R}$.

Definition (Pointwise convergence).

$$\forall x\in A,\forall ϵ>0,\exists N\in \mathbb{N},\text{ s.t. }\forall n\ge N,|f_n(x)-f(x)|<ϵ.$$

Example (Pointwise convergence). $f_n(x)=x^n$ converges pointwise to $f(x)=0$ on $[0,1)$.

Definition (Uniform convergence).

$$\forall ϵ>0,\exists N\in \mathbb{N},\text{ s.t. }\forall x\in A,\forall n>N,|f_n(x)-f(x)|<ϵ.$$

Notice the difference between pointwise and uniform convergence is the position of the quantifiers, i.e. $\forall x$ and $\forall n$. In pointwise convergence, $\exists N_{x,ϵ}$ is dependent on $x$ and $ϵ$, while in uniform convergence, $\exists N_{ϵ}$ is independent of $x$.

Definition (Integral convergence).

$$\Vert f_n-f\Vert ={\int }_A|f_n(x)-f(x)|dx\to 0,\text{as }n\to \infty .$$

In this note, we introduce four types of convergence in probability theory:

• Mean square convergence
• Almost sure convergence
• Convergence in probability
• Convergence in distribution

The essential difference between them is how we measure distance between random variables.

2. Convergence of Random Variables

2.1 Mean Square Convergence

In mean square convergence, we measure the distance between two random variables by the mean square error:

$$d(X,Y)=\sqrt{\mathbb{E}|X-Y{|}^2}.$$

Definition (Mean square convergence). $X_n$ converges to $X$ in mean square, written $X_n\stackrel{ms}{\to }X$, if:

$$\underset{n\to \infty }{\lim }\mathbb{E}|X_n-X{|}^2=0.$$

Equivalently,

$$\forall ϵ>0,\exists N\in \mathbb{N},\text{ s.t. }\forall n\ge N,\mathbb{E}|X_n-X{|}^2<ϵ.$$

Proposition (Mean square algebra). Given $X_n\stackrel{ms}{\to }X$ and $Y_n\stackrel{ms}{\to }Y$:

• $X_n+Y_n\stackrel{ms}{\to }X+Y$ as $n\to \infty$.

  Proof of additivity

  $$d(X_n+Y_n,X+Y)\le d(X_n,X)+d(Y_n,Y)\to 0.$$

  $\square$

• $X_n{Y}_n\stackrel{ms}{\to }XY$ as $n\to \infty$.

  Proof of multiplicativity

  $$\Vert X_n{Y}_n-XY\Vert =\Vert X_n{Y}_n-X_{n}Y+X_{n}Y-XY\Vert$$ $$\le \Vert X_n\Vert \cdot \Vert Y_n-Y\Vert +\Vert Y\Vert \cdot \Vert X_n-X\Vert \to 0.$$

  $\square$

• Cauchy Criterion: $\Vert X_n-X_m\Vert \stackrel{ms}{\to }0$ as $n,m\to \infty$ implies there exists $X$ such that $X_n\stackrel{ms}{\to }X$.

Note that, in statistics, consistency is a concept similar to mean square convergence. If a sequence of estimators ${\hat{\theta }}_n$ converges to $\theta$ in mean square, then ${\hat{\theta }}_n$ is a consistent estimator of $\theta$.

Note: $L_p$ convergence

Convergence in mean square is a special case of convergence in $L_p$ with $p=2$:

$$\underset{n\to \infty }{\lim }\mathbb{E}|X_n-X{|}^p=0.$$

Thus we sometimes write it as $X_n\stackrel{L_2}{\to }X$.

2.2 Almost Sure Convergence

Definition (Almost sure convergence). A sequence of random variables $X_1,X_2,\cdots$ converges almost surely to a random variable $X$, written $X_n\stackrel{a.s.}{\to }X$, if, for every $ϵ>0$:

$$\mathbb{P}\left(\underset{n\to \infty }{\lim }|X_n-X|<ϵ\right)=1.$$

Equivalently,

$$\mathbb{P}\{\omega \in \Omega :X_n(\omega )\to X(\omega )\}=1.$$

Here, almost surely means $X_n$ converges to $X$ for almost all $\omega \in \Omega$, except perhaps for some set $N\subseteq \Omega$ with $\mathbb{P}(N)=0$. Thus, $\text{a.s.}\approx \text{pointwise}-\text{zero-probability set}$.

Example (Almost sure convergence). Let sample space $\Omega =[0,1]$ with uniform distribution. Define $X_n(\omega )=\omega +{\omega }^n$ and $X(\omega )=\omega$. Then for any $\omega \in [0,1)$, ${\omega }^n\to 0$ as $n\to \infty$. Thus $X_n\to X$ for almost all $\omega \in \Omega$. The only exception is $\omega =1$, but $\mathbb{P}(\{1\})=0$.

Proposition (No direct relation to mean square convergence). Almost sure convergence and mean square convergence have no direct implication either way:

$$X_n\stackrel{a.s.}{\to }X\nLeftrightarrow X_n\stackrel{ms}{\to }X.$$

Theorem (Strong law of large numbers). Let $X_1,X_2,\dots ,X_n$ be i.i.d. random variables with $\mathbb{E}[X_i]=\mu$ and $\mathrm{Var}[X_i]={\sigma }^2<\infty$. Denote ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$. Then, for any $ϵ>0$,

$$\mathbb{P}\left(\underset{n\to \infty }{\lim }|{\bar{X}}_n-\mu |<ϵ\right)=1.$$

2.3 Convergence in Probability

Definition (Convergence in probability). A sequence of random variables $X_1,X_2,\cdots$ converges in probability to a random variable $X$, written $X_n\stackrel{p}{\to }X$, if, for every $ϵ>0$:

$$\underset{n\to \infty }{\lim }\mathbb{P}(|X_n-X|<ϵ)=1.$$

Equivalently,

$$\underset{n\to \infty }{\lim }\mathbb{P}(|X_n-X|\ge ϵ)=0.$$

It means that the outliers $\omega$ that make $X_n(\omega )$ far away from $X(\omega )$ become less and less likely as $n\to \infty$.

Theorem (Almost sure convergence implies convergence in probability). If $X_n\stackrel{a.s.}{\to }X$, then $X_n\stackrel{p}{\to }X$.

Theorem (Continuous mapping for convergence in probability). Suppose $X_1,X_2,\cdots$ converges to $X$ in probability, and $h(\cdot )$ is continuous. Then $h(X_1),h(X_2),\cdots$ converges to $h(X)$ in probability.

Theorem (Weak law of large numbers). Let $X_1,X_2,\dots ,X_n$ be i.i.d. random variables with $\mathbb{E}[X_i]=\mu$ and $\mathrm{Var}[X_i]={\sigma }^2<\infty$. Denote ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$. Then, for any $ϵ>0$,

$$\underset{n\to \infty }{\lim }\mathbb{P}(|{\bar{X}}_n-\mu |<ϵ)=1.$$

2.4 Convergence in Distribution / Weak Convergence

Definition (Convergence in distribution). A sequence of random variables $X_1,X_2,\cdots$ converges in distribution to a random variable $X$, written $X_n\stackrel{d}{\to }X$, $X_n\stackrel{L}{\to }X$, or $F_n(x)\stackrel{w}{\to }{F}_X(x)$, if for every $x\in \mathbb{R}$ where $F_X(x)$ is continuous:

$$\underset{n\to \infty }{\lim }{F}_{X_n}(x)=F_X(x).$$

Equivalently, if $X_n\sim F_{X_n}$ and $X\sim F_X$, then $X_n\stackrel{d}{\to }X$ iff $F_{X_n}(x)\to F_X(x)$ for all continuity points of $F_X$.

Theorem (Slutsky’s theorem). Let $X_n\stackrel{d}{\to }X$ and $Y_n\stackrel{p}{\to }c$, where $c$ is a constant. Then:

• $X_n+Y_n\stackrel{d}{\to }X+c$.
• $X_n{Y}_n\stackrel{d}{\to }cX$.

Theorem (Convergence in probability implies convergence in distribution). If $X_n\stackrel{p}{\to }X$, then $X_n\stackrel{d}{\to }X$.

Corollary (Almost sure convergence implies convergence in distribution). If $X_n\stackrel{a.s.}{\to }X$, then $X_n\stackrel{d}{\to }X$.

Note: CDF-level convergence

Convergence in distribution is essentially the convergence of the CDFs, rather than the random variables themselves. It is different from the other types of convergence, yet those types can imply convergence in distribution.

Theorem (Constant limit equivalence). Random variables $X_n$ converge in probability to some constant $\mu$ if and only if $X_n$ converges in distribution to $\mu$:

$$X_n\stackrel{p}{\to }\mu \iff X_n\stackrel{d}{\to }\mu .$$

3. Relationship Summary

$$\begin{array}{rl}& \text{Almost Sure Convergence}\Rightarrow \text{Convergence in Probability}\Rightarrow \text{Convergence in Distribution},\\ & \text{Mean Square Convergence}\Rightarrow \text{Convergence in Probability}.\end{array}$$

However, there is no direct relationship between almost sure convergence and mean square convergence.

Related Notes

• Chebyshev Inequality
• Central Limit Theorem