Cramér-Rao Lower Bound

1. Intuition

Starting from ancillary statistics, we can derive a better and better estimator by the Rao-Blackwell procedure. Is there a limit for the quality of an estimator? The answer is yes, and it is the Cramér-Rao Lower Bound.

2. Cramér-Rao Lower Bound

Here, we first give a simple version of Cramér-Rao Lower Bound, where we only focus on the parameter $\theta$ rather than a function of $\theta$.

Theorem (Cramér-Rao Lower Bound, simple version). Given a random sample $\mathbf{X}=(X_1,\cdots ,X_n{)}^T$ from a distribution with pdf $f(\mathbf{x}|\theta )$. Then for any unbiased estimator $\hat{\theta }(\mathbf{X})$,

$$\text{Var}(\hat{\theta }(\mathbf{X}))\ge \frac{1}{I(\theta )},$$

where $I(\theta )$ is the Fisher Information of $\theta$:

$$I(\theta )={\mathbb{E}}_{\theta }\left[{\left(\frac{\partial }{\partial \theta }\log f(\mathbf{X}|\theta )\right)}^2\right]=-{\mathbb{E}}_{\theta }\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(\mathbf{X}|\theta )\right].$$

Note: Fisher information conventions

There are several ways to define Fisher Information.

• The first one is derived directly from the pdf of a distribution, i.e. the population distribution.
• The second way is to view it from a sampling perspective. Then $f(\mathbf{x}|\theta )$ is replaced by the likelihood function $L(\theta |\mathbf{x})$, or regarded as the pdf of the joint distribution.

In the likelihood convention,

$$\tilde{I}(\theta )={\mathbb{E}}_{\theta }\left[{\left(\frac{\partial }{\partial \theta }\log L(\theta |\mathbf{X})\right)}^2\right]=-{\mathbb{E}}_{\theta }\left[\frac{{\partial }^2}{\partial {\theta }^2}\log L(\theta |\mathbf{X})\right].$$

The two definitions are essentially equivalent, as the likelihood function is just the pdf of the sample:

$$\frac{\partial \log L(\theta |\mathbf{X})}{\partial \theta }=\sum _{i=1}^n\frac{\partial \log f(X_i|\theta )}{\partial \theta }.$$

Thus for Fisher Information,

$$\tilde{I}(\theta )=nI(\theta ).$$

We have to pay attention to the dimension or meaning of the function when calculating Fisher Information, and we have to be consistent. Usually, the log-likelihood function is used in practice.

Proof of the simple Cramér-Rao Lower Bound

First, as $\hat{\theta }(\mathbf{X})$ is unbiased,

$$\begin{array}{rl}\theta =\mathbb{E}[\hat{\theta }(\mathbf{X})]& ={\int }_{\mathbb{R}}\hat{\theta }(\mathbf{x})f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}.\end{array}$$

Taking the derivative with respect to $\theta$ on both sides:

$$\begin{array}{rl}\frac{\partial }{\partial \theta }\theta & =\frac{\partial }{\partial \theta }{\int }_{\mathbb{R}}\hat{\theta }(\mathbf{x})f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}\\ \iff 1& ={\int }_{\mathbb{R}}\hat{\theta }(\mathbf{x})\frac{\partial }{\partial \theta }f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}.\cdots \text{(1)}\end{array}$$

Meanwhile, consider the pdf ${\int }_{\mathbb{R}}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}=1$ and take the derivative with respect to $\theta$ on both sides:

$$\begin{array}{rl}0& ={\int }_{\mathbb{R}}\frac{\partial }{\partial \theta }f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}\\ \iff 0\cdot \theta & =0={\int }_{\mathbb{R}}\theta \frac{\partial }{\partial \theta }f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}.\cdots \text{(2)}\end{array}$$

Subtracting (2) from (1):

$$\begin{array}{rl}(1)-(2):1& ={\int }_{\mathbb{R}}(\hat{\theta }(\mathbf{x})-\theta )\frac{\partial }{\partial \theta }f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}\\ & ={\int }_{\mathbb{R}}(\hat{\theta }(\mathbf{x})-\theta )\left(\frac{\partial }{\partial \theta }\log f(\mathbf{x};\theta )\right)f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}\text{(Fisher Trick)}\\ & ={\int }_{\mathbb{R}}(\hat{\theta }(\mathbf{x})-\theta )\sqrt{f(\mathbf{x};\theta )}\frac{\partial }{\partial \theta }\log f(\mathbf{x};\theta )\sqrt{f(\mathbf{x};\theta )}\mathrm{d}\mathbf{x}\\ & \le \sqrt{{\int }_{\mathbb{R}}(\hat{\theta }(\mathbf{x})-\theta {)}^{2}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}}\sqrt{{\int }_{\mathbb{R}}{\left(\frac{\partial }{\partial \theta }\log f(\mathbf{x};\theta )\right)}^{2}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}}\\ & \text{(Cauchy-Schwarz Inequality)}\\ \iff 1& \le {\int }_{\mathbb{R}}(\hat{\theta }(\mathbf{x})-\theta {)}^{2}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}{\int }_{\mathbb{R}}{\left(\frac{\partial }{\partial \theta }\log f(\mathbf{x};\theta )\right)}^{2}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}\\ & =\mathbb{E}\left[(\hat{\theta }(\mathbf{X})-\theta {)}^2\right]\mathbb{E}\left[{\left(\frac{\partial }{\partial \theta }\log f(\mathbf{X};\theta )\right)}^2\right]\\ & =\text{Var}(\hat{\theta }(\mathbf{X}))I(\theta ).\end{array}$$

Thus,

$$\text{Var}(\hat{\theta }(\mathbf{X}))\ge \frac{1}{I(\theta )}.$$

$\square$

Theorem (Cramér-Rao Lower Bound, general version). If we generalize the estimator from $\mathbb{E}(\hat{\theta })=\theta$ to a function of the parameter $\mathbb{E}(\hat{g}(\mathbf{X}))=g(\theta )={\int }_{\mathbb{R}}\hat{g}(\mathbf{x})f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}$, then its derivative with respect to $\theta$ is

$$g^{\prime }(\theta )={\int }_{\mathbb{R}}\hat{g}(\mathbf{x})\frac{\partial }{\partial \theta }f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}.$$

Following the same procedure of the proof,

$$\text{Var}(\hat{g}(\theta ))\ge \frac{g^{\prime }(\theta {)}^2}{\mathcal{I}(\theta )}.$$

Here, $\mathcal{I}(\theta )$ is the Fisher Information defined from the joint distribution.

Note: Multivariate Cramér-Rao Lower Bound

The source note only records a placeholder heading for the multivariate version. The statement is left as a planned expansion rather than presented as a complete theorem here.

3. Examples

Example (Normal mean with known variance). Let $X_1,\cdots ,X_n{\sim }_{i.i.d.}\mathcal{N}(\mu ,{\sigma }_{\text{known}}^2)$. Find the Cramér-Rao Lower Bound for $\theta =\mu$.

1. Clarify the joint distribution and log-likelihood function.

   $$\begin{array}{rl}f(\mathbf{x};\mu )& =\prod _{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right),\\ \log f(\mathbf{x};\mu )& =-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(x_i-\mu {)}^2.\end{array}$$

2. Take the derivative.

   $$\begin{array}{rl}\frac{\partial }{\partial \mu }\log f(\mathbf{x};\mu )& =\frac{1}{{\sigma }^2}\sum _{i=1}^n(x_i-\mu ),\\ \frac{{\partial }^2}{\partial {\mu }^2}\log f(\mathbf{x};\mu )& =-\frac{n}{{\sigma }^2}.\end{array}$$

3. Calculate the Fisher Information.

   $$\begin{array}{rl}I(\mu )& =-{\mathbb{E}}_{\mu }\left[\frac{{\partial }^2}{\partial {\mu }^2}\log f(\mathbf{X};\mu )\right]=\frac{n}{{\sigma }^2}\text{or equivalently}\\ I(\mu )& ={\mathbb{E}}_{\mu }\left[{\left(\frac{\partial }{\partial \mu }\log f(\mathbf{X};\mu )\right)}^2\right]={\mathbb{E}}_{\mu }\left[\frac{n^2}{{\sigma }^4}{\left(\sum _{i=1}^n\frac{X_i}{n}-\mu \right)}^2\right]=\frac{n}{{\sigma }^2}.\end{array}$$

4. Give the Cramér-Rao Lower Bound.

   $$\text{Var}(\hat{\mu }(\mathbf{X}))\ge \frac{1}{I(\mu )}={\sigma }^2/n.$$

Example (Poisson rate). Let $X_1,\cdots ,X_n{\sim }_{i.i.d.}\text{Poisson}(\lambda )$. Find the Cramér-Rao Lower Bound for $\theta =\lambda$.

1. Clarify the joint distribution and log-likelihood function.

   $$\begin{array}{rl}f(\mathbf{x};\lambda )& =\prod _{i=1}^n\frac{{\lambda }^{x_i}}{x_i!}\exp (-\lambda ),\\ \log f(\mathbf{x};\lambda )& =\sum _{i=1}^n{x}_i\log (\lambda )-n\lambda -\sum _{i=1}^n\log (x_i!).\end{array}$$

2. Take the derivative.

   $$\begin{array}{rl}\frac{\partial }{\partial \lambda }\log f(\mathbf{x};\lambda )& =\frac{1}{\lambda }\sum _{i=1}^n{x}_i-n,\\ \frac{{\partial }^2}{\partial {\lambda }^2}\log f(\mathbf{x};\lambda )& =-\frac{{\sum }_{i=1}^n{x}_i}{{\lambda }^2}.\end{array}$$

3. Calculate the Fisher Information.

   $$I(\lambda )=-{\mathbb{E}}_{\lambda }\left[\frac{{\partial }^2}{\partial {\lambda }^2}\log f(\mathbf{X};\lambda )\right]=\frac{n}{\lambda }.$$

4. Give the Cramér-Rao Lower Bound.

   $$\text{Var}(\hat{\lambda }(\mathbf{X}))\ge \frac{1}{I(\lambda )}=\frac{\lambda }{n}.$$

4. Conditions for Equality

The Cramér-Rao Lower Bound is a lower bound for the variance of any unbiased estimator. When equality holds, the estimator is called an efficient estimator.

Recall the proof of Cramér-Rao Lower Bound, by the Cauchy-Schwarz inequality:

$$\begin{array}{rl}1& \le {\int }_{\mathbb{R}}(\hat{\theta }(\mathbf{x})-\theta {)}^{2}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}{\int }_{\mathbb{R}}{\left(\frac{\partial }{\partial \theta }\log f(\mathbf{x};\theta )\right)}^{2}f(\mathbf{x};\theta )\mathrm{d}\mathbf{x}.\end{array}$$

The equality holds if and only if the two functions are linearly dependent, i.e. there exists $c(\theta )$ such that

$$c(\theta )\left(\hat{\theta }(\mathbf{x})-\theta \right)=\frac{\partial }{\partial \theta }\log f(\mathbf{x};\theta ).$$

Actually it can be shown that equality holds if and only if the distribution has the form

$$f(\mathbf{x};\theta )=\exp \left(A(\theta )\hat{\theta }(\mathbf{x})\right)\exp \left(B(\theta )\right)\exp \left(h(\mathbf{x})\right),$$

which is the form of the exponential family distribution.

Claim (Exponential-family equality condition). As long as Cramér-Rao Lower Bound is achieved, it must be from an exponential family distribution, and vice versa.

It can also be shown that such $c(\theta )\equiv \mathcal{I}(\theta )$, where $\mathcal{I}(\theta )$ is the Fisher Information of $\theta$.

Related Notes

• Fisher Information
• Rao-Blackwell Theorem
• UMVUE