Rao-Blackwell Theorem

1. Statement

Theorem (Rao-Blackwell Theorem). If $\hat{\theta }(\mathbf{X})$ is an unbiased estimator for $\theta$, and $S(\mathbf{X})$ is any sufficient statistic, let

$${\hat{\theta }}_{RB}(\mathbf{X})={\mathbb{E}}_{\mathbf{X}}[\hat{\theta }(\mathbf{X})|S(\mathbf{X})]:=g(S).$$

Then ${\hat{\theta }}_{RB}(\mathbf{X})$ is a better estimator of $\theta$ than $\hat{\theta }(\mathbf{X})$, as

$$\text{Var}({\hat{\theta }}_{RB}(\mathbf{X}))\le \text{Var}(\hat{\theta }(\mathbf{X})).$$

Equivalently,

$$\text{MSE}({\hat{\theta }}_{RB}(\mathbf{X}))\le \text{MSE}(\hat{\theta }(\mathbf{X})).$$

2. Proof

Proof

First it is easy to verify that ${\hat{\theta }}_{RB}(\mathbf{X})$ is unbiased:

$${\mathbb{E}}_{\mathbf{X}}[{\hat{\theta }}_{RB}(\mathbf{X})]={\mathbb{E}}_{\mathbf{X}}[{\mathbb{E}}_{\mathbf{X}}[\hat{\theta }(\mathbf{X})|S(\mathbf{X})]]={\mathbb{E}}_{\mathbf{X}}[\hat{\theta }(\mathbf{X})]=\theta .$$

Then,

$$\begin{array}{rl}\text{Var}(\hat{\theta })& =\text{Var}(\mathbb{E}[\hat{\theta }|\mathbf{S}])+\mathbb{E}[\text{Var}(\hat{\theta }|\mathbf{S})]\\ & =\text{Var}({\hat{\theta }}_{RB})+\mathbb{E}[\text{Var}(\hat{\theta }|\mathbf{S})]\text{(by definition)}\\ & \ge \text{Var}({\hat{\theta }}_{RB})\text{(as Var(θ^∣S)≥0)}.\end{array}$$

Moreover, given both $\hat{\theta }(\mathbf{X})$ and ${\hat{\theta }}_{RB}(\mathbf{X})$ are unbiased,

$$\text{MSE}({\hat{\theta }}_{RB})=\text{Var}({\hat{\theta }}_{RB})\le \text{Var}(\hat{\theta })=\text{MSE}(\hat{\theta }).$$

$\square$

3. Example

Example (Normal mean estimator). Consider $X_1,\cdots ,X_n\sim \mathcal{N}(\mu ,{\sigma }_{\text{known}}^2)$. Let $S={\sum }_{i=1}^n{X}_i$. Then $S$ is a sufficient statistic for $\mu$. Consider a naive estimator $\hat{\mu }(\mathbf{X})=X_1$. Applying the Rao-Blackwell theorem, let

$${\hat{\mu }}_{RB}(\mathbf{X})={\mathbb{E}}_{\mathbf{X}}[\hat{\mu }(\mathbf{X})|S(\mathbf{X})].$$

Now solve ${\mathbb{E}}_{\mathbf{X}}[X_1|S(\mathbf{X})]$. Note that, for any $i,j$,

$$\mathbb{E}(X_i|S)=\mathbb{E}(X_j|S).$$

Thus,

$$\mathbb{E}(X_1|S)=\frac{1}{n}\sum _{i=1}^n\mathbb{E}(X_i|S)=\frac{1}{n}\mathbb{E}\left(\sum _{i=1}^n{X}_i|\sum _{k=1}^n{X}_k\right)=\frac{1}{n}\sum _{k=1}^n{X}_k.$$

We can see that

$$\begin{array}{rl}\text{MSE}({\hat{\mu }}_{RB})& =\text{Var}({\hat{\mu }}_{RB})=\frac{{\sigma }^2}{n},\\ \text{MSE}(\hat{\mu })& =\text{Var}(\hat{\mu })={\sigma }^2.\end{array}$$

Related Notes

• Sufficiency
• UMVUE
• Lehmann-Scheffé Theorem