Delta Method

For linear transformations, expectation and variance of the transformed random variable are easy to infer. For nonlinear transformations, the distribution of the transformed random variable is harder to infer. The Delta Method estimates the distribution of a function of a random variable, and is widely used in the inference of MLE estimators.

1. Taylor Expansion

Theorem (Taylor expansion for univariate functions). For a function $g(x)$, if the $k$-th derivative of $g(x)$ exists around $x=a$, then for any constant $a$, the Taylor polynomial of order $r$ about $a$ is

$$T_r(x)=g(a)+g^{\prime }(a)(x-a)+\frac{g^{\prime \prime }(a)}{2!}(x-a{)}^2+\cdots +\frac{g^{(r)}(a)}{r!}(x-a{)}^r,$$

where $g^{(r)}(a)$ is the $r$-th derivative of $g(x)$ at $x=a$.

For the remainder term $R_r(x)$,

$$\underset{x\to a}{\lim }\frac{g(x)-T_r(x)}{(x-a{)}^r}=0.$$

Theorem (Taylor expansion for multivariate random functions). Let $\mathbf{T}=(T_1,T_2,\cdots ,T_k)$ be random variables with means $\mathbf{\mu }=({\mu }_1,{\mu }_2,\cdots ,{\mu }_k)$. Suppose $g(\mathbf{T})$ is differentiable, and is the function whose expectation and variance we want to estimate.

Define the derivatives of $g(\mathbf{T})$ about $\mathbf{\mu }$ in scalar form as

$$g_i^{\prime }(\mathbf{\mu })=\frac{\partial g(\mathbf{\mu })}{\partial {\mu }_i}.$$

Then the Taylor expansion of $g(\mathbf{T})$ about $\mathbf{\mu }$ is

$$g(\mathbf{T})\approx g(\mathbf{\mu })+\sum _{i=1}^k{g}_i^{\prime }(\mathbf{\mu })(T_i-{\mu }_i).$$

The expectation and variance of $g(\mathbf{T})$ can be estimated as

$$\begin{array}{rl}E[g(\mathbf{T})]& \approx g(\mathbf{\mu })+\sum _{i=1}^k{g}_i^{\prime }(\mathbf{\mu })E[T_i-{\mu }_i]=g(\mathbf{\mu }),\\ Var[g(\mathbf{T})]& \approx \mathbb{E}[(g(\mathbf{T})-g(\mathbf{\mu }){)}^2]\\ & \approx \mathbb{E}\left[{\left(\sum _{i=1}^k{g}_i^{\prime }(\mathbf{\mu })(T_i-{\mu }_i)\right)}^2\right]\\ & =\sum _{i=1}^k[g_i^{\prime }(\mathbf{\mu }){]}^{2}Var[T_i]+2\sum _{i>j}{g}_i^{\prime }(\mathbf{\mu })g_j^{\prime }(\mathbf{\mu })Cov(T_i,T_j).\end{array}$$

2. Univariate Delta Method

Theorem (Univariate Delta Method). Suppose we have a random variable $X$ with mean $\mu$ and variance ${\sigma }^2$, and we are interested in the function $Y=h(X)$. Expand $h(X)$ around $\mu$:

$$h(X)\approx h(\mu )+h^{\prime }(\mu )(X-\mu ),$$

where $h(\mu )$ and $h^{\prime }(\mu )$ are non-random values.

Then

$$\begin{array}{rl}\mathbb{E}(Y)& \approx h(\mu ),\\ \text{Var}(Y)& \approx h^{\prime }(\mu {)}^2\text{Var}(X),\end{array}$$

or equivalently,

$$\boxed{\mathbb{E}[h(X)]\approx h(\mathbb{E}[X])\text{Var}[h(X)]\approx h^{\prime }(\mathbb{E}[X]{)}^2\text{Var}[X]}.$$

Especially, let $h(X)=1/X$. Then

$$\begin{array}{rl}\mathbb{E}[1/X]& \approx \frac{1}{\mathbb{E}[X]}=\frac{1}{\mu },\\ \text{Var}[1/X]& \approx \frac{1}{\mathbb{E}[X{]}^4}\text{Var}[X]=\frac{{\sigma }^2}{{\mu }^4}.\end{array}$$

If $X\sim \mathcal{N}(\mu ,{\sigma }^2)$, then

$$h(X)\dot{\sim }\mathcal{N}(h(\mu ),h^{\prime }(\mu {)}^2{\sigma }^2).$$

3. Multivariate Delta Method

Theorem (Multivariate Delta Method). Let $\mathrm{X}=(X_1,X_2,\cdots ,X_d{)}^T$ be a random vector with mean $\mu =({\mu }_1,{\mu }_2,\cdots ,{\mu }_d{)}^T$ and covariance matrix $\Sigma$. Given a function $h(\mathrm{X}):{\mathbb{R}}^d\to \mathbb{R}$, we are interested in the mean and variance of $Y=h(\mathrm{X})$.

By Taylor expansion,

$$h(\mathrm{X})\approx h(\mu )+\nabla h(\mu {)}^T(X-\mu ),$$

where $\nabla h(\mu )$ is the gradient of $h$ at $\mu$:

$$\nabla h(\mu )={\left(\frac{\partial h}{\partial x_1}(\mu ),\frac{\partial h}{\partial x_2}(\mu ),\cdots ,\frac{\partial h}{\partial x_d}(\mu )\right)}^T.$$

Then

$$\begin{array}{rl}\mathbb{E}[Y]& \approx h(\mu ),\\ \text{Var}[Y]& \approx \nabla h(\mu {)}^T\Sigma \nabla h(\mu ).\end{array}$$

Equivalently,

$$\boxed{\mathbb{E}[h(\mathrm{X})]\approx h(\mu )\text{Var}[h(\mathrm{X})]\approx \nabla h(\mu {)}^T\Sigma \nabla h(\mu )}.$$

4. Delta Method for MLE Inference

For MLE inference, we know that the MLE estimator has asymptotic normality. Given MLE estimator $\hat{\beta }\in {\mathbb{R}}^d$, for large sample size,

$$\hat{\beta }\dot{\sim }\mathcal{N}(\beta ,I^{-1}(\beta )),$$

where $I(\beta )$ is the Fisher information matrix.

Then by the Delta Method, for $h(\hat{\beta }):{\mathbb{R}}^d\to \mathbb{R}$,

$$h(\hat{\beta })\dot{\sim }\mathcal{N}\left(h(\beta ),\nabla h(\beta {)}^T{I}^{-1}(\beta )\nabla h(\beta )\right).$$

Equivalently, the confidence interval for $h(\beta )$ is

$$h(\hat{\beta })\pm z_{\alpha /2}\sqrt{\nabla h(\beta {)}^T{I}^{-1}(\beta )\nabla h(\beta )}.$$

Related Notes

• Fisher Information
• Law of Large Numbers