Change of Variables for Probability Distributions

1. Change of Variable Theorem

Setup. Assume we have input $z$ with distribution $\pi (z)$, a function $f:{\mathbb{R}}^n\to {\mathbb{R}}^n$, output $x=f(z)$, and $x$ has distribution $p(x)$.

Example (Linear transformation of a uniform variable).

$$z\sim U(0,1),x=f(z)=2z+1\Rightarrow x\sim U(1,3).$$

Given that

$$\int p(x)dx=1,\int \pi (z)dz=1,$$

we have

$$p(x)=\frac{1}{2}\pi (z).$$

2. General Derivation

• Given point $z^{\prime }$ from distribution $\pi (z^{\prime })$, point $x^{\prime }$ from distribution $p(x^{\prime })$, and known transformation $f$.
• Take a small volume $\Delta z$ around $z^{\prime }$ to get $(z^{\prime },z^{\prime }+\Delta z)$, and accordingly $(x^{\prime },x^{\prime }+\Delta x)$.
• Since $\Delta z$ and $\Delta x$ are small, we can assume $(z^{\prime },z^{\prime }+\Delta z)$ is uniformly distributed, and so is $(x^{\prime },x^{\prime }+\Delta x)$.
• Moreover, since $x$ is transformed from $z$, the probability mass of the small volume should be preserved:

$$p(x^{\prime })\Delta x=\pi (z^{\prime })\Delta z.$$

Thus,

$$\boxed{p(x^{\prime })=\pi (z^{\prime })\frac{\Delta z}{\Delta x}=\pi (z^{\prime })\left|\frac{\partial z}{\partial x}\right|}.$$

As long as $f$ is given, $\frac{\partial z}{\partial x}$ is fixed, so we can get $p(x)$ from $\pi (z)$.

3. Multidimensional Formula

Theorem (Change of variables formula).

$$\boxed{p(x)=\pi (z)\left|\det \left(\frac{\partial z}{\partial x}\right)\right|=\pi (z)\left|\det \left(J_{f^{-1}}\right)\right|}.$$

3.1 More Mathematical Form

Assume that we have a mapping rule $z=f_{\theta }(x)$ over random variables $x$ and $z$. It can be proved that

$$p_{\theta }(x)\mathrm{d}x=p(z)\mathrm{d}z.$$

Note: Intuition

Personally I would say that although the random variable has been transformed from one to another, in a grand view of an event happening, the shift in space would not affect the probabilities. Thus this equation holds. Li Hongyi’s explanation is that if we scale down to a very small area $\mathrm{d}x$ and $\mathrm{d}z$, then both approximately follow a uniform distribution, with the same total probability mass.

Thus we get the change of variables formula:

$$p_{\theta }(x)=p(f_{\theta }(x))\left|\frac{\partial f_{\theta }(x)}{\partial x}\right|,$$

where $f_{\theta }$ is required to be invertible and differentiable.

Related Notes

• Box-Muller Transformation
• Generating Continuous Random Variables