Regression Splines

1. Piecewise Polynomials

• Piecewise polynomials fit low-degree polynomials over different regions of $X$.

1.1 Knots

• Knots are the cut points that divide the range of $X$ into distinct regions.

Example (Piecewise cubic polynomial with one knot at $c$).

$$\{\begin{array}{ll}{y}_i={\beta }_0+{\beta }_1{x}_i+{\beta }_2{x}_i^2+{\beta }_3{x}_i^3+{ϵ}_i& \text{if }{x}_i<c,\\ y_i={\tilde{\beta }}_0+{\tilde{\beta }}_1{x}_i+{\tilde{\beta }}_2{x}_i^2+{\tilde{\beta }}_3{x}_i^3+{ϵ}_i& \text{if }{x}_i\ge c.\end{array}$$

• More knots bring more flexibility.
• A piecewise constant Step Functions are polynomials of degree $0$.

2. Constraints and Splines

• To avoid overfitting, we can constrain the piecewise polynomials to be continuous and smooth at the knots.

• Each additional constraint reduces one degree of freedom.

• If we constrain the piecewise polynomials, e.g. cubic, to be:

  • continuous at the knots;
  • continuous for the first derivative at the knots;
  • continuous for the second derivative at the knots;

  then we call it a cubic spline.

Example (Cubic spline). In this case, each region has 4 parameters, thus 8 parameters in total. We have 3 constraints: continuity, continuity of the first derivative, and continuity of the second derivative. Thus 3 degrees of freedom are reduced.

In general, if we have $K$ knots, the degree of freedom is $K+4$.

2.1 Degree-$d$ Spline

We define a degree-$d$ spline as:

• A piecewise polynomial of degree $d$.
• Constrained to be continuous up to the $(d-1)$th derivative at the knots.

3. The Spline Basis Representation

Spline basis representation helps with fitting the model under the constraints.

A cubic spline with $K$ knots can be represented as:

$$\begin{array}{rl}{y}_i& ={\beta }_0+{\beta }_1{b}_1(x_i)+{\beta }_2{b}_2(x_i)+\cdots +{\beta }_{K+3}{b}_{K+3}(x_i)+{ϵ}_i,\end{array}$$

where $b_j(x)$ are the basis functions.

3.1 Choice of Basis Functions

• The most common choice is the cubic spline with truncated power basis function per knot.

• The truncated power basis function per knot is defined as:

  $$h(x,\xi )=(x-\xi {)}_{+}^3=\{\begin{array}{ll}(x-\xi {)}^3& \text{if }x>\xi ,\\ 0& \text{if }x\le \xi ,\end{array}$$

  where $\xi$ is the knot.

• It can be easily proved that the truncated power basis function per knot remains continuous up to the second derivative at the knot.

• Thus to fit a data set with $K$ knots, we perform LSE with $K+4$ predictors, including the intercept: $X,X^2,X^3,h(X,{\xi }_1),h(X,{\xi }_2),\cdots ,h(X,{\xi }_K),{\beta }_0$.

3.2 Natural Splines

• Splines have high variance at the outer range of the predictors.
• Natural Spline is a regression spline with additional constraints: the function is linear at the boundary, i.e. in the region where $X$ is smaller than the smallest knot, or larger than the largest knot, the function is linear.
• Natural splines generally produce more stable estimates at the boundaries, with smaller variance.

4. Choosing the Number and Locations of the Knots

The regression spline is most flexible in regions.

4.1 Locating Knots

• One option is to place more knots where we feel the function might vary most rapidly, and to place fewer knots where it seems more stable.
• Another option is to place knots in a uniform fashion, i.e. specify the desired degrees of freedom of the spline, and then have software automatically place the corresponding number of knots at uniform quantiles of the data.

4.2 Choosing the Number of Knots

• One option is to try out different numbers of knots, and then choose the model with the smallest RSS or the largest $R^2$.
• Another option is to use cross-validation:
  1. Remove a portion of the data, say 10%.
  2. Fit a spline with a certain number of knots to the remaining data.
  3. Use the spline to make predictions for held-out portion.
  4. Repeat the above steps until each observation has been left out once.
  5. Compute the overall cross-validation error, e.g. MSE.

Note: Polynomial regression comparison

The source note has a heading for comparison with polynomial regression, but does not include the comparison content. It is left as a planned expansion.

Related Notes

• Step Functions
• Smoothing Splines