Solved – DOF of Natural Cubic Spline

degrees of freedomsplines

I am curious as to what the answer to the below question is? The question specifies a modeler has a cubic spline with knots at {10, 20, 30, 50}. They realize their model is overfitting at the ends of the distribution and wants to impose an additional constraint that the curve before the first knot and after the last knot are linear, then to calculate the DOF of the new model.

My thought process is that the answer should be 4 DOF for the new natural cubic spline model. Below is how I arrived at my answer:

The cubic spline has 8 DOF: 4+(4)(4)-(4)(3)=8 where the first 4 is for the intercept, X, X^2, X^3; then add 4 terms for earch knot; subtract out the 3 constraints at each knot to account for continuity, and the first and second derivatives to be zero.

Then to make the cubic spline with 4 knots a natural cubic spline, subtract (2)(2)=4 to get 4 DOF since the interval below the lowest knot and above the largest knot need to be made linear.

I’ve been told the correct answer is 6 DOF and I don’t understand how. I’ve been told it has something to do with interior knots vs boundary knots. Can someone shed some light as to how the answer could possibly be 6? Is my answer of 4 wrong?enter image description here

Best Answer

Let there be $K$ knots $\xi_1 < \dots < \xi_K$. I'll use $x_\min$ for the smallest observed $x$ value, and $x_\max$ is analogous.

I think the answer depends on whether or not the $\xi_j$ are assumed to be interior knots, and if we want the spline to change in $[x_\min, \xi_1]$ and $[x_\max, \xi_K]$ or not (and this also depends on whether or not $x_\min < \xi_1$ and/or $\xi_K < x_\max$).

If we only care about behavior between $\xi_1$ and $\xi_K$, then we could use the following truncated power basis: $$ h_j(x) = x^j, j=0,1,2,3 $$ and $$ h_j(x) = (x-\xi_{j-3})_+^3, \hspace{5mm} j = 4,\dots, K+3 $$ leading to $K+4$ DoF. This represents four DoF coming from the global cubic that we start with and one DoF being added for every knot we pass.

Restricting this to a natural spline means we'll constrain $\beta_2 = \beta_3 = 0$ which frees up 2 DoF, and we'll need the coefficients of $x^3$ and $x^2$ to be zero when every basis is active, so this further frees up two DoF meaning there are now $K$ DoF.

In interpolation problems and smoothing splines I think this is the right way to account for DoFs, because we have every point as a knot so there aren't separate boundary knots from $x_\min$ and $x_\max$.

But if we are always thinking of the $\xi_j$ as interior knots then we would have two more regions where the spline in changing. This is where setting $\xi_0 = x_\min$ and $\xi_{K+1} = x_\max$ makes sense and really we now have $K+2$ knots and therefore the full cubic spline has $K+6$ DoF and the natural spline has $K+2$.

In Figure 7.5 of Introduction to Statistical Learning (in the question you linked) we can tell that they are using the $K+2$ DoF version because the spline is nonlinear on all of $[x_\min, x_\max]$, rather than just on $[\xi_1, \xi_3]$.


To answer the exact question: I can't tell which formulation they want but I think $6$ is more likely correct.

For a spline that is not interpolating or smoothing, which you have here, I think having $\xi_1$ and $\xi_K$ as interior knots makes sense so the $K+2 = 6$ answer makes sense.

But in the exact wording of the question they say "the curve before the first knot and after the last knot will be linear". If they mean out of the four given knots, then that means an answer of $K=4$ is correct, but I'm guessing they worded this poorly and are thinking of $x_\min$ and $x_\max$ as the first and last knots respectively, so even though at face value this suggests an answer of $4$, with this boundary knot inclusion then again the answer is $6$.