Solved – How to chose the order for polynomial regression

polynomialregression

How can we know which degree polynomial is the best fir for a data set composed of one predictor and one variable? And how can we evaluate them?

I have developed the linear regression and then went up to the third polynomial degree, but I just need to make how to assess the goodness of fit?

Best Answer

This question can be generalized for selecting any machine learning algorithm hyper-parameters. For example, number of clusters in K-means, number of Hidden unit in neural networks, etc.

At very high level, there are two ways (not mutually exclusive, in fact combining two ways would be ideal.): Data driven and knowledge driven.

Data driven means using data to figure out which one is the best. We usually have training set and testing set. There are some other variations, such as adding one additional validation data set, run repeated cross validation etc. But the overall idea is and pick the best one in testing set, and we can make sure testing set is very close to production data.
Knowledge driven means using "domain knowledge" to make the decision on parameter tuning. For example, we are fitting some data from some trajectory data and we know our data from physics would generally follow a parabola trend, not a 5th order polynomial curve. Then we would like pick the 2nd order polynomial to fit. In addition, if we know our data is periodic, we may choose Fourier expansion on the data instead of polynomials. See this post What's wrong to fit periodic data with polynomials?

In sum, if we have a lot of data, and can make sure we have a fair representation to production data in testing set. Then data driven would be good. On the other hand, if we have lots of domain knowledge about the relationship between input and output, then knowledge driven is good. The ideal case would be combining two: know the relationship in data and testing it carefully using a good testing set.

Related Solutions

Solved – Interpretation of polynomial regression output in R

Your "first guess" can't be evaluated just from this output. You'd have to look at the models without the cubic term to decide.

What you can say is that, in the model with the cubic term, the linear and quadratic terms are not significant. Still, they should almost surely be included due to the "hierarchy principle".

The non-significance of the lower order terms is not, in itself, a bad thing.

As to how to interpret the model - since you have only one independent variable, you can easily make a graph with the IV on the x axis and the DV on the y axis. I think this happens automatically as one of the plots of plot(model) but, if not, you can do it with

m1 <- lm(deviance ~ poly(myDF$distance,3,raw=T))
    plot(x = myDF$distance, y = m1$fitted.values)

You can also get the fitted value for any distance by plugging it into the regression equation (or using predict with newdata).

Solved – Regression: find the best degree of polynomial with the best regularization parameter

I would treat this as a standard cross-validation task where we optimise both the degree of the polynomial $k$ as well as the associated regularisation parameter $\lambda$ at the same time. Hyper-parameter optimisation procedures do this all the time.

An immediate example is when optimising SVM where the degree of the polynomial kernel $k$ is optimised along-side the regularization parameter $\lambda$. There is a rather large literature on the matter; Wainer & Cawley (2017) Empirical evaluation of resampling procedures for optimising SVM hyperparameters is relatively concise recent work, I found it very readable. Chapelle et al. (2002) Choosing Multiple Parameters for Support Vector Machines offers a more formal treatment if you want to explore this further. (Sometimes the regularisation parameter $\lambda$ is denoted by $C$, the inverse of it.)

Regarding the parameter search routine: Aside standard grid-search it is probably worth looking into Bayesian Optimisation approach. CV.SE has a great thread on the matter: Optimization when Cost Function Slow to Evaluate where the main mechanics of Bayesian Optimisation are presented. Particular for the case here, we will effectively fit a two-dimensional Gaussian Process against the parameters $\lambda$ and $k$.

Two final points:

$k$ is discrete. A quick and dirty solution is to just "round/floor/ceil" the associated estimate. That works but it can be on occasion misleading. There is some very recent work on the subject (e.g. Garrido-Merchán & Hernández-Lobato (2020) Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes, Luong et al. (2019) Bayesian Optimization with Discrete Variables).
The final estimate from BOpt or any other hyper-parameters grid-/random- search procedure will probably not be the MLE of that linear model. That is not the end of the world but if we want to use certain follow-up statistical procedures that assume MLE it would be reasonable to make one final optimisation step with $k$ being fixed and optimising for $\lambda$ only.

Best Answer

Related Solutions

Solved – Interpretation of polynomial regression output in R

Solved – Regression: find the best degree of polynomial with the best regularization parameter

Related Question