I have a somewhat general question. Recently I have been learning about various ways to do regression analysis on data. I have learned about chebyshev polynomial regression which to my understanding is much more accurate than multi-linear regression.
Question: Why not always use Chebyshev polynomial regression?
Motivation for question: In several published research papers in statistics I see the researchers use multi-linear regression for, what I consider, small data sets (628 people for example).
I do not think the computational time is an issue due to many departments having access to supercomputers specifically for data analysis. So, what gives?
Best Answer
I think you have misunderstood the motivation for Chebyshev polynomials. Chebyshev polynomials are not used for statistical modelling at all --- their purpose is quite different. Chebyshev polynomials are a good and convenient solution to the classic numerical interpolation problem. Suppose I want to approximate a general function smooth $f(x)$ on a bounded interval with a polynomial of a given order. How can I choose the polynomial $g(x)$ to minimize the maximum difference $|f(x)-g(x)|$ on the interval? Then the Chebyshev polynomial on the interval is a good approximation to the optimal solution for $g(x)$. I gave a simple example of this in Figure 1 of Smyth (1998). The Chebyshev polynomial is defined by an interpolation problem. It interpolates the function $f(x)$ exactly at $n+1$ points, where $n$ is the order of the polynomial. The Chebyshev idea is to choose the $n+1$ points so to minimize the maximum discrepancy between $f(x)$ and $g(x)$ on the interval.
Chebyshev polynomials have little to do with statistics. Their purpose in numerical analysis is to derive well-behaved and efficiently computed approximations to smooth but complex mathematical functions. They are used to approximate frequently used mathematical functions that don't have algebraic closed forms. If you dig down into the numerical implement of functions like
pnormorqnormin R, you may well find a Chebyshev polynomial lurking somewhere deep down in the low-level C or Fortran code. To everyday users of the functions, the Chebyshev approximation will be invisible, and so it should be.The linear regression problem is quite different: different data, different scientific purpose, and different optimality function. In linear regression, we have response values $y_i$ and one or more covariates $x_i$. We have data values, not a smooth function. The covariate values $x_i$ are given, we do not get to choose them. We seek to fit the $y_i$ in a least squares sense rather than a mini-max sense. We are seeking to separate signal from noise rather than to reproduce every decimal point in $y$. We are looking for a parsimonious and interpretable representation of the signal that has a scientific meaning.
In the linear regression context, there is no interpolating Chebyshev polynomial because we don't observe $y$ at the $x$-values necessary to define it. It is possible to use standardized Chebyshev polynomials as basis functions for polynomial regression, but, in the least squares regression context, any polynomial basis gives rise to the same fitted values and residuals, just with different coefficients. Chebyshev polynomials have no advantage here --- defining instead polynomials that are orthogonal with respect to the observed $x_i$ is more usual and more useful (see my article referenced below).
All this has nothing to do with dataset size or with computing power. There are very efficient numerical implementations of multiple linear regression. Like many statisticians, I find myself fitting linear regressions every day for problems as small as $n=3$ or as large as $n=10^9$. And that's just on my PC.
References
Smyth, G. K. (1998). Polynomial approximation. In: Encyclopedia of Biostatistics, P. Armitage and T. Colton (eds.), Wiley, London, pp. 3425-3429. http://www.statsci.org/smyth/pubs/PolyApprox-Preprint.pdf [Very brief treatment of Chebyshev polynomials, but has the advantage of treating polynomial interpolation and polynomial regression in one article.]
Stewart, G. W. (1996). Afternotes on Numerical Analysis. SIAM, Philadelphia. [My favourite reference on polynomial interpolation!]