Solved – What are the Assumptions required in Regression Models, Ordinary Least Square, and Multiple Regression Models

least squaresmultiple regressionregression

I consulted various books and get confused about what the differences in Assumptions for Regression Models, Ordinary Least Square (OLS), and Multiple Regression Models are?

As I read more about it I just get more confused. Is there a rationally understandable way to explain this that would not be confusing?

Do these assumptions vary with respect to objective (maybe prediction or descriptive)?

What would be the reasonable list of assumptions for

  1. Linear Regression Models
  2. Simple Linear Regression Models
  3. Multiple Linear Regression Models
  4. Ordinary Least Square (OLS) Method?

I have also read the related questions on What are the Complete List…. and
Assumption Needed for Multiple linear Regression. But still it is not clear to me.

Best Answer

It would be difficult to be clearer than what has been said for the other posts. Nevertheless I will try to say something to the point that addresses the different assumptions that are needed for OLS and various other estimation techniques to be appropriate to use.

OLS estimation: This is applied in both simple linear and mutliple regression where the common assumptions are (1) the model is linear in the coefficients of the predictor with an additive random error term (2) the random error terms are (a) normally distributed with 0 mean and (b) a variance that doesn't change as the values of the predictor covariates (i.e. IVs) change, Note also that in this framework which applies in both simple and multiple regression the covariates are assumed to be known without any uncertainty in their given values. OLS can be used when either A) only (1) holds with 2(b) or B) both (1) and (2) hold.

If B) can be assumed OLS has some nice properties that make it attractive to use. (I) MINIMUM VARIANCE AMONG UNBIASED ESTIMATORS (II) MAXIMUM LIKELIHOOD (III) CONSISTENT AND ASYMPTOTICALLY NORMALITY AND EFFICIENCY UNDER CERTAIN REGULARITY CONDITIONS

Under B) OLS can be used for both estimation and predictions and both confidence and prediction intervals can be generated for the fitted values and predictions.

IF only A) holds we still have property (I) but not (II) or (III). If your objective is to fit the model and you don't need confidence or prediction interval for the repsonse given the covariate and you don't need confidence intervals for the regression parameters then OLS can be used under A). But you cannot test for significance of the coefficients in the model using the t tests that are often used nor can you apply the F test for overall model fit or the one for equality of variances. But the Gauss-Markov theorem tells you that property I still holds. However in case A) since (II) and (III) no longer hold other more robust estimation procedures may be better than least squares even though they are not unbiased. This is particularly true when the error distribution is heavytailed and you see outliers in the data. The least squares estimates are very sensitive to outliers.

What else can go wrong with using OLS?

Error variances not homogeneous means a weighted least squares method may be preferable to OLS.

High degree of collinearity among predictors means that either some predictors should be removed or another estimation procedure such as ridge regression should be used. The OLS estimated coefficients can be highly unstable when there is a high degree of multicollinearity.

If the covariates are observed with error (e.g. measurement error) then the model assumption that the covariates are given without error is violated. This is bad for OLS because the criteria minimizes the residuals in the direction of the response variable assuming no error to worry about in the direction of the covariates. This is called the error in variables problem and a solution that takes account of these errors in the covariate directions will do better. Error in variables (aka Deming) regression minimizes the sum of squared deviations in a direction that takes account of the ratios of these variances.

This is a little complicated because many assumptions are involved in these models and objectives play a role in deciding which assumptions are crucial for a given analysis. But if you focus on the properties one at a time to see the consequences of the violation of an assumption it might be less confusing.