Logistic Regression – Why Beta Cannot Be Estimated Using Least Squares?

generalized linear modelleast squareslogisticmathematical-statisticsmaximum likelihood

As the title says, why can you not estimate Beta (coefficients) in logistic regression with least-squares? I read in a book that it is not possible and that we use maximum likelihood instead, but I have found no motivation for this anywhere.

Best Answer

Linear regression is, as the name suggests, a linear model. This enables us to use linear algebra to find its parameters, this is called ordinary least squares. We cannot use OLS for generalized linear models like logistic regression, because they are non-linear. GLMs are defined in terms of a linear predictor

$$ \eta = \boldsymbol{X} \beta $$

that is passed through the link function $g$ to obtain the prediction

$$ E(Y\,|\,\boldsymbol{X} ) = \mu = g^{-1}(\eta) $$

Because the link function is non-linear, we cannot use linear algebra to find the parameters, but we need an optimization algorithm. Since generalized linear models are defined in terms of conditional distributions, we can fit them using maximum likelihood. On another hand, you can fit non-linear curves to the data by minimizing squared error, but because of the non-linearity, to do it you would use an optimization algorithm as well.