[Math] the difference between Linear Least Squares and Ordinary Least Squares

Question

My understanding is that Ordinary Least Squares (Usually taught in Statistics classes) uses the vertical distance only when minimizing error/residuals (see Wikipedia for Ordinary Least Squares) with a modeled line. On the other hand, Linear Least Squares (Usually taught in Linear Algebra classes) uses vertical and horizontal distance components when minimizing the error/residuals (See Wikipedia for Linear Least Squares) with the modeled line, in effect minimizing the "closest" distance.

Is this correct?

Normally would one expect to get the same estimation of parameters for a linear model?

Answer 1

Ordinary Least Squares and Linear Least Squares are the same in the sense they minimize the vertical distance between the plane estimated and the measurements.
Yet, they have different assumption about the data:

• Ordinary Least Squares (OLS) - In its stochastic model assumes IID white noise.
• Linear Least Squares (LLS) - Allows white noise with different parameters per sample or correlated noise (Namely can have the form of Weighted Least squares).

Total Least Squares and PCA are the ones which minimize the "Shortest" distance (Perpendicular distance).