The most common methods I've seen to find a line of best fit are Least Squares regression and median-median. Are there other good ways? Is there a way to minimize the absolute value difference and find a line of best fit that way? Or to find the distance straight to a line instead of the vertical distance to the line? Thoughts?
Solved – Other ways to find line of “best” fit
regression
Related Solutions
In the case you want to make a linear regression where Y values do not depend on X values, the method is to look for the solution of equation ny + p = x instead of the usual mx + p = y
The principle is the same, using the least square method. It solves the equation ax = b by computing a vector x that minimizes the Euclidean 2-norm || b - a x ||^2
I've made a script where I compare the two methods (ny + p = x versus mx + p = y) using 3 lines, a diagonal, a horizontal line and a vertical one. I'm displaying the Sum of the residuals (Sum of Res.) in the legend. You could see that for the diagonal line, the results are similar, but for the vertical and horizontal one, 1 method is giving good results while the other is giving is not.
Method 1: mx + p = y
The green line equation is not well fitting the values. The sum of residuals is very high compared to the good regression fits like diagonal or horizontal lines.
Method 2: ny + p = x
The red line equation is not well fitting the values. The sum of residuals is very high compared to the good regression fits like diagonal or vertical lines.
That image was taken from Karl Pearson's 1901 article "On Lines and Planes of Closest Fit to Systems of Points in Space" (p 566) which is the paper that introduced the concept of principal component analysis.
The introduction of the paper implies that regression is applicable to the case where the independent variables are exactly known. In this paper, Pearson is exploring the case of finding a line of best fit when the independent variables are corrupted by some error. He goes on to define the best-fit line as follows:
The best-fitting straight line for the system of points coincides in direction with the major axis of the correlation ellipse... . Physically the axes of the correlation-type ellipse are the directions of independent or uncorrelated variation. Hence the line of best fit is a direction of uncorrelated variation.
So the line of best fit in the figure corresponds to the direction of maximum uncorrelated variation, which is not necessarily the same as the regression line. You are correct: it is the line which minimizes the sum of the squares of the perpendicular distance between each point and the line.
I think the figure is just intended for visualization of this point. There may be no numerical reason that the regression line has a smaller slope than the best-fit line for this case. Section 7 of Pearson's paper gives some numerical examples which may be useful to you.
Best Answer
Minimizing the sum of absolute differences is quite common, as Nick Cox suggests, it's often called L1 regression or Least absolute deviations regression; it's also a specific case of quantile regression and many posts here relate to it.
http://en.wikipedia.org/wiki/Least_absolute_deviations
http://en.wikipedia.org/wiki/Quantile_regression
The orthogonal distance (what I assume you mean by "straight-line distance") would correspond to a particular case of Deming regressing, itself a particular case of the total least squares line, called orthogonal regression, which will give the line of the first principal component.
https://en.wikipedia.org/wiki/Principal_component_analysis
http://en.wikipedia.org/wiki/Deming_regression
http://en.wikipedia.org/wiki/Total_least_squares
There are many, many other lines that might be fitted; a couple of examples include Theil-Sen regression or more generally, robust regression, which includes many different techniques.
Some discussion of robust regression (including some comparison of Theil-Sen and L1 regression) is here.
There's some interesting discussion relating correlation measures to straight-line fits here