MATLAB: Polyfit vs mldivide upon inversion of arrays

inverselinearmldividepolyfit

OK, this is probably a stupid question, but I'm seeing some unexpected output when trying to do a linear fit to two sets of data. When I invert the x/y axes, the two fits are very different:
linearfitcoeff=polyfit(x,y,1)
linearfitcoeff =
    0.9600    6.1664
>> linearfitcoeff=polyfit(y,x,1)
linearfitcoeff =
    0.7659  177.7362
Data sets are large (over 75000 data points each). This is not what I would expect from a least-squares fit. Can someone explain this behaviour to me?
Update: mldivide works as expected, but I'm still curious about why polyfit doesn't
Thanks,
Matthew

Best Answer

Sigh. This is in fact, EXTREMELY common. Exactly as expected. If there is any noise at all, swapping x and y will NOT produce a fit with an inverse slope. NOT. NOT. In fact, you will expect to see slope estimates with a bias when you swap the variables.
Lets make up some data.
x = rand(20,1);
y = x + randn(size(x))/5;
polyfit(x,y,1)
ans =
       1.1032     -0.10593
polyfit(y,x,1)
ans =
      0.67926       0.1779
So the first fit had a slope of 1.1. The second fit, if the slope was the inverse, would be roughly 0.9. But it was around 0.68.
Hmm. WHY? Why does this make complete sense? Why is it exactly the expected result? Here is the linear fit, essentially polyfit(x,y,1).
plot(x,y,'o')
The reason is leverage.
Look at the points I generated, completely at random. The errors were purely in y.
Leverage tells us that points near the center of the linear fit have very little impact on the estimate of the slope. It is the points near the extremes that sway the fit one way or another.
Now, suppose I have two points near the very end of the curve, at the bottom. To one of those points, I add some noise that is +delta (for some fixed delta.) To the other point, I subtract delta. I'll add two points in red so you can see the idea.
Since I added the same absolute noise, +delta and -delta to each y value, they will have relatively little influence on the estimated slope, when fit as polyfit(x,y,1).
Now, imagine I were to swap the axes and perform a straight line fit?
plot(y,x,'bo')
hold on
plot(ye,xe,'rs')
Look at the red point where I added delta. That point is now near the very middle of the data.
polyfit([y;ye'],[x;xe'],1)
ans =
      0.63971       0.1755
Note that I added two points to my data fit, one high in y, the other low, and I got a result with a slope that was biased low from the previous fit in that direction.
What did I say before? Points near the middle have no impact on the slope. The point with noise that was low however is now a very high leverage point. It has a great deal of impact on the slope.
So what does this tell us? When you do a linear fit with data that has noise in y, then swap the axes and do another linear fit, so now the noise is in the wrong variable, it tends to produce a biased estimate of the slope. The new slope will tend to be biased LOW compared to the inverse of the original slope, because the high leverage points will drag the curve to have a lower slope than if you had fit the line the normal way.
So you saw EXACTLY what you should have expected to see. No surprise at all.
Of course, if you had no noise at all, then the line will be perfect, with the points falling exactly on the line. So no difference would now be expected. As the noise variance increases, the expected bias in the slope will grow, in a very predictable way.
Related Question