Given $N$ data points $(x_i, y_i)$ with unique x-coordinates, one is asked to plot the least squares line. Then, a single point is asked to be shifted vertically. The conjecture that I've been trying to prove is that there are two possible scenarios for the new least squares line:
1) The point chosen is such that $x_k = \frac{1}{N}\sum x_i$. Then, the new least squares line is a translation of the former one.
2) Otherwise, the new least squares line is a rotation of the former one around a point on the former line that is the intersection of the former line with some vertical line described by $x=x^*_k$, where $x^*_k$ is unique for every point and can be calculated given only all of the x-coordinates $x_i$.
If the above is true, I would highly appreciate it if someone could point me towards a way to derive the rotation point for any given point $x_k$. I tried looking at this problem from the point of view of physics (where the line is being pulled by vertical springs), but had no progress here either.
EDIT: Following @gigo318 's directions, I ended up with the following formula for the x-coordinate of the rotation point, if point $k$ is perturbed:
$$x_k^* = \frac{x_k\sum x_i – \sum x_i^2}{Nx_k – \sum x_i}$$
If the denominator is 0, this means translation. As you can see, the rotation point is independent of y-coordinates and of the chosen value of $\epsilon$.
I am now curious if there is a physical interpretation for the rotation point. The translation case is more or less trivial – if you pull the rod (the least squares line) at its the center of mass (i.e. the point $\frac{\sum x_i}{N}$), the rod will not rotate and will translate instead. The intuition behind the rotation case is still not obvious to me.
Best Answer
First, I would recommend starting with the solution for the coefficients of the linear least squares regression:
$$A = (X^TX)^{-1}X^TY $$
Where: $$ A = \begin{pmatrix} m \\ b \\ \end{pmatrix} $$
m and b are the slope and intercept of the line, respectively.
$$ X = \begin{pmatrix} x_1 & 1 \\ x_2 & 1 \\ \cdots & \cdots \\ x_n & 1 \\ \end{pmatrix} $$
$$Y = \begin{pmatrix} y_1 \\ y_2 \\ \cdots \\ y_n \\ \end{pmatrix} $$
Then I would try to figure out how the coefficient matrix "A" would change for a change in one of the entries in Y, that is:
$$A' - A = (X^TX)^{-1}X^T(Y'-Y)$$
Where Y' is the original vector Y modified by a shift of some value, say $\epsilon$, in one of the entries:
$$Y' = \begin{pmatrix} y_1 \\ y_2 + \epsilon \\ \cdots \\ y_n \\ \end{pmatrix} $$
For Case 1, you want to show that that 'm' doesn't change for any value epsilon added to an entry in Y' (which you should find only happens if the x value associated to that entry is the average of all the x values).
For Case 2, you know that the slope of the line will change and hence the new line will intersect the old line. So then you need to calculate the slope and intercept of both the the old and new lines by by evaluating $A = (X^TX)^{-1}X^TY$ and $A' = (X^TX)^{-1}X^TY'$ and then determine the intersection point for these two lines.
Also, as a further hint, I would recommend explicitly evaluating $(X^TX)^{-1}$ ($X^TX$ is only a 2x2 matrix so this is relatively straightforward) and $X^TY$.