data analysisleast squaresoptimizationvectors
I have set of points scattered around the origin. How to find a vector, such that the average squared distance (perpendicular distance) from points to the vector is minimised?
For example, let the dimension be 2. So a vector can be written (from the origin) as $(a_1,a_2)$. Now take a point $(x_1, x_2)$. Drop a perpendicular line from the point to the vector and I want to find the length of that line segment.
There are a few ways to solve this problem.
Method 1.
Method 2.
Write an equation (in the form $f(x)=mx+b$) for the line passing through the given points $P_0$ and $P_1$.
Using the equation from 1, the coordinates for any point on the given line can be written in the form $(x, f(x))$.
Use the distance formula to write a formula for the distance between the given point $P$ and an arbitrary point $(x, f(x))$ on the line. This formula will be a function of $x$.
Find the minimum value of the function from 7. There are a few different ways to do this, too:
This is really two questions in one.
As regards the first question, it’s a basic geometric fact that the shortest distance from a point to a hyperplane (line in 2-D, plane in 3-D, &c) is along the perpendicular to the hyperplane. You can prove it to yourself by using calculus to minimize the distance between the fixed point and a point on the hyperplane, but it’s easier to use basic trigonometry:
If $d_\perp$ is the perpendicular distance from the origin to the pictured line, then the distance $d$ measured along a line that makes an angle $\theta$ with the perpendicular is $d={d_\perp\over\cos\theta}$, which is obviously minimized when $\cos\theta$ is maximized, i.e., when $\theta=0$. This holds in any number of dimensions.
As for the second part, one form of the equation for a hyperplane is $$\mathbf n\cdot\mathbf x=-d\tag{1}$$ where $\mathbf n$ is a vector (not necessarily of unit length) normal to the hyperplane. From above, we know that $\mathbf x$ will have minimal length when it is parallel to $\mathbf n$, i.e., when $\mathbf x=\lambda\mathbf n$ for some scalar $\lambda$. If we normalize $\mathbf n$ to get a unit vector, this scalar will be equal to the (signed) distance of the hyperplane to the origin. Substituting this back into the equation of the line: $$\mathbf n\cdot\lambda{\mathbf n\over\|\mathbf n\|}=\lambda{\mathbf n\cdot\mathbf n\over\|\mathbf n\|}=\lambda\|\mathbf n\|=-d$$ therefore $\lambda={-d\over\|\mathbf n\|}$. Equation (1) tells us that the dot product of $\mathbf n$ with any point on the hyperplane is equal to the constant value $-d$, so the distance of the hyperplane to the origin is ${\mathbf n\cdot\mathbf x\over\|\mathbf n\|}$, where $\mathbf x$ is any point that satisfies (1). Of course, $\mathbf n\cdot\mathbf x$ can also be expressed as $\mathbf n^T\mathbf x$, and if we set $\mathbf n=(w_1,w_2,w_3)^T$, we have your boundary equation (with $d=w_0$) and distance formula from the preceding question.
Observe that ${\mathbf n\over\|\mathbf n\|}\cdot\mathbf x=\|\mathbf x\|\cos\theta$, where $\theta$ is again the angle between the normal and $\mathbf x$, so this quantity can be understood to be the length of the projection of the vector from the origin to any point on the hyperplane onto the hyperplane’s normal, which is again simply $\|\mathbf x\|$ when $\theta=0$, i.e., the perpendicular distance to the hyperplane.
Best Answer
If I well understand the question, this is the problem of fitting with perpendicular offsets.
In case of 2D. : http://mathworld.wolfram.com/LeastSquaresFittingPerpendicularOffsets.html
In case of 3D. , see pages 2-12 in section "3D Linear Regression" of the paper : http://fr.scribd.com/doc/31477970/Regressions-et-trajectoires-3D