Using the Pythagorean formula on positions given in latitude and longitude makes as little sense as, say, computing the area of a circle using the formula for a square: although it produces a number, there is no reason to suppose it ought to work.
Although at small scales any smooth surface looks like a plane, the accuracy of the Pythagorean formula depends on the coordinates used. When those coordinates are latitude and longitude on a sphere (or ellipsoid), we can expect that
Distances along lines of longitude will be reasonably accurate.
Distances along the Equator will be reasonably accurate.
All other distances will be erroneous, in rough proportion to the differences in latitude and longitude.
The error depends on the start and end point of the distance calculations. However, because both the sphere and ellipsoid have a circular symmetry around the axis, the error depends only on the difference of the longitudes, so to study this error we might as well take the point of origin to be at the Prime Meridian. Because both the sphere and ellipsoid are symmetric under a north-south reflection, we only need to study points of origin in the southern hemisphere. For any such point we may draw a contour map of the relative error, equal to [Pythagorean calculation] / [True distance].
The Pythagorean formula, using the mean radius of the earth, is
Pythagorean distance = 6371000. * Sqrt[dx^2 + dy^2]] * pi / 180 meters
where dx is the difference in longitudes and dy is the difference in latitudes, both in degrees. (The difference in longitude values is reduced modulo 360 to give the correct value of dx when crossing the antimeridian; not doing so would introduce artificially large errors that tell us nothing about the Pythagorean formula itself.)
The following plots show the relative error compared to the correct distance on the WGS 84 ellipsoid for latitudes from -70 to 0 in increments of 10 degrees. The horizontal coordinate is the difference in longitudes and the vertical coordinate is the latitude of the destination. Light regions have relatively small error: the contour lines are at 1, 1.01, 1.02, 1.05, 1.1, 1.2, 1.5, 2, etc. (The pure white areas in the corners are places where the error goes beyond the range of these contours.) The red dots show the point of origin.
The vertical white bands testify to the correctness of expectation (1): Pythagorean distances are accurate when there is a small difference in longitudes. The horizontal white bands at low latitudes confirm expectation (2): near the Equator, horizontal distances are reasonably accurate. Otherwise, as witnessed by the extensive darker regions, at all other distances the Pythagorean formula is bad.
We can make quantitative estimates of the maximum error attained for pairs of nearby points (within, say, a few hundred kilometers of each other). Scale--using an appropriate value for the radius--is true along the meridian but along a circle of latitude it errs approximately by the secant of the latitude. For example, at a latitude of 40 degrees the secant is 1.31, implying the Pythagorean formula will give distances about 31% too large in the east-west direction. (This is evident in the upper right contour plot, for a point of origin at -40 degrees latitude, where the region immediately east-west of the red dot lies between the 1.2 and 1.5 contours.) Short distances in all other directions will be too large by some amount between 0% and 31%; longer distances may err by even more (as the contour plots show).
The radius, r, of the small circle joining all points at latitude, φ is
r = R cos φ
where R is the radius of the sphere. That simplifies to
r = cos φ
if we assume a "unit sphere" (R = 1) for convenience.
--------------------- A/D
| r φ /
| /
| /
| /
|a /
|x /
|i /
|s /
| / R
| /
| /
| /
| /
| /
| / ("side" view)
| /
| /
| /
|/ φ equatorial radius
-----------------------------------------------
The chord length of a straight line, AD, joining two points on the same latitude is
AD = 2 r sin dλ/2
where dλ is the difference in longitude of A and D. Thus
AD = 2 R cos φ sin dλ/2
or
AD = 2 cos φ sin dλ/2
if R = 1
A-----------------D
\ | /
\ | /
\ | /
\ | / r
\ | /
\ dλ /
\ /
\ /
("top" view)
Best Answer
This is terrible code for general-purpose use because it can give erroneous results or even fail altogether for short distances. Use the Haversine Formula instead.
(The formula on which your code is based converts two points on the sphere (not an ellipse) into their 3D Cartesian coordinates (xa,ya,za) and (xb,yb,zb) on the unit sphere and forms their dot product, which will equal the cosine of the angle between them. The ACos function returns that angle, which when scaled by the earth's radius will estimate the distance. The problem is that the cosine of a small angle, say of size 'e' in radians, differs from 1 by an amount close to e^2/2. This disappears into the floating point error cloud when e is smaller than the square root of twice the floating point precision. If you're computing in single precision this means values of e less than 0.001 -- about one kilometer -- will be confused with zero! In double precision the cutoff is around e = 10^-8, but by the time e = 10^-4 or so (about 10 meters) you potentially can lose so much precision that you need to worry, unless the implementation of the ACos function is unusually accurate (e.g., has some high-precision internal computations built in)).