[Math] get a domain error for the following computation using law of cosines

trigonometry

Following are two points. lat1, lon1 represent one point and lat2, lon2 represent another point.

lat1, lon1 = 37.317233, -121.94331
lat2, lon2 = 37.317233, -121.94331

When I use the following formula (law of cosines) I get a math domain error

sine_part = sin(lat1_in_radians) * sin(lat2_in_radians)
distance = RADIUS_OF_EARTH_IN_METRES * acos(sine_part + cos(lat1_in_radians) * cos(lat2_in_radians) * cos(lon2_in_radians - lon1_in_radians))

where degrees to radians is converted by decimal_degree / 57.2957795

The weird thing is, the following points work as expected though. I get 0 as my distance.

lat1 = 54.11102, lon1 = -111.12345
lat2 = 54.11102, lon2 = -111.12345

I do understand that the values resulted by applying one of these trignometric functions is not in the interval it could work with. However I would like to know why this error happens in more detail from one of the experts here please.

I'm hoping I haven't done a school boy error, however I have forgotten my school maths so I won't be surprised if I have missed a trivial point!

Best Answer

This should fix your code:

sine_part = sin(lat1_in_radians) * sin(lat2_in_radians)
temp = sine_part + cos(lat1_in_radians) * cos(lat2_in_radians) * cos(lon2_in_radians - lon1_in_radians)
if temp > 1:
    temp = 1
if temp < -1:
    temp = -1
distance = RADIUS_OF_EARTH_IN_METRES * acos(temp)

As everyone is telling you in the comments, because of floating point errors, temp, which should be between $-1$ and $1$, might be fractionally outside of this range.

(By the way, I don't know what programming language you are using. But I think the code I wrote is self explanatory.)

Related Question