The spherical law of cosines is about spherical triangles with side lengths $a$, $b$, $c$ (measured along arcs of great circles), and angles $\alpha$, $\beta$, $\gamma$. Your $a$, in whatever units it is measured, is not such a side.
Instead of the triangle in your figure you should consider the spherical triangle with vertices $A=(\phi_1,\theta_1)$ ($(1^\circ,1^\circ)$ in your example), $B=(\phi_2,\theta_2)$ ($(3^\circ,3^\circ)$ in your example), and the north pole $D$. The sides $a$ and $b$ of this triangle (which are not the $a$, $b$ of yours) lie on meridians, i.e., great circles, and you know their lengths, namely $a={\pi\over2}-\theta_2$ and $b={\pi\over2}-\theta_1$. Furthermore you know the angle $\delta=\phi_2-\phi_1$ at $D$ (this is the "real" $a$ of yours!). These data suffice to compute the side $d$ by means of one of the spherical cosine laws. The formula in question is
$$\cos d=\cos a\ \cos b+\sin a\ \sin b\ \cos\delta\ .$$
Finally we express $a$, $b$, and $\delta$ by the given quantities $\phi_i$, $\theta_i$:
$$\cos d=\sin\theta_1\ \sin\theta_2+\cos\theta_1\ \cos\theta_2\ \cos(\phi_2-\phi_1)\ .$$
I'll tell you how to do this in a moment.
First, I'll tell you why it's a bad idea.
Your data, unless it comes from, say, an orbiting satellite with a huge velocity, consists of points whose distance is very small compared to the radius of the earth. The error introduced by treating lat/lon coordinates as $xy$-coords (after suitable scaling for the compression of longitude away from the equator) will generally be tiny compared to the error you make in simply applying the D-P algorithm. The exceptions are (1) the poles, (2) very fast motion, which probably should not be simplified because of the 3D analogue of "aliasing" in computer graphics, and (3) the international dateline. For cases 1 and 3, the best solution is probably to change to a different set of lat/long coords. For the dateline, for instance, you can add 180 to all longitudes, and then subtract 360 until the result is in the range -180 to 180. Most of your points will end up with "new longitudes" near zero. Once you've done the DP algorithm, you convert back.
For polar regions, you switch to a lat-lon system whose "poles" are two opposite points near what we call the equator, and whose "Greenwich meridian" is half of the thing we call the equator. This involves converting to rectangular coordinates, swapping y and z, and converting back.
OK. Now the "how to":
Given a lat-long point with latitude $u$ and longitude $v$ (which I'll write $[u, v]$) you can compute $xyz$ coordinates via the following:
\begin{align}
x &= \sin u \cos v \\
y&= \sin v\\
z &= \cos u \cos v
\end{align}
I'll write the result as $(x, y, z)$.
Your problem is that you have points $A, B, C$ in $[u, v]$ coordinates, and want to project $C$ onto the arc between $A$ and $B$, and find the length of the projection arc.
To do this, I'm going to talk about $A$, $B$, and $C$ as if they've been converted to $xyz$ triples already, to avoid new notation. I'll use the dot-product of two triples,
$$
(x_1, y_1, z_1) \cdot (x_2, y_2, z_2) = x_1 y_1 + x_2 y_2 + x_3 y_3
$$
and the "cross product," which I'll let you look up on Wikipedia for fear that I'll get a sign wrong.
OK.
a. Compute $n = A \times B$ (the cross product) and
$$
N = n / \sqrt{n \cdot n}
$$
Explanation: There's a plane passing through $A$, $B$, and the earth's center; the line from the earth's center to $N$ is perpendicular to this plane, and $N$ is on the surface of the earth.
b. Compute
$$
s = 90^\circ- |\arccos (C \cdot N)|.
$$
Explanation: this is the angular distance between a ray from the earth's center to $C$ and the plane described above.
You can also compute the "distance" between $A$ and $B$ as
$$
s' = \arccos(A \cdot B)
$$
Assuming that you're working in degrees, this will range from $0$ to $180$. If you want miles, you need to multiply this by the circumference of the earth, about 25,000 miles. (All of this assumes that the earth is spherical -- another source of error).
Anyhow, this gives you an alternative approach to the DP algorithm: simply use the formulas I've given you for $s$ and $s'$ as the "distances" that DP requires. They'll all be between 0 and 180 degrees (or $0$ and $\pi$ radians), but they'll be proportional (ignoring non-sphericity) to earth's-surface distances, which is all that you need.
And then be careful not to apply the algorithm in any situation in which a distance is greater than about, say, 15 degrees, because any assumptions of "flatness" in D-P is likely to be broken by the curvature of the earth at that scale.
Best Answer
The north/south distance is 1 nautical mile per minute of latitude. The east/west distance is $\cos \phi$ nautical mile per minute of longitude. So you have the total distance in nautical miles is $\sqrt{\Delta \phi^2+\cos^2 \phi \Delta \lambda}$ where the distance is in nautical miles and the angles are in minutes. Alternately, you can express it as $R\sqrt{\Delta \phi^2+\cos^2 \phi \Delta \lambda}, R$ the radius of the earth in your favorite units and angles in radians.