A rotation is really what you think of in $\mathbb R^2$: Pick any orthonormal basis $e_1, e_2$ of $T_pM$, a rotation $R$ is given by
$$R(e_1) = \cos\theta e_1 + \sin\theta e_2, \ \ R(e_2)= -\sin\theta e_1 + \cos\theta e_2$$
First of all, every parallel transport preserves the metric, we have $\langle v, w\rangle = \langle P_\alpha v, P_\alpha w\rangle$ for all $v, w\in T_pM$. Also, we must have $\det P_\alpha = 1$, as $\det$ is a continuous function and the properties on metric imply that the determinant has to be $\pm 1$.
These two conditions imply that $P_\alpha$ has to be a rotational metric. But the question is., whether every rotations can be realized as a parallel transport. For example, if $M$ is the plane, it turns out that only the identity matrix can be realized as a parallel transport. This is due to the fact that the plane has Euclidean curvature.
Thus parallel transport is a measurement of the curvature of your surfaces. In higher dimensional case, the holomony group (groups of all parallel transport) has been classified, and different holomony group gives us quite different geometry.
From my understanding, by defining a connection on your manifold, you
provide a way to identify vectors at one point of the manifold with
vectors at another point on the manifold via parallel transporting the
vector.
Parallel transport depends on 1. a Riemannian manifold $(M, g)$, for example the round unit sphere; 2. a pair of points $p$ and $q$ of $M$, not necessarily distinct; 3. a piecewise-smooth path $\gamma:[0, 1] \to M$ starting at $p$ and ending at $q$, i.e., satisfying $p = \gamma(0)$ and $q = \gamma(1)$. (It's not essential that the parameter interval be the unit interval $[0, 1]$; an arbitrary closed, bounded interval will do.)
A Riemannian metric induces a Levi-Civita connection. In your example, the round sphere has a connection already, for which a tangent vector to a great circle arc remains tangent to the arc under parallel transport along the arc.
The example of the round sphere demonstrates dependence of parallel transport on $\gamma$. If $\gamma$ were a constant path, or a great circle arc traced forward and backward, parallel transport along $\gamma$ would be the identity map. For the spherical triangle $\gamma$ in your diagram, parallel transport along $\gamma$ is not the identity.
To emphasize (what seems to be) the underlying issue: The path $\gamma$ is a crucial piece of data in parallel transport; there's no well-defined notion of "parallel transport from $p$ to $q$" except in very special circumstances, such as parallel transport in a Euclidean plane, or on a flat torus. (Flatness—identically-vanishing Gaussian/sectional curvature—is necessary but not sufficient.)
Best Answer
If the circle you're transporting along is not a great circle, then you should not expect parallel transport all the way around the circle to be the identity.
Imagine a conic surface that touches your circle everywhere. If you cut up the cone, you can unroll it into part of a flat plane, but then you will find that the two ends of the circle that meet the cut will point in different directions after the cone is unfolded. Therefore, if you start parallel transporting a vector from the cut point an all the way around the circle, it will not point int he same direction relative to the circle locally when you reach the other side of the cut.
(Parallel transport in the plane keeps the vector actually parallel at all times, and since the cone is tangent to your spherical circle everywhere, parallel transport in the cone agrees with parallel transport in the sphere).