Consider the two following potential definitions of stability of a locally free coherent sheaf $E$.
(A) Every subbundle (i.e. locally free subsheaf) has strictly smaller slope.
(B) Every subsheaf has strictly smaller slope.
I claim that over a curve $X$ these two definitions are equivalent. Indeed, suppose (A) holds, and let $F\subset E$ be a subsheaf. Consider the sequence
$$0\to F\to E \to Q\to 0.$$
Here $Q$ may fail to be locally free; Put $G = Q/Q_{tors}$, which is then locally free since $X$ is a curve. Define $K$ by the sequence
$$0\to K \to E \to G\to 0,$$
and observe that $K$ is locally free of the same rank as $F$. Furthermore, $c_1(K) \geq c_1(F)$ since $c_1(Q_{tors})\geq 0$, so $\mu(F) \leq \mu(E)$.
Thus it is equivalent to only consider subbundles for stability, so long as we are working on a curve. Together with GAGA, this shows the two approaches (and any possible combination of definitions) are essentially the same.
The extensions of this form that admit a flat holomorphic connection are precisely the non-split ones.
The work of Hitchin, Donaldson, Corlette, and Simpson from the late 1980's shows that a bundle admits a flat holomorphic connection if and only if it has a Higgs field, i.e. a section $\phi \in H^0(K \otimes \mbox{End } V)$, making the Higgs pair $(V,\phi)$ semistable. See Hitchin's "The self-duality equations on a Riemann surface," for example. On an elliptic curve, $K \cong O$.
The classification of vector bundles on a smooth elliptic curve is rather easy and is accomplished in an early paper of Atiyah. Using this, it is not hard to show that every non-split extension of the form you state is isomorphic to $L \oplus L^{-1}$ for some $L \in \mbox{Pic}_0$, except on four lines in the two-dimensional extension space $H^1(O(-2p))$, where it is a non-split extension
$$0 \longrightarrow L \longrightarrow V \longrightarrow L \longrightarrow 0$$
for some $L$ with $L^2 \cong O$.
In either case, it is easy to find a Higgs field making the pair semistable: a diagonal Higgs field (i.e. an endomorphism preserving the splitting) in the case of $L \oplus L^{-1}$, and a nilpotent Higgs field (i.e. the composite map $V \to L \to V$ ) in the case of the non-split extension.
On the other hand, $O(p) \oplus O(-p)$ will not admit a Higgs field making it semistable, for $O(p)$ will always be an invariant, hence destabilizing, subbundle.
Regarding your question about what holonomies arise, let's first recall the answer for line bundles. The flat $C^\times$-connections, or representations $\pi_1 (E) \to C^\times$, are parametrized by $C^\times \times C^\times$. This is analytically, but not algebraically, isomorphic to the moduli space of flat $C^\times$ connections on $E$, which is a $C$-bundle over $\mbox{Pic}_0 E \cong E$. In particular, the projection to $E$ gives a holomorphic map $C^\times \times C^\times \to E$. If I remember correctly, this turns out to be nothing but $(w,z) \mapsto (\log w + \tau \log z)/(2 \pi i)$, where $E = C/\langle 1, \tau \rangle$. That tells you very explicitly which holonomies map to which line bundles. All this is a small fragment of the work of Simpson. I think it is spelled out explicitly in an expository paper by Goldman and Xia.
Now, the moduli space of flat $SL(2,C)$-connections on $E$, modulo gauge equivalence, is the space of two commuting elements of $SL(2,C)$, modulo conjugation. On a dense open set these elements are semisimple, and then (by a result of Borel and Steinberg) they lie in a common maximal torus $\cong C^\times$. They are conjugate in $G$ if and only if they are exchanged by an element of the Weyl group $W \cong Z/2$. So (ignoring the non-semisimple elements) the moduli space will be $(C^\times \times C^\times)/W$. The map from moduli of flat connections to moduli of bundles is then pretty clearly the $W$-quotient $(C^\times \times C^\times)/W \to E/W \cong P^1$ of the map of the previous paragraph. Here $P^1$ is parametrizing bundles of the form $L \oplus L^{-1}$. It is the projectivization of your vector space $H^1(O(-2p))$, since the bundle $V$ only depends on the extension class up to a scalar.
In other words, the holonomies that appear are conjugate to those in the fibers of this map, at least away from those four special points in $P^1$ where the non-split extensions with $L^2 \cong O$ appear. There there will be non-semisimple holonomies. For example, over the identity, there will be pairs of upper-triangular matrices with 1's on the diagonal. But perhaps I will leave this case as an exercise...
Best Answer
Rex: A holomorphic bundle doesn't always admit a flat connection. You need to assume further that each of its indecomposable pieces has degree 0. This is the result of Weil [J. Math. Pures Appl. (9) 17 (1938), 47--87] and Atiyah [Trans. Amer. Math. Soc. 85 (1957), 181–207].