Your question is not trivial and any proof of it that I know uses the Frobenius theorem which is a non-trivial analytic result. Let me give you an analogy which is in fact a particular case of what you're asking. Assume you have a one-form $\omega$ on some open ball $B$ in $\mathbb{R}^n$ and you want to determine a condition which guarantees that the path integral of $\omega$ depends only on the end points. Starting with such $\omega$, fix a point $p \in B$ and define a potential function $f \colon B \rightarrow \mathbb{R}$ by the formula
$$ f(x) = \int_p^x \omega $$
where the integral is done over any path which connects $p$ to $x$. Since $f$ is a smooth function, the second mixed partial derivatives of $f$ must commute and by calculating them, we see that this happens iff $d\omega = 0$. Hence, a necessary condition for the path independence of the integral is that $d\omega = 0$. This is a first order condition on $\omega$. However, by differentiating again we can get higher order conditions on $\omega$ which are also necessary. A priori, it is not clear at all that $d\omega = 0$ should be sufficient to obtain path independence but this is indeed the case which is the content of Poincare's lemma.
The situation with curvature is the same. If you have a rank $k$ vector bundle $E$ over $B$ with a connection, fix some trivialization $(e_1,\dots,e_k)$ and consider the associated connection $1$-form $\omega$ which is a lie-valued one-form. If the parallel transport is independent of the path, you can define a "potential" function $f \colon B \rightarrow \operatorname{GL}_k(\mathbb{R})$ by requiring that
$$ P_{\gamma,p,x}(e_i(p)) = f(x)_{i}^{j} e_j(x). $$
That is, $f(x)$ tells you the matrix you need to "multiply" the frame $(e_1(x),\dots,e_k(x))$ in order to get the parallel transport of the frame $(e_1(p),\dots,e_k(p))$ from $p$ to $x$ along some (any) path. By calculating "the second derivative" of $f$, you'll see that the curvature $d\omega + \omega \wedge \omega$ must vanish and by differentiating again, you'll get other, higher order, necessary conditions in terms of $\omega$ for the path-independence of the parallel transport. However, the condition $d\omega + \omega \wedge \omega = 0$ will turn out to be sufficient by the Frobenius theorem.
If $E$ is a rank $1$-bundle then $\omega$ is a $\mathbb{R}$-valued form and the curvature becomes $d\omega$ so everything boils down to the previous case (and indeed, the Poincare lemma can be proved using the Frobenius theorem).
Best Answer
[Added 7.10.2018:]
Due to public demand, let me specify explicitely that this is an answer to the following question:
[end added content]
Here is a proof: We pick a homotopy $h_s(t) = H(s,t)$ between two paths $h_0$ and $h_1$ in $M$ and consider the parallel transport along each of the curves $h_s$. We can then show that the resulting vector field along $H$ will be parallel, not only in the $t$-direction, but also in the $s$ direction. This step uses the vanishing curvature assumption. From this the claim follows, because a homotopy leaves the endpoints fixed. Therefore the vector field – being parallel in the $s$-direction at $t=1$ – must be constant there, if we only vary $s$. Here are the details.
Let $I= [0,1]$. Suppose we are given a homotopy $$H: I\times I \to M, \quad (t,s)\mapsto H(t,s) = h_s(t)$$ between two paths $h_0 = H(\cdot,0)$ and $h_1 = H(\cdot,1)$. Let $x = h_s(0)$ and $y=h_s(1)$ denote the two endpoints (which are assumed to be fixed during the homotopy). Let $v\in E$ with $p(v) = x$. We denote by $t\mapsto V(s,t)$ the parallel transport of $v$ along $t\mapsto h_s(t)$ for any $s\in I$, i.e. $V$ satisfies $$\nabla_{t} V = 0 \;\; \forall s,t\in I \quad \text{and} \quad V(s,0) = v\;\; \forall s\in I.$$
Vanishing curvature implies that $\nabla_t \nabla_s V = \nabla_s \nabla_t V$ for all $s,t\in I$. Thus, for any fixed $s\in I$, we have that $t\mapsto \nabla_s V(s,t)$ satisfies $\nabla_t \nabla_s V(s,t) = \nabla_s\nabla_t V(s,t) = 0$ and $\nabla_sV(s,0) = \nabla_s v = 0$. By uniqueness of parallel transport, it follows that $\nabla_s V = 0$. In particular, for $t=1$, we obtain that $\nabla_s V(s,1) = 0$. Since $h_s(1) = y$ is the constant path at $y$, it follows that $V(0,1) = V(1,1)$. So the parallel transport of $v$ along $h_0$ and $h_1$ agrees at their common endpoint. $\square$