Your question is not trivial and any proof of it that I know uses the Frobenius theorem which is a non-trivial analytic result. Let me give you an analogy which is in fact a particular case of what you're asking. Assume you have a one-form $\omega$ on some open ball $B$ in $\mathbb{R}^n$ and you want to determine a condition which guarantees that the path integral of $\omega$ depends only on the end points. Starting with such $\omega$, fix a point $p \in B$ and define a potential function $f \colon B \rightarrow \mathbb{R}$ by the formula
$$ f(x) = \int_p^x \omega $$
where the integral is done over any path which connects $p$ to $x$. Since $f$ is a smooth function, the second mixed partial derivatives of $f$ must commute and by calculating them, we see that this happens iff $d\omega = 0$. Hence, a necessary condition for the path independence of the integral is that $d\omega = 0$. This is a first order condition on $\omega$. However, by differentiating again we can get higher order conditions on $\omega$ which are also necessary. A priori, it is not clear at all that $d\omega = 0$ should be sufficient to obtain path independence but this is indeed the case which is the content of Poincare's lemma.
The situation with curvature is the same. If you have a rank $k$ vector bundle $E$ over $B$ with a connection, fix some trivialization $(e_1,\dots,e_k)$ and consider the associated connection $1$-form $\omega$ which is a lie-valued one-form. If the parallel transport is independent of the path, you can define a "potential" function $f \colon B \rightarrow \operatorname{GL}_k(\mathbb{R})$ by requiring that
$$ P_{\gamma,p,x}(e_i(p)) = f(x)_{i}^{j} e_j(x). $$
That is, $f(x)$ tells you the matrix you need to "multiply" the frame $(e_1(x),\dots,e_k(x))$ in order to get the parallel transport of the frame $(e_1(p),\dots,e_k(p))$ from $p$ to $x$ along some (any) path. By calculating "the second derivative" of $f$, you'll see that the curvature $d\omega + \omega \wedge \omega$ must vanish and by differentiating again, you'll get other, higher order, necessary conditions in terms of $\omega$ for the path-independence of the parallel transport. However, the condition $d\omega + \omega \wedge \omega = 0$ will turn out to be sufficient by the Frobenius theorem.
If $E$ is a rank $1$-bundle then $\omega$ is a $\mathbb{R}$-valued form and the curvature becomes $d\omega$ so everything boils down to the previous case (and indeed, the Poincare lemma can be proved using the Frobenius theorem).
Best Answer
Not in general. Such map is actually called holonomy. See holonomy.