For the first question I think you have some misunderstandings. Firstly Chern classes are defined for complex vector bundles - or equivalently principal $U(n)$-bundles. If $G$ is an arbitrary compact Lie group then the Chern classes of a principal $G$-bundle will be understood in terms of some representation $\rho:G\rightarrow U(n)$ (these always exist). For instance when $G=SU(n)$ the representation $\rho$ is the canonical inclusion.
Now if you are given a principal $G$-bundle $p:P\rightarrow X$ over a suitable space $X$ for some topological group $G$, and a representation $\rho:G\rightarrow U(n)$ then you can form the associated vector bundle $P\times_\rho\mathbb{C}^n\rightarrow X$ with total space
$P\times_\rho\mathbb{C}^n=P\times \mathbb{C}^n/[(e,v)\sim(eg,\rho(g)^{-1}v)]$
where $e\in P$, $v\in\mathbb{C}^n$ and $g\in G$. You can now ask for characteristic classes of this vector bundle, and in particular its Chern classes (and there are an infinte number of Chern classes, not just two).
Note that the Chern classes of this bundle are not the characteristic classes of the principal bundle $P$, but are the characteristic classes of some principal $U(n)$-bundle $Q\rightarrow X$ that admits a $(G,\rho)$-reduction of structure to $P$.
The principal bundle $P$ will have its own set of characteristic classes deriving from the cohomology of the classifying space $BG$ of $G$.
Note in particular that the vector bundle $P\times_\rho\mathbb{C}^n$ is not the adjoint bundle $ad(P)$, which is another vector bundle whose characteristic classes have even less to do with those of $P$. Consider for instance the canonical line bundle $\eta$ over $\mathbb{C}P^1=S^2$. Since $U(1)$ is abelian we have that $ad(\eta)\cong S^2\times \mathbb{R}$ is trivial (recall that the Lie algebra of $U(1)$ is $i\mathbb{R}$). However $\eta$ itself is not trivial and is in fact associated to the Hopf $S^1$-bundle $S^3\rightarrow S^2$ which has first Chern class $1\in H^2(S^2)$.
In summary it makes equal sense to ask for the characteristic classes of a principal bundle or the characteristic classes of a vector bundle with structure group $G$ if the representation $\rho$ is understood.
As for the second question, instantons cannot have a Chern class. Instantons are generally understood as certain field configurations. Given for instance as a suitable connection on a principal of vector bundle. I think what you are referring to is the instanton charge, which in the case above is defined to be the Chern class of the bundle to which the instanton is related. As I understand it a $U(1)$-instanton can not have non-zero second Chern class. It can have non-zero first Chern class. In fact for rank $N=1$ we have $c_1=n$ and $c_2=0$.
*Edit: Added to clarify comments.
I wasn't quite clear above. In the following I'll limit attention of $U(n)$-bundles. There is a one-to-one correspondence between (complex) vector bundles over $X$ and $U(n)$-bundles over $X$. This happens in one direction through the construction I gave above, and in the other by taking the frame bundle of a given vector bundle.
Thus, given a complex vector bundle $E\rightarrow X$ of rank $n$ we have a principal $U(n)$-bundle $Fr(E)\rightarrow X$ and an isomorphism of vector bundles $Fr(E)\times_{U(n)}\mathbb{C}^n\cong E$ over $X$. We can then take a connection on either bundle and ask for the Chern classes it gives. To see that the classes defined by each are the same recall firstly that a connection on a principal bundle induces one on any associated bundle. So a connection $A$ on the principal bundle $Fr(E)$ induces one, $\nabla_A$, on the vector bundle $Fr(E)\times_{U(n)}\mathbb{C}^n\cong E$. Moreover it is seen that both connections $A$ and $\nabla_A$ yield identical curvature 2-forms. Thus
$c_i(E)^{\nabla_A}=c_i(Fr(E))^A\in H^*(X)$
where we add a superscript to denote which connection we use to define the classes.
Now recall that the Chern classes are independent of the choice of connection. It follows that any other connection $A'$ on $Fr(E)$ yields identical Chern classes
$c_i(Fr(E))^A=c_i(Fr(E))^{A'}\in H^*(X)$.
By the same reasoning any other connection $\nabla'$ on your original vector bundle $E$ yields identical Chern classes
$c_i(E)^{\nabla'}=c_i(E)^{\nabla_A}\in H^*(X)$.
The point is that it doesn't matter which bundle $E$ or $Fr(E)$ we use to define the Chern classes: they are the same. The one-to-one correspondence between vector bundles and principal bundles carries over to their characteristic classes and you can consider the Chern classes to 'belong' to either bundle.
As for your comments regarding the Yang-Mills functional. Firstly I believe what you say about it being bounded by the second Chern class is only relevant to Yang-Mills theory over a 4-dimensional base. It is certainly not true for Yang-Mills on a surface, or 8-manifold, say. As for what bundle, it is the one you are working with, which should be granted by deeper context. By the above, it does not matter it is a vector bundle or the corresponding principal bundle.
And if you are talking about some $G$-bundle then yes, there must be an implicit unitary representation. The Chern classes are the characteristic classes belonging to $U(n)$. They can be defined as certain cohomology classes in the cohomology of the classifying space $BU(n)$. You cannot ask for the Chern classes of a $G_2$-bundle, say, which will have its own set of characteristic classes. If you are working with a vector bundle which has a $G$-structure then this $G$-structure provides a unitary representation for $G$.
*Edit 2: Clarification of further comments.
The Chern classes may be defined as certain classes in the cohomology of the classifying space $BU(n)$ of $U(n)$. The integral cohomology of $BU(n)$ is the polynomial ring $H^*(BU(n))\cong \mathbb{Z}[c_1,\dots c_n]$ on classes $c_i\in H^{2i}(BU(n))$. The classes $c_i$ are the universal Chern classes: the Chern classes belonging to the universal $U(n)$-bundle $EU(n)\rightarrow BU(n)$ with contractible total space (you can think of $BU(n)$ as the Grassmanian of $n$-planes in $\mathbb{C}^\infty$).
Then if $f:X\rightarrow BU(n)$ is a suitable classifying map for the bundle $E\rightarrow X$ (principal $U(n)$ or complex vector bundle, we have already decided these are the same things in a certain sense), in that $f^*EU(n)\cong E$, you can define the Chern classes of $E$ to be
$c_i(E)=f^*(c_i)\in H^{2i}(X)$.
This is the point of view in algebraic topology, and its truly quite marvelous that this same information can be extracted by studying the geometric data contained in a connection on $E$.
It may have been misleading when I said that there were an infinte number of Chern classes. On the one hand it's true if you consider the infinite unitary group which has $H^*(BU(\infty))\cong\mathbb{Z}[c_1,\dots,c_n,\dots]$, and you can certainly embed a rank $n$ vector bundle $E$ into the rank $n+1$ vector bundle $E\oplus\epsilon^1$ and ask for its Chern classes. However whilst you can take $n$ all the way to infinity in this manner, there will only be at most $n$ non-trivial Chern classes for a rank $n$ bundle. There are an infinite number you can ask for, but only $n$ interesting ones.
Now more generally you may be interested not in the individual Chern classes but in certain polynomial combinations of them, and there are indeed an infinite number of these you could potentially ask for.
Now the final point is that over a 4-manifold $X$ there will only be two potentially non-trivial Chern classes. This is simply because $H^*(X)$ vanishes in dimensions greater than $4$. Thus we have $c_1(E)\in H^2(X)$ and $c_2(E)\in H^4(X)$ and anything else lives in higher degrees and is thus trivial.
An application of this? If $E\rightarrow X$ is a rank $n$ complex vector bundle over a $4$-manifold with $n>2$, then there is a rank $2$ vector bundle $E'\rightarrow X$ and a bundle morphism $E\cong E'\oplus\epsilon^{n-2}$.
Best Answer
Hey Thiago:D I really like your question; I'll illustrate my thoughts in some detail so my apologies if I say lots of really obvious things.
Fix a scheme $X$, a sheaf of groups $G$ on $X$ and a $G$-torsor $P$; suppose we work in the Zariski topology for concreteness (as in your case with $\text{GL}_n$ torsors anyway, although I'm rather sure this generally works) and let $Y = \bigcup_i U_i$ be an open cover of $Y\subset X$ such that $P_{\mid U_i}$ is isomorphic to the trivial left $G$-torsor $G_{\mid U_i} \xrightarrow{\phi_i} P_{\mid U_i}$. I first think it's worth noting how $\text{Aut}_G(P)$ is a sheaf of $G$-sets: given any automorphism of $G$-torsors $\alpha : P_{\mid Y} \to P_{\mid Y}$, we have that $\phi_i\alpha_{\mid U_i}\phi_i^{-1}: G_{\mid U_i}\to G_{\mid U_i}$ is given by right multiplication by some element $a_i \in G(U_i)$ and it's rather tempting, at least for me, to define the action on $\alpha \in \text{Aut}_G(P)(Y)$ by an element $g\in G(Y)$ in imposing $\phi_i(g\cdot \alpha)_{\mid U_i}\phi_i^{-1}$ to be right multiplication by $a_i\cdot g^{-1}$, but of course there are gluing problems... we can examine them: if we set $U_{i,j}:= U_i\cap U_j$ then we have a commutative diagram $$ \require{AMScd} \begin{CD} G_{\mid U_{i,j}} @>{\cdot (a_i)_{\mid U_{i,j}}}>> G_{\mid U_{i,j}}\\ @V{(\phi_i)_{\mid U_{i,j}}}VV @VV{(\phi_i)_{\mid U_{i,j}}}V \\ P_{\mid U_{i,j}} @>{\alpha_{\mid U_{i,j}}}>> P_{\mid U_{i,j}}\\ @A{(\phi_j)_{\mid U_{i,j}}}AA @AA{(\phi_j)_{\mid U_{i,j}}}A \\ G_{\mid U_{i,j}} @>{\cdot (a_j)_{\mid U_{i,j}}}>> G_{\mid U_{i,j}} \end{CD} $$ which reads $$ a_{i} = (\phi_j^{-1}(\phi_i(1_G))\cdot a_{j}\cdot \phi_i^{-1}(\phi_j(1_G)) $$ and thus substituting $a_i,a_j$ for $a_i\cdot g^{-1},a_j\cdot g^{-1}$ might potentially hinder this equality, preventing us from getting a new automorphism $g\cdot \alpha : P_{\mid Y} \to P_{\mid Y}$. If instead we try replacing $a_i$ and $a_j$ with $a_i^g,a_j^g$ (where I use the group theorist's notation $x^y := yxy^{-1}$) we get an equality $$ a_i^g = (\phi_j^{-1}(\phi_i(1_G))^g\cdot a_{j}^g \cdot (\phi_i^{-1}(\phi_j(1_G)))^g $$ and the sweet thing is that we can replace the trivialisation $\phi_i$ with $\phi_i\circ(\cdot g)$ (which of course still provides an isomorphism between $G_{\mid U_i}$ and $P_{\mid U_i}$) thus defining an action of $G(Y)$ on $\text{Aut}_G(P)$ swapping $a_i$ for $a_i^g$, by the equation above. In particular, the automorphism sheaf of the trivial $G$-torsor $\text{Aut}_G(G) \cong G$ is endowed with the action by conjugation.
If we now consider the presheaf $$ F : X_{\text{Zar}}^{\text{op}} \to (\text{sets}) $$ given by $$ F(U) = P(U) \times G(U) / (h\cdot p,g)\sim (p,hgh^{-1}) $$ whose sheafification is the contracted product $F^\# = P \wedge^G G$ where $G$ acts on itself via inner automorphisms, we can construct a morphism $F \to \text{Aut}_G(P)$ by mapping the pair $(p,g)$ to the automorphism $$ P_{\mid U} \xrightarrow{\phi_p^{-1}} G_{\mid U} \xrightarrow{\cdot g} G_{\mid U} \xrightarrow{\phi_p} P_{\mid U} $$ where $\phi_p : G_{\mid U} \to P_{\mid U}$ is the trivialisation corresponding to the section $p \in P(U)$. Then applying $(-)^{\#}$ yields an isomorphism of sheaves $P\wedge^G G \xrightarrow{\cong} \text{Aut}_G(P)$ since for the trivial torsor, as mentioned above, we have $\text{Aut}_G(G) \cong G \cong G\wedge^G G$.
Finally, turning to your particular situation, we can actually make the same considerations for $\text{Aut}_{\mathcal{O}_X}(E)$ where $E$ is a vector bundle on $X$ (by which I mean a locally free sheaf of $\mathcal{O}_X$-modules of rank $n$): the $\text{GL}_{n,X}$-torsor $P$ corresponding to $E$ is the sheaf $$ U \mapsto \text{Isom}_{\mathcal{O}_X}(E_{\mid U}, \mathcal{O}_U^{\oplus n}) $$ and thus we can define a morphism of presheaves $$ F \to \text{Aut}_{\mathcal{O}_X}(E) $$ by mapping the pair $(p,g) \in P(U) \times \text{GL}_n(\mathcal{O}_U)$ to $$ E_{\mid U} \xrightarrow{p^{-1}} \mathcal{O}_U^{\oplus n} \xrightarrow{g} \mathcal{O}_U^{\oplus n} \xrightarrow{p} E_{\mid U} $$ which once again defines an isomorphism $F^\# = P\wedge^{\text{GL}_n}\text{GL}_n \cong \text{Aut}_{\mathcal{O}_X}(E)$ for the same reason.
So in conclusion we have $$ \text{Aut}_{\text{GL}_{n,X}}(P) \cong P\wedge^{\text{GL}_{n,X}}\text{GL}_{n,X} \cong \text{Aut}_{\mathcal{O}_X}(E).$$
I do want to mention that generally speaking I dislike working with cocycles... I'd love to see an explanation of the $G$ action on $\text{Aut}_G(P)$ without needing to fix a trivialisation :P
Hope this helps and I didn't mess something silly up (: