For the first question I think you have some misunderstandings. Firstly Chern classes are defined for complex vector bundles - or equivalently principal $U(n)$-bundles. If $G$ is an arbitrary compact Lie group then the Chern classes of a principal $G$-bundle will be understood in terms of some representation $\rho:G\rightarrow U(n)$ (these always exist). For instance when $G=SU(n)$ the representation $\rho$ is the canonical inclusion.
Now if you are given a principal $G$-bundle $p:P\rightarrow X$ over a suitable space $X$ for some topological group $G$, and a representation $\rho:G\rightarrow U(n)$ then you can form the associated vector bundle $P\times_\rho\mathbb{C}^n\rightarrow X$ with total space
$P\times_\rho\mathbb{C}^n=P\times \mathbb{C}^n/[(e,v)\sim(eg,\rho(g)^{-1}v)]$
where $e\in P$, $v\in\mathbb{C}^n$ and $g\in G$. You can now ask for characteristic classes of this vector bundle, and in particular its Chern classes (and there are an infinte number of Chern classes, not just two).
Note that the Chern classes of this bundle are not the characteristic classes of the principal bundle $P$, but are the characteristic classes of some principal $U(n)$-bundle $Q\rightarrow X$ that admits a $(G,\rho)$-reduction of structure to $P$.
The principal bundle $P$ will have its own set of characteristic classes deriving from the cohomology of the classifying space $BG$ of $G$.
Note in particular that the vector bundle $P\times_\rho\mathbb{C}^n$ is not the adjoint bundle $ad(P)$, which is another vector bundle whose characteristic classes have even less to do with those of $P$. Consider for instance the canonical line bundle $\eta$ over $\mathbb{C}P^1=S^2$. Since $U(1)$ is abelian we have that $ad(\eta)\cong S^2\times \mathbb{R}$ is trivial (recall that the Lie algebra of $U(1)$ is $i\mathbb{R}$). However $\eta$ itself is not trivial and is in fact associated to the Hopf $S^1$-bundle $S^3\rightarrow S^2$ which has first Chern class $1\in H^2(S^2)$.
In summary it makes equal sense to ask for the characteristic classes of a principal bundle or the characteristic classes of a vector bundle with structure group $G$ if the representation $\rho$ is understood.
As for the second question, instantons cannot have a Chern class. Instantons are generally understood as certain field configurations. Given for instance as a suitable connection on a principal of vector bundle. I think what you are referring to is the instanton charge, which in the case above is defined to be the Chern class of the bundle to which the instanton is related. As I understand it a $U(1)$-instanton can not have non-zero second Chern class. It can have non-zero first Chern class. In fact for rank $N=1$ we have $c_1=n$ and $c_2=0$.
*Edit: Added to clarify comments.
I wasn't quite clear above. In the following I'll limit attention of $U(n)$-bundles. There is a one-to-one correspondence between (complex) vector bundles over $X$ and $U(n)$-bundles over $X$. This happens in one direction through the construction I gave above, and in the other by taking the frame bundle of a given vector bundle.
Thus, given a complex vector bundle $E\rightarrow X$ of rank $n$ we have a principal $U(n)$-bundle $Fr(E)\rightarrow X$ and an isomorphism of vector bundles $Fr(E)\times_{U(n)}\mathbb{C}^n\cong E$ over $X$. We can then take a connection on either bundle and ask for the Chern classes it gives. To see that the classes defined by each are the same recall firstly that a connection on a principal bundle induces one on any associated bundle. So a connection $A$ on the principal bundle $Fr(E)$ induces one, $\nabla_A$, on the vector bundle $Fr(E)\times_{U(n)}\mathbb{C}^n\cong E$. Moreover it is seen that both connections $A$ and $\nabla_A$ yield identical curvature 2-forms. Thus
$c_i(E)^{\nabla_A}=c_i(Fr(E))^A\in H^*(X)$
where we add a superscript to denote which connection we use to define the classes.
Now recall that the Chern classes are independent of the choice of connection. It follows that any other connection $A'$ on $Fr(E)$ yields identical Chern classes
$c_i(Fr(E))^A=c_i(Fr(E))^{A'}\in H^*(X)$.
By the same reasoning any other connection $\nabla'$ on your original vector bundle $E$ yields identical Chern classes
$c_i(E)^{\nabla'}=c_i(E)^{\nabla_A}\in H^*(X)$.
The point is that it doesn't matter which bundle $E$ or $Fr(E)$ we use to define the Chern classes: they are the same. The one-to-one correspondence between vector bundles and principal bundles carries over to their characteristic classes and you can consider the Chern classes to 'belong' to either bundle.
As for your comments regarding the Yang-Mills functional. Firstly I believe what you say about it being bounded by the second Chern class is only relevant to Yang-Mills theory over a 4-dimensional base. It is certainly not true for Yang-Mills on a surface, or 8-manifold, say. As for what bundle, it is the one you are working with, which should be granted by deeper context. By the above, it does not matter it is a vector bundle or the corresponding principal bundle.
And if you are talking about some $G$-bundle then yes, there must be an implicit unitary representation. The Chern classes are the characteristic classes belonging to $U(n)$. They can be defined as certain cohomology classes in the cohomology of the classifying space $BU(n)$. You cannot ask for the Chern classes of a $G_2$-bundle, say, which will have its own set of characteristic classes. If you are working with a vector bundle which has a $G$-structure then this $G$-structure provides a unitary representation for $G$.
*Edit 2: Clarification of further comments.
The Chern classes may be defined as certain classes in the cohomology of the classifying space $BU(n)$ of $U(n)$. The integral cohomology of $BU(n)$ is the polynomial ring $H^*(BU(n))\cong \mathbb{Z}[c_1,\dots c_n]$ on classes $c_i\in H^{2i}(BU(n))$. The classes $c_i$ are the universal Chern classes: the Chern classes belonging to the universal $U(n)$-bundle $EU(n)\rightarrow BU(n)$ with contractible total space (you can think of $BU(n)$ as the Grassmanian of $n$-planes in $\mathbb{C}^\infty$).
Then if $f:X\rightarrow BU(n)$ is a suitable classifying map for the bundle $E\rightarrow X$ (principal $U(n)$ or complex vector bundle, we have already decided these are the same things in a certain sense), in that $f^*EU(n)\cong E$, you can define the Chern classes of $E$ to be
$c_i(E)=f^*(c_i)\in H^{2i}(X)$.
This is the point of view in algebraic topology, and its truly quite marvelous that this same information can be extracted by studying the geometric data contained in a connection on $E$.
It may have been misleading when I said that there were an infinte number of Chern classes. On the one hand it's true if you consider the infinite unitary group which has $H^*(BU(\infty))\cong\mathbb{Z}[c_1,\dots,c_n,\dots]$, and you can certainly embed a rank $n$ vector bundle $E$ into the rank $n+1$ vector bundle $E\oplus\epsilon^1$ and ask for its Chern classes. However whilst you can take $n$ all the way to infinity in this manner, there will only be at most $n$ non-trivial Chern classes for a rank $n$ bundle. There are an infinite number you can ask for, but only $n$ interesting ones.
Now more generally you may be interested not in the individual Chern classes but in certain polynomial combinations of them, and there are indeed an infinite number of these you could potentially ask for.
Now the final point is that over a 4-manifold $X$ there will only be two potentially non-trivial Chern classes. This is simply because $H^*(X)$ vanishes in dimensions greater than $4$. Thus we have $c_1(E)\in H^2(X)$ and $c_2(E)\in H^4(X)$ and anything else lives in higher degrees and is thus trivial.
An application of this? If $E\rightarrow X$ is a rank $n$ complex vector bundle over a $4$-manifold with $n>2$, then there is a rank $2$ vector bundle $E'\rightarrow X$ and a bundle morphism $E\cong E'\oplus\epsilon^{n-2}$.
Best Answer
Chern classes can be defined in multiple different but roughly equivalent ways. As you have seen, they can be defined to be certain classes in $H^{2i}(X;\mathbb{Z})$, or they can be defined to be certain classes in $H^{2i}(X;\mathbb{R})$ (in the latter case, usually because they arise naturally via de Rham cohomology). However, these two definitions are very closely related and there is usually no confusion in calling both of them "Chern classes".
To relate the definitions, recall that any space $X$, there is a natural homomorphism $H^*(X;\mathbb{Z})\to H^*(X;\mathbb{R})$, induced by the inclusion map $\mathbb{Z}\to\mathbb{R}$. The two types of Chern classes mentioned above are related by this natural homomorphism: if you take the Chern class in $H^{2i}(X;\mathbb{Z})$ and apply the natural homomorphism $H^{2i}(X;\mathbb{Z})\to H^{2i}(X;\mathbb{R})$, you get the Chern class in $H^{2i}(X;\mathbb{R})$. When people talk about classes in de Rham cohomology as actually being in $H^{*}(X;\mathbb{Z})$, they really mean they are in the image of this homomorphism.
In general, beware that the homomorphism $H^{2i}(X;\mathbb{Z})\to H^{2i}(X;\mathbb{R})$ may not be injective (the kernel is the torsion elements of $H^{2i}(X;\mathbb{Z})$), and so the integral Chern classes are "stronger" than the real Chern classes. For instance, a complex line bundle is determined up to isomorphism by its first integral Chern class, but it is not determined by the first real Chern class (if $H^2(X;\mathbb{Z})$ has torsion).