SECTION A : What remains invariant for a complex $\:3\times 3\:$ tensor depends upon its transformation law under $\:U \in SU(3)\:$
CASE 1 : $\:\boldsymbol{3}\boldsymbol{\otimes}\boldsymbol{3}=\boldsymbol{6}\boldsymbol{\oplus}\overline{\boldsymbol{3}}\:$
The transformation law for the complex $\:3\times 3\:$ tensor $\:\mathrm{X}\:$ in this case is
\begin{equation}
\mathrm{X }^{\prime}=U\mathrm{X}U^{\mathsf{T}}\quad
\tag{A-01}
\end{equation}
Here the symmetry (+) or antisymmetry (-) is invariant since
\begin{equation}
\mathrm{X}^{\mathsf{T}}=\pm\:\mathrm{X} \Longrightarrow
{(\mathrm{X }^{\prime})}^{\mathsf{T}}=(U\mathrm{X}U^{\mathsf{T}})^{\mathsf{T}}=
{(U^{\mathsf{T}})}^{\mathsf{T}}\mathrm{X}^{\mathsf{T}}U^{\mathsf{T}}=U(\pm\:\mathrm{X})U^{\mathsf{T}}=\pm\:\mathrm{X }^{\prime}
\tag{A-02}
\end{equation}
In this case it makes sense to split the tensor in its symmetric and anti-symmetric parts
\begin{equation}
\mathrm{\Psi}=\dfrac{1}{2} \left(\mathrm{X}+\mathrm{X}^{\mathsf{T}}\right)\:, \quad\mathrm{\Omega}=\dfrac{1}{2} \left(\mathrm{X}-\mathrm{X}^{\mathsf{T}}\right)
\tag{A-03}
\end{equation}
The symmetric part $\:\mathrm{\Psi}\:$ depends on 6 parameters, so is identical to a complex $\:6$-vector $\:\boldsymbol{\psi}\:$ which belongs to a complex 6-dimensional invariant subspace and is transformed under a special unitary transformation $\:W \in SU(6)\:$
\begin{equation}
\boldsymbol{\psi}^{\prime}=W\boldsymbol{\psi}\:, \quad W \in SU(6)
\tag{A-04}
\end{equation}
while, on the other hand, the anti-symmetric part $\:\mathrm{\Omega}\:$ depends on 3 parameters, so is identical to a complex $\:3$-vector $\:\boldsymbol{\omega}\:$ which belongs to a complex 3-dimensional invariant subspace and is transformed under the special unitary transformation $\:\overline{U} \in SU(3)\:$
\begin{equation}
\boldsymbol{\omega}^{\prime}=\overline{U}\boldsymbol{\omega}\:, \quad \overline{U} \in SU(3)
\tag{A-05}
\end{equation}
That's why the symmetric and anti-symmetric parts give rise to the terms $\:\boldsymbol{6}\:$ and $\:\overline{\boldsymbol{3}}\:$ in the right hand of equation $\:\boldsymbol{3}\boldsymbol{\otimes}\boldsymbol{3}=\boldsymbol{6}\boldsymbol{\oplus}\overline{\boldsymbol{3}}\:$ respectively.
CASE 2 : $\:\boldsymbol{3}\boldsymbol{\otimes}\overline{\boldsymbol{3}}=\boldsymbol{8}\boldsymbol{\oplus}\boldsymbol{1}\:$
The transformation law for the complex $\:3\times 3\:$ tensor $\:\mathrm{X}\:$ in this case is
\begin{equation}
\mathrm{X }^{\prime}=U\mathrm{X}U^{\boldsymbol{*}}=U\mathrm{X}U^{-1}
\tag{A-06}
\end{equation}
For those interested, this is proved in SECTION B, motivated by the adventure of explaining structure of mesons under quark theory.
Here the symmetry (+) or antisymmetry (-) is NOT invariant
\begin{equation}
\mathrm{X}^{\mathsf{T}}=\pm\:\mathrm{X} \Longrightarrow
{(\mathrm{X }^{\prime})}^{\mathsf{T}}=(U\mathrm{X}U^{\boldsymbol{*}})^{\mathsf{T}}=
{(U^{\boldsymbol{*}})}^{\mathsf{T}}\mathrm{X}^{\mathsf{T}}U^{\mathsf{T}}=\overline{U}(\pm\:\mathrm{X})U^{\mathsf{T}} \ne\pm\:\mathrm{X }^{\prime}
\tag{A-07}
\end{equation}
So it makes NO SENSE to split the tensor in its symmetric and anti-symmetric parts.
On the contrary :
(1) if $\:\mathrm{X}\:$ is a constant tensor, that is a scalar multiple of the identity, $\:\mathrm{X}=z\mathrm{I}\:$ ($\:z \in \mathbb{C}\:$) , then is invariant $\:\mathrm{X }^{\prime}= U\mathrm{X}U^{-1}=U\left(z\mathrm{I}\right)U^{-1}=z\mathrm{I}=\mathrm{X}\:$
or
(2) since the transformation (A-06) is a similarity transformation, it preserves the Trace (=sum of the elements on the main diagonal) of $\:\mathrm{X}\:$, that is $\:Tr \left(\mathrm{X}^{\prime}\right)=Tr\left(\mathrm{X}\right)\:$. So a traceless tensor remains traceless.
It would sound not very well, but in this case the invariants are the "tracelessness" and the "scalarness".
In this case it makes sense to split the tensor in a traceless and in a scalar part :
\begin{equation}
\mathrm{\Phi}=\mathrm{X}-\left[\dfrac{1}{3}Tr\left(\mathrm{X}\right)\right]\cdot\mathrm{I}\:, \quad \mathrm{\Upsilon}=\left[\dfrac{1}{3}Tr\left(\mathrm{X}\right)\right]\cdot\mathrm{I}
\tag{A-08}
\end{equation}
The traceless part $\:\mathrm{\Phi}\:$ depends on 8 (=3x3-1) parameters, so is identical to a complex $\:8$-vector $\:\boldsymbol{\phi}\:$ which belongs to a complex 8-dimensional invariant subspace NOT FURTHER REDUCED TO INVARIANTS SUBSPACES and is transformed under a special unitary transformation $\:V \in SU(8)\:$
\begin{equation}
\boldsymbol{\phi}^{\prime}=V\boldsymbol{\phi}\:, \quad V \in SU(8)
\tag{A-09}
\end{equation}
while, on the other hand, the scalar part $\:\mathrm{\Upsilon}\:$ depends on 1 parameter, so is identical to a complex $\:1$-vector $\:\boldsymbol{\upsilon}\:$ which belongs to a complex 1-dimensional invariant subspace (identical to the set of complex numbers $\:\mathbb{C}\:$) and is transformed under the special unitary transformation $\:\mathrm{I} \in SU(1)\:$
(identical to the identity)
\begin{equation}
\boldsymbol{\upsilon}^{\prime}=\mathrm{I}\boldsymbol{\upsilon}=\boldsymbol{\upsilon}
\tag{A-10}
\end{equation}
Note that $\:SU(1)\equiv \{\:\mathrm{I}\:\}\:$, that is the group $\:SU(1)\:$ has only one element, the identity $\:\mathrm{I}\:$, while $\:U(1)\equiv\{\:U\::\:U=e^{i\theta}\mathrm{I}\:, \quad \theta \in \mathbb{R} \}\:$, that is mathematically identical to the unit circle in $\:\mathbb{C}\:$.
================================================================================
SECTION B : Mesons from three quarks
Suppose we know the existence of three quarks only : $\boldsymbol{u}$, $\boldsymbol{d}$ and $\boldsymbol{s}$. Under full symmetry (the same mass) these are the basic states, let
\begin{equation}
\boldsymbol{u}=
\begin{bmatrix}
1\\
0\\
0
\end{bmatrix}
\qquad
\boldsymbol{d}=
\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}
\qquad
\boldsymbol{s}=
\begin{bmatrix}
0\\
0\\
1
\end{bmatrix}
\tag{B-01}
\end{equation}
of a 3-dimensional complex Hilbert space of quarks, say $\mathbf{Q}\equiv \mathbb{C}^{\boldsymbol{3}}$. A quark $\boldsymbol{\xi} \in \mathbf{Q}$ is expressed in terms of these basic states as
\begin{equation}
\boldsymbol{\xi}=\xi_1\boldsymbol{u}+\xi_2\boldsymbol{d}+\xi_3\boldsymbol{s}=
\begin{bmatrix}
\xi_1\\
\xi_2\\
\xi_3
\end{bmatrix}
\qquad \xi_1,\xi_2,\xi_3 \in \mathbb{C}
\tag{B-02}
\end{equation}
For a quark $\boldsymbol{\eta} \in \mathbf{Q}$
\begin{equation}
\boldsymbol{\eta}=\eta_1\boldsymbol{u}+\eta_2\boldsymbol{d}+\eta_3\boldsymbol{s}=
\begin{bmatrix}
\eta_1\\
\eta_2\\
\eta_3
\end{bmatrix}
\tag{B-03}
\end{equation}
the respective antiquark $\overline{\boldsymbol{\eta}}$ is expressed by the complex conjugates of the coordinates
\begin{equation}
\overline{\boldsymbol{\eta}}=\overline{\eta}_1 \overline{\boldsymbol{u}}+\overline{\eta}_2\overline{\boldsymbol{d}}+\overline{\eta}_3\overline{\boldsymbol{s}}=
\begin{bmatrix}
\overline{\eta}_1\\
\overline{\eta}_2\\
\overline{\eta}_3
\end{bmatrix}
\tag{B-04}
\end{equation}
with respect to the basic states
\begin{equation}
\overline{\boldsymbol{u}}=
\begin{bmatrix}
1\\
0\\
0
\end{bmatrix}
\qquad
\overline{\boldsymbol{d}}=
\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}
\qquad
\overline{\boldsymbol{s}}=
\begin{bmatrix}
0\\
0\\
1
\end{bmatrix}
\tag{B-05}
\end{equation}
the antiquarks of $\boldsymbol{u},\boldsymbol{d}$ and $\boldsymbol{s}$ respectively. The antiquarks belong to a different space, the space of antiquarks $\overline{\mathbf{Q}}\equiv \mathbb{C}^{\boldsymbol{3}}$.
Since a meson is a quark-antiquark pair, we'll try to find the product space
\begin{equation}
\mathbf{M}=\mathbf{Q}\boldsymbol{\otimes}\overline{\mathbf{Q}}\: \left(\equiv \mathbb{C}^{\boldsymbol{9}}\right)
\tag{B-06}
\end{equation}
Using the expressions (B-02) and (B-04) of the quark $\boldsymbol{\xi} \in \mathbf{Q}$ and the antiquark $\overline{\boldsymbol{\eta}} \in \overline{\mathbf{Q}}$ respectively, we have for the product meson state $ \mathrm{X} \in \mathbf{M}$
\begin{equation}
\begin{split}
\mathrm{X}=\boldsymbol{\xi}\boldsymbol{\otimes}\overline{\boldsymbol{\eta}}=&\xi_1\overline{\eta}_1 \left(\boldsymbol{u}\boldsymbol{\otimes}\overline{\boldsymbol{u}}\right)+\xi_1\overline{\eta}_2 \left(\boldsymbol{u}\boldsymbol{\otimes}\overline{\boldsymbol{d}}\right)+\xi_1\overline{\eta}_3 \left(\boldsymbol{u}\boldsymbol{\otimes}\overline{\boldsymbol{s}}\right)+ \\
&\xi_2\overline{\eta}_1 \left(\boldsymbol{d}\boldsymbol{\otimes}\overline{\boldsymbol{u}}\right)+\xi_2\overline{\eta}_2 \left( \boldsymbol{d}\boldsymbol{\otimes}\overline{\boldsymbol{d}}\right)+\xi_2\overline{\eta}_3 \left(\boldsymbol{d}\boldsymbol{\otimes}\overline{\boldsymbol{s}}\right)+\\
&\xi_3\overline{\eta}_1 \left(\boldsymbol{s}\boldsymbol{\otimes}\overline{\boldsymbol{u}}\right)+\xi_3\overline{\eta}_2 \left(\boldsymbol{s}\boldsymbol{\otimes}\overline{\boldsymbol{d}}\right)+\xi_3\overline{\eta}_3 \left(\boldsymbol{s}\boldsymbol{\otimes}\overline{\boldsymbol{s}}\right)
\end{split}
\tag{B-07}
\end{equation}
In order to simplify the expressions, the product symbol $"\boldsymbol{\otimes}"$ is omitted and so
\begin{equation}
\begin{split}
\mathrm{X}=\boldsymbol{\xi}\overline{\boldsymbol{\eta}}=&\xi_1\overline{\eta}_1 \left(\boldsymbol{u}\overline{\boldsymbol{u}}\right)+\xi_1\overline{\eta}_2 \left(\boldsymbol{u}\overline{\boldsymbol{d}}\right)+\xi_1\overline{\eta}_3 \left(\boldsymbol{u}\overline{\boldsymbol{s}}\right)+ \\
&\xi_2\overline{\eta}_1 \left(\boldsymbol{d}\overline{\boldsymbol{u}}\right)+\xi_2\overline{\eta}_2 \left( \boldsymbol{d}\overline{\boldsymbol{d}}\right)+\xi_2\overline{\eta}_3 \left(\boldsymbol{d}\overline{\boldsymbol{s}}\right)+\\
&\xi_3\overline{\eta}_1 \left(\boldsymbol{s}\overline{\boldsymbol{u}}\right)+\xi_3\overline{\eta}_2 \left(\boldsymbol{s}\overline{\boldsymbol{d}}\right)+\xi_3\overline{\eta}_3 \left(\boldsymbol{s}\overline{\boldsymbol{s}}\right)
\end{split}
\tag{B-08}
\end{equation}
Due to the fact that $\mathbf{Q}$ and $\overline{\mathbf{Q}}$ are of the same dimension, it's convenient to represent the meson states in the product 9-dimensional complex space $\:\mathbf{M}=\mathbf{Q}\boldsymbol{\otimes}\overline{\mathbf{Q}}\:$ by square $3 \times 3$ matrices instead of row or column vectors
\begin{equation}
\mathrm{X}=\boldsymbol{\xi}\overline{\boldsymbol{\eta}}=
\begin{bmatrix}
\xi_1\overline{\eta}_1 & \xi_1\overline{\eta}_2 & \xi_1\overline{\eta}_3\\
\xi_2\overline{\eta}_1 & \xi_2\overline{\eta}_2 & \xi_2\overline{\eta}_3\\
\xi_3\overline{\eta}_1 & \xi_3\overline{\eta}_2 & \xi_s\overline{\eta}_3
\end{bmatrix}=
\begin{bmatrix}
\xi_1\\
\xi_2\\
\xi_3
\end{bmatrix}
\begin{bmatrix}
\overline{\eta}_1 \\
\overline{\eta}_2 \\
\overline{\eta}_3
\end{bmatrix}^{\mathsf{T}}
=
\begin{bmatrix}
\xi_1\\
\xi_2\\
\xi_3
\end{bmatrix}
\begin{bmatrix}
\overline{\eta}_1 & \overline{\eta}_2 & \overline{\eta}_3
\end{bmatrix}
\tag{B-09}
\end{equation}
Now, under a unitary transformation $\;U \in SU(3)\;$ in the 3-dimensional space of quarks $\;\mathbf{Q}\;$, we have
\begin{equation}
\boldsymbol{\xi}^{\prime}= U\boldsymbol{\xi}
\tag{B-10}
\end{equation}
so in the space of antiquarks $\overline{\mathbf{Q}}\;$, since $\;\boldsymbol{\eta}^{\prime}=U\boldsymbol{\eta}\;$
\begin{equation}
\overline{\boldsymbol{\eta}^{\prime}}= \overline{U}\;\overline{\boldsymbol{\eta}}
\tag{B-11}
\end{equation}
and for the meson state
\begin{equation}
\mathrm{X}^{\prime}=\boldsymbol{\xi}^{\prime}\boldsymbol{\otimes}\overline{\boldsymbol{\eta}^{\prime}}=\left(U\boldsymbol{\xi}\right)\left(\overline{U}\overline{\boldsymbol{\eta}}\right)
=
\Biggl(U\begin{bmatrix}
\xi_1\\
\xi_2\\
\xi_3
\end{bmatrix}\Biggr)
\Biggl(\overline{U}\begin{bmatrix}
\overline{\eta}_1\\
\overline{\eta}_2\\
\overline{\eta}_3
\end{bmatrix}\Biggr)^{\mathsf{T}}
\\=
U\Biggl(\begin{bmatrix}
\xi_1\\
\xi_2\\
\xi_3
\end{bmatrix}
\begin{bmatrix}
\overline{\eta}_1 & \overline{\eta}_2 & \overline{\eta}_3
\end{bmatrix}\Biggr)\overline{U}^{\mathsf{T}}
=
U\left(\boldsymbol{\xi}\boldsymbol{\otimes}\overline{\boldsymbol{\eta}}\right)U^{*}=U\;\mathrm{X}\;U^{*}
\tag{B-12}
\end{equation}
so proving the transformation law (A-06).
$===================\text{end of answer}=======================$
FIGURE : The quark structure of $\:\boldsymbol{\eta}^{\prime}\:$,$\:\boldsymbol{\eta}\:$ and $\:\boldsymbol{\pi}^{0}\:$ mesons
(Note: Meson symbols $\:\boldsymbol{\eta}^{\prime}\:$ and $\:\boldsymbol{\eta}\:$ must not be confused with the complex 3-vectors in the text)
(1) Meson $\:\boldsymbol{\eta}^{\prime}\:$ is a singlet, representative of $\:\boldsymbol{1}\:$ in $\:\boldsymbol{3}\boldsymbol{\otimes}\overline{\boldsymbol{3}}=\boldsymbol{8}\boldsymbol{\oplus}\boldsymbol{1}\:$
(2) Mesons $\:\boldsymbol{\eta}\:$ and $\:\boldsymbol{\pi}^{0}\:$ are members of the octet
$\;\boldsymbol{\lbrace}\boldsymbol{\pi}^{+},\boldsymbol{\pi}^{-},\boldsymbol{\pi}^{0},\mathbf{K}^{+},\mathbf{K}^{-},\mathbf{K}^{0},\overline{\mathbf{K}}^{0},\boldsymbol{\eta}\boldsymbol{\rbrace}\;$, basic meson states of $\boldsymbol{8}\:$ in $\:\boldsymbol{3}\boldsymbol{\otimes}\overline{\boldsymbol{3}}=\boldsymbol{8}\boldsymbol{\oplus}\boldsymbol{1}\:$ where
$\:\boldsymbol{\pi}^{+}\equiv\boldsymbol{u}\overline{\boldsymbol{d}}\:$ ,
$\:\boldsymbol{\pi}^{-}\equiv\boldsymbol{d}\overline{\boldsymbol{u}}\:$ ,
$\:\mathbf{K}^{+}\equiv\boldsymbol{u}\overline{\boldsymbol{s}}\:$ ,
$\:\mathbf{K}^{-}\equiv\boldsymbol{s}\overline{\boldsymbol{u}}\:$ ,
$\:\mathbf{K}^{0}\equiv\boldsymbol{d}\overline{\boldsymbol{s}}\:$ ,
$\:\overline{\mathbf{K}}^{0}\equiv\boldsymbol{s}\overline{\boldsymbol{d}}\:$ .
$\:\mathbf{Q}\boldsymbol{\otimes}\overline{\mathbf{Q}}=\boldsymbol{\lbrace}\boldsymbol{\pi}^{+},\boldsymbol{\pi}^{-},\boldsymbol{\pi}^{0},\mathbf{K}^{+},\mathbf{K}^{-},\mathbf{K}^{0},\overline{\mathbf{K}}^{0},\boldsymbol{\eta}\boldsymbol{\rbrace}\boldsymbol{\oplus}\boldsymbol{\lbrace}\boldsymbol{\eta}^{\prime}\boldsymbol{\rbrace}\:$
Your text assumes you are familiar with the quantum number content of the elementary particle fermions, determined by the Millikan oil-drop experiment, structure functions of the light quarks, V-A structure of the weak currents, etc. These are experimental inputs and they come from out there, your world.
It helps you summarize the self-evident logic of their apparently diverse quantum numbers so you could write a compact QFT for them, that's all. I assume you seek an appreciation of the manifest logic involved.
It gives you the SU(3) color rep, singlet for leptons, color 3 for quarks, or color anti triplet for antiquarks. Likewise, their SU(2) weak isospin, vanishing for right handed singlets, and doublet for left-handers. (No separate 2-bars, of course, as SU(2) is pseudoreal.)
And, of course, mutatis mutandis for their CPT conjugates. You only have singlets and fundamental reps, since these are fundamental fermion building blocks of our world.
Thus,
- (1,2,-1/2) , e.g. for $e _L$
- (1,1,1), e.g. for $\overline {e _R}$
- (3,2,1/6) e.g. for $u_L, d_L$
- ($\bar 3$,1,-2/3) for $\overline{u_R}$
- ($\bar 3$,1,1/3) for $\overline{d_R}$.
The hypercharge in the third entry is dross -- an error-correction number, if you wish, given by $Y_W\equiv Q-T_3$, once you input the charge, in the "minority usage", but actually modern mainstream definition, so you might have to multiply it by 2 to agree with hidebound historical listings, like those linked here. It is the eigenvalue of U(1), as your particles are all singlets, of course, under it, and multiplies the B coupling charge of the fermion currents. The sooner you get used to its Golden Mnemonic, the better: It is the average charge of isomultiplets.
There is nothing more to it. Given these numbers you may completely, and concisely specify the fermion sector of the SM QFT.
Best Answer
Actually the Lie group $$G:=SU(3)\times SU(2) \times U(1)$$ is not a subgroup of $SU(5)$. However the standard model gauge group $G/\mathbb{Z}_6$ is a subgroup of the GUT gauge group $SU(5)$, cf. e.g. this, this & this Phys.SE posts.
Here we will argue at the level of Lie algebras $$su(3)\oplus su(2) \oplus u(1)\subseteq su(5).$$ In detail, we identify $su(5)$ with anti-Hermitian traceless $5\times 5$ matrices; $su(3)$ with anti-Hermitian traceless $3\times 3$ block matrices in rows/columns 1,2,3; and $su(2)$ with the anti-Hermitian traceless $2\times 2$ block matrices in rows/columns 4,5; while $u(1)$ is generated by the diagonal traceless matrix ${\rm diag}(-2,-2,-2,3,3)$ times an imaginary number.
The vectorspace $V_5=V_3\oplus V_2$ of the defining/fundamental representation $\underline{\bf 5}$ of $su(5)$ is decomposed into the defining/fundamental representation $\underline{\bf 3}$ of $su(3)$ in rows 1,2,3; and the defining/fundamental representation $\underline{\bf 2}$ of $su(2)$ in rows 4,5.
On the other hand, the first three rows $V_3$ are a singlet under $su(2)$; while the last two rows $V_2$ are a singlet under $su(3)$.
Also note that the generator of $u(1)$ has the same weak hypercharge/eigenvalue $-2i$ and $3i$ on $V_3$ and $V_2$, respectively. The overall normalization of the weak hypercharge depends on conventions/choice of the $u(1)$ generator.
Altogether, the decomposition of the $su(5)$ representation becomes $$\underline{\bf 5}~~\cong~~(\underline{\bf 3}\otimes\underline{\bf 1})_{-\frac{1}{3}}~~\oplus~~ (\underline{\bf 1}\otimes\underline{\bf 2})_{\frac{1}{2}}.$$