You just need to find two matrices that don't commute.
\begin{gather}
\begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}
\begin{bmatrix}
1 & 0 \\
1 & 1
\end{bmatrix}=
\begin{bmatrix}
2 & 1 \\
1 & 1
\end{bmatrix}
\\[6px]
\begin{bmatrix}
1 & 0 \\
1 & 1
\end{bmatrix}
\begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}
=
\begin{bmatrix}
1 & 1 \\
1 & 2
\end{bmatrix}
\end{gather}
For $n\ge2$ just take these as the upper left block and complete with ones on the diagonal and zero elsewhere. The coefficients at $(1,1)$ are different.
Let $G $ be a group and $A $ a set. A (left) group action\begin{eqnarray} \star: G \times A &&\to A \\ (g,a) &&\mapsto g \star a \end{eqnarray} is a function from the cartesian product that is compatible with the groups structure ,i.e. $\forall a \in A, g,h \in G$
\begin{eqnarray} e\star a &&=a \\ g \star(h\star a) &&= (gh)\star a\end{eqnarray}
where e is the neutral element.
The prime example would be the generic group action, a group acting on its own underlying set via leftmultiplication with a group element, that is A=G and \begin{eqnarray} \circ: G \times G &&\to G \\ (g,h) &&\mapsto g\circ h= gh \end{eqnarray} where we did nothing but relabel the groups inner operation for clarity. So this is obviously a group action because per definition $G$'s operation is associative and $e$ acts trivially on every element.
So how does this lead us to graphs?
A group action on a set A gives us an associated graph $X=(A,E)$ via $E=\{(g \star a,a): g \in G\}$. in case of the generic operation on the group $G$ on itself we optain a graphic representation of the group operation.
So this graph is really just the same thing as the multiplication table, for each element you can see what comes out if you multiply any element from the left.
We may notice that a lot of the edges in this graph are redundant. So we may specify and put special attention to a certain subset $ S \subset G$ and only add edges that represent leftmultiplication with elements from $(S \cup S^{-1}) \backslash \{e\}$, where $S^{-1}=\{s^{-1}: s \in S\}$ (omitting the neutral element since all it does is produce loops in the graph).
Usually one looks at so-called generating sets $S$ (that is there is no proper subgroup of $G$ containing $S$) and thats what i refer to as the Cayley-Graph of $G$ by the generator $S$
Because Cayley diagram talk about actions ,I thought It could be related to group action somehow.
I interpret that as saying that the edges in the cayley graph represent the action of an element of the generating set. And although the "action" of S on G is not a group action in the forementioned sense (since $S$ is just a subset, not a group), it is however the restriction of the generic group action $\circ: G \times G \to G$ to $(S \cup S^{-1}) \backslash \{e\} \times G$,
The problem is ,I know from the abstract definition that how is group actions came from groups from defintions .But I can't really see how cayley diagram(with generators and relations) and the abstract group action is related to each other. And if they are related which one I should use for intuitive studying?
There are quite a few theorems linking them. For example the cayley graph of a free group is a tree. What do you mean by relations in a Caleygraph?
Well use whatever works for you; for calculations tables seem more practical, for me graphs provide a better overview of the groups structure.
Best Answer
First, verify that $|A|=4$ and $|B|=2$.
Next compute $AB,A^2B,A^3B,BA,$ and so on.
But from here you will get that $$BA=A^3B$$ By using this relation, we obtain that $$BA^2=A^2B, BA^3=AB$$ By using these relation, every element in $H$ can be written as $A^iB^j$ where $0\le i\le3$ and $0\le j \le1$.
So we get $$H=\{1,A,A^2,A^3,B,AB,A^2B,A^3B\}$$ and this is actually isomorphic to $D_8$.