A binary operation on a set $A$ is a function from $A\times A \to A$, which means that it assigns exactly one element of $A$ to every ordered pair in $A \times A$.
In your example, $A=\{a,b\}$, so $$A\times A = \{(a,a),(a,b),(b,b),(b,a)\}$$
You can describe an operation on $A$ by saying exactly where it sends the four pairs (see Cameron Williams's comment).
Thus, for example, the second operation is the one that sends
$$(a,a) \text{ to } a, \text{ written } (a*a)=a\\
(a,b) \text{ to } b, \text{ written }(a*b)=b \\
(b,a) \text{ to } a, \text{ written } (b*a)=a \\
(b,b) \text{ to } b, \text{ written } (b*b)=b$$
To be commutative, an operation must send $(x,y)$ and $(y,x)$ to the same element for all $x,y \in A$. In the operation notation, this means that
$$\forall x,y \in A, \; x*y=y* x$$
Is this true for the second operation? There are only two distinct elements, so we just need to check that $a*b =b*a$.
But we have $a*b=a \neq b=b*a$, which means the operation is not commutative.
Now, existence of the identity: the identity is an element that fixes every element in $A$. Is there an element $e$ in $A$ such that $$e *x=x* e=x \quad\forall x \in A?$$
To check this, you need to look at each of the elements in $A$:
The equation
$$a*b=a \neq b=b*a$$
tells you that $a$ does not fix $b$ (because $a*b=a$) and that $b$ does not fix $a$ (because $b*a=b$). There are no more elements in $A$, so we can conclude that this operation does not have an identity.
Since it doesn't have an identity, it doesn't have inverses.
To check associativity, you need to check if performing the operation in a different order affects the result, i.e., if different ways of parenthesizing affect the result. The operation is associative if
$$(x*y)*z=x*(y*z) \quad \forall x,y,z \in A$$
Note that $x,y,z$ in the definition are not necessarily distinct.
I'll let you check this one; the operation will be associative if all the following hold (just go through each using the definition of the operation, performing what's in parenthesis first):
$$(a*a)*a\overset{?}{=}a*(a*a)$$
$$(b*b)*b\overset{?}{=}b*(b*b)$$
$$(a*b)*a\overset{?}{=}a*(b*a)$$
$$(b*a)*a\overset{?}{=}a*(b*a)$$
$$(a*a)*b\overset{?}{=}a*(a*b)$$
$$(a*b)*b\overset{?}{=}a*(b*b)$$
$$(b*b)*a\overset{?}{=}b*(b*a)$$
$$(b*a)*b\overset{?}{=}b*(a*b)$$
The definitions will become less confusing as you practice them; try checking them for familiar operations on familiar sets. For example, try checking them for addition and subtraction on the integers.
Best Answer
If your domain has at least two elements, a simple example of two functions meeting your conditions are $f(x,y) = x$ and $g(x,y) = y$. Picking $a\neq b$ gives you $f(a,b) = a \neq b = f(b,a)$, and $g(a,b) = b \neq a = g(b,a)$.
Moreover, for any $a,b$ we have: $$\begin{align*} g(f(a,b),a) &= a &&\text{(your first equation)}\\ g(a,f(a,b)) &= a &&\text{(your third equation)}\\ g(f(a,b),b) &= b &&\text{(your sixth equation)}\\ g(b,f(a,b)) &= a &&\text{(your seventh equation)} \end{align*}$$
(If your domain has exactly one element, then you cannot find “noncommuting” functions, since $f(x,y) = f(y,x)$ as we must have $x=y$)