A binary operation on a set $A$ is a function from $A\times A \to A$, which means that it assigns exactly one element of $A$ to every ordered pair in $A \times A$.
In your example, $A=\{a,b\}$, so $$A\times A = \{(a,a),(a,b),(b,b),(b,a)\}$$
You can describe an operation on $A$ by saying exactly where it sends the four pairs (see Cameron Williams's comment).
Thus, for example, the second operation is the one that sends
$$(a,a) \text{ to } a, \text{ written } (a*a)=a\\
(a,b) \text{ to } b, \text{ written }(a*b)=b \\
(b,a) \text{ to } a, \text{ written } (b*a)=a \\
(b,b) \text{ to } b, \text{ written } (b*b)=b$$
To be commutative, an operation must send $(x,y)$ and $(y,x)$ to the same element for all $x,y \in A$. In the operation notation, this means that
$$\forall x,y \in A, \; x*y=y* x$$
Is this true for the second operation? There are only two distinct elements, so we just need to check that $a*b =b*a$.
But we have $a*b=a \neq b=b*a$, which means the operation is not commutative.
Now, existence of the identity: the identity is an element that fixes every element in $A$. Is there an element $e$ in $A$ such that $$e *x=x* e=x \quad\forall x \in A?$$
To check this, you need to look at each of the elements in $A$:
The equation
$$a*b=a \neq b=b*a$$
tells you that $a$ does not fix $b$ (because $a*b=a$) and that $b$ does not fix $a$ (because $b*a=b$). There are no more elements in $A$, so we can conclude that this operation does not have an identity.
Since it doesn't have an identity, it doesn't have inverses.
To check associativity, you need to check if performing the operation in a different order affects the result, i.e., if different ways of parenthesizing affect the result. The operation is associative if
$$(x*y)*z=x*(y*z) \quad \forall x,y,z \in A$$
Note that $x,y,z$ in the definition are not necessarily distinct.
I'll let you check this one; the operation will be associative if all the following hold (just go through each using the definition of the operation, performing what's in parenthesis first):
$$(a*a)*a\overset{?}{=}a*(a*a)$$
$$(b*b)*b\overset{?}{=}b*(b*b)$$
$$(a*b)*a\overset{?}{=}a*(b*a)$$
$$(b*a)*a\overset{?}{=}a*(b*a)$$
$$(a*a)*b\overset{?}{=}a*(a*b)$$
$$(a*b)*b\overset{?}{=}a*(b*b)$$
$$(b*b)*a\overset{?}{=}b*(b*a)$$
$$(b*a)*b\overset{?}{=}b*(a*b)$$
The definitions will become less confusing as you practice them; try checking them for familiar operations on familiar sets. For example, try checking them for addition and subtraction on the integers.
Best Answer
Asserting that the operation $*$ is not commutative means that there are elements $a$ and $b$ such that $a*b\neq b*a$. It does not mean that $a*b\neq b*a$ for any two distinct elements $a$ and $b$. Therefore, an operation may well not be commutative and, even so, to have an identity element. There is no contradiction here.
For an example of a non-commutative and non-associative algebraic structure with an identity element, take, for instance, the octonions.