The problem is that the definition is for multiplicative notation, while you consider an additive group (what makes the situation worse is that multiplication also makes sense for residue classes).
For a group where you use additive notation the defintion transforms to:
For $(G,+)$ a group and $x\in G$ define the order of $x$ to be the smalles positive integer $n$ such that $nx=0_G$, and denote this integer by $|x|$.
The point is you always want to know how often do I have to combined (according to the group law) the element with itself to get the neutral element of the group.
For residue classes you need to be extra careful to know if you consider addition or multiplication. You consider addition. And, so for order of $\overline{6}$ equal $3$ you care about $\overline{6} + \overline{6} + \overline{6} = \overline{0}$, or written differently $3 \ \overline{6} =\overline{0}$.
First, I hope you understand that the two definitions in your post are actually the same definition of the same concept, with the only difference being the notation chosen for the group operation — additive or multiplicative.
Second, as a minor omission, your list of element of $(\mathbb{Z}/18\mathbb{Z},+)$ is missing $\bar{0}$. Also, the order or $\bar{5}$ is $18$, not $10$, but it's probably a typo. Overall, you understand that one correctly.
Third, as for your last question, it's because of the very definition of what $((\mathbb{Z}/18\mathbb{Z})^{*},\cdot)$ is. That asterisk means the set of all invertible (with respect to multiplication) elements of the given ring. If we want to use multiplication as the group operation, then to actually have a group, each element has to have a multiplicative inverse — it's one of the group axioms. So elements that don't have inverses will have to be discarded.
A few examples. Obviously, $\bar{0}\notin(\mathbb{Z}/18\mathbb{Z})^{*}$. But $\bar{2}\notin(\mathbb{Z}/18\mathbb{Z})^{*}$ either. One way to see that is to notice that $\bar{2}\cdot\bar{9}=\bar{0}$, so $\bar{2}$ (as well as $\bar{9}$) is a zero-divisor, and therefore can't have an inverse. Or you can test all elements of $\mathbb{Z}/18\mathbb{Z}$ to see that none of them works as the inverse of $\bar{2}$.
So the hint simply tells you to actually right down all elements of $(\mathbb{Z}/18\mathbb{Z})^{*}$ to see what you're dealing with. You should see that discarding all zero-divisors modulo $18$ results in keeping only the elements that are relatively prime to $18$. And then by repetetive multiplication you can find the (multiplicative) order of each of them.
Best Answer
As been pointed out you confuse the different notations. As for the elements $$0*0=0$$ $$8*1=0$$ $$4*2=0$$ $$8*3=0$$ $$2*4=0$$ $$8*5=0$$ $$4*6=0$$ $$8*7=0$$