Is there any rules to find a set of representative??
I understand a set of representative is a kind of notion for a set that contains many subsets (whose size is larger 1) as elements.
I'm wondering how to find those representatives.
And What is the representative for?
Consider the following example:
We have a set (group) $G$
$$
G = \{1,3,5,7,9,11,13,15\}
$$
Now, we have $\langle 3 \rangle = \{1, 3, 9, 11\}$.
So we can have the quotient group
$$
Z = G/\langle 3 \rangle = \{\{1, 3, 9, 11\}, \{5, 7, 13 ,15\}\}.
$$
We now have two groups as elements of $Z$.
What is the representatives of those elements? I think $3$ would be the one for the former subgroup. If so, why not the others? $1, 9, or 11$.
Also what is the representative of the latter group?
Best Answer
Your question is not completely clear as you have not specified what the group operation is.
However from looking at your question, I think that your group $G$ is the group of units modulo 16. This is sometimes denoted as $ \mathbb{Z}_{16}^{*}$. The binary operation is multiplication mod 16.
Now assuming this is the group you mean, I will now work through your question.
The group generated by $\langle 3 \rangle $ does indeed consist of $\{1,3,9,11\}$. As your group $G$ is abelian, you know that the subgroup $\langle 3 \rangle $ is normal, and so the quotient is well defined.
You have then correctly found the two cosets of $\langle 3 \rangle$ in $G$ (or put another way, the two elements of the quotient group).
In your question you refer to these as two groups, but this is incorrect, these are two elements of a (quotient) group. Alternatively you might refer to them as cosets (of $\langle 3 \rangle$ in $G$ ). Notice that in general a coset of $G$ is not a subgroup of $G$, for instance ${5,7,13,15}$ does not contain the identity element.
Now I think what your question boils down to is what representatives can you pick. Well you can pick anything that is in the coset as a representative, although sometimes there is a natural choice.
Notice $\{1,3,9,11\}=1 \cdot \langle 3 \rangle = 3 \cdot \langle 3 \rangle = 9 \cdot \langle 3 \rangle = 11 \cdot \langle 3 \rangle $ . So here you could pick as your representative any of $1$ , $3$, $9$ or $11$.
And $\{5, 7, 13, 15\}= 5 \cdot \langle 3 \rangle = 7 \cdot \langle 3 \rangle = 13 \cdot \langle 3 \rangle = 15 \cdot \langle 3 \rangle $ . Here you could pick as your representative any of $5$ , $7$ ,$13$ , $15$.
Sometimes a more natural choice will become apparent, for instance if you have a coset containing even numbers and another containing odd numbers, perhaps $0$ and $1$ would be a natural choice, however you could of course choose something else!
I hope this helps and that I have correctly understood what you are asking.
For some further reading on this please see the following:
A very similar example
Is a coset a subgroup?
Canonical coset representatives