Your thinking is more or less correct. Generally, when we wish to think about the subgroups of a group $G$, we take different subsets of $G$ and look at what groups those subsets generate.
The obviously thing to do is start with one element. Let's say I start with a rotation $r^k$ for some $k = 1, 2, 3, 4$. Since $5$ is a prime, the numbers $1, 2, 3, 4$ have multiplicative inverses mod $5$. This implies that $\langle r^k\rangle$ actually contains $r$. Hence, $\langle r^k\rangle = \langle r\rangle = \{1, r, r^2, r^3, r^4\}$ for each $k$. Now, let's say I start with a reflection $z$ instead. It's easy to see that $\langle z\rangle = \{1, z\}$. Thus, we have $5$ different subgroups of order $2$, one for each reflection. The only remaining case is $\langle 1\rangle = \{1\}$.
Now let's consider a generating set with two elements. If we take $\langle r^k, r^l\rangle$ for some $k, l$, this is clearly just $\langle r\rangle$ again. Suppose instead we take $\langle r^k, z\rangle$. Then this subgroup contains $r$ and $z$, hence it contains $r^l z$ for each $l=0, 1, 2, 3, 4$, i.e. it contains every reflection. It also contains the powers of $r$, which are the rotations, hence $\langle r^k, z\rangle = G$. Finally, we could take $\langle z_1, z_2\rangle$ for two reflections $z_1, z_2$. But then $z_1 z_2$ is a rotation which is in $\langle z_1, z_2\rangle$, hence by the previous case we have $\langle z_1, z_2\rangle = G$.
From here it's easy to see that we've found every subgroup of $G$. Obviously this took a lot less time than checking all $2^{10}$ subsets of $G$.
Think of the circle as being oriented, say counterclockwise. Then rotations preserve this orientation, while reflections change it. Hence there are at least two kind of symmetries...
Note however that rotations can be obtained by composing two reflections about different axes. So in some sense reflections alone do suffice.
Best Answer
The reflections along diagonals are not symmetries of a general rectangle - they "exchange" (imperfectly except for a square) the long side and the short side.
To have a symmetry by reflection in a diagonal the adjacent sides have to be the same length.