I want to know to what extent is the group action determined by its fixed point data and orbit data, i.e. if $G$ acts on $M$ in two ways with the same fixed point set and orbit space, on what occasions it is true that the two ways are equivalent (i.e.the two actions are conjugate in $Homeo(M)$)?
[Math] Group action, Fixed point set and Orbit Space
at.algebraic-topologygroup-actionsgt.geometric-topology
Related Solutions
EDIT: As explained in my comment below, this answer does not really address the question, but rather the changed question where we have the (stronger) assumption that all stabilizers are discrete.
Suppose $M$ is closed and orientable. We can then consider the Borel fiber sequence $$M \to M \times_{SO(3)} ESO(3) \to BSO(3);$$ in case that the given action $SO(3) \curvearrowright M$ has discrete stabilizers, the map $M \times_{SO(3)} ESO(3) \to M/SO(3) = X$ is a rational equivalence. Now $X$ is a finite complex, and $H^{\ast}(BSO(3);\mathbb Q) = \mathbb Q[p_1]$, with $p_1$ the first Pontryagin class in degree $4$. Ananalyzing the Serre spectral sequence of the above fibration then yields $b_1(M) = b_4(M)$, $b_2(M) = b_5(M)$ and $b_3(M) = b_0(M)+b_6(M) = 2$, where $b_i(M) = \text{dim}_{\mathbb Q}H^i(M)$ stands for the $i^{\text{th}}$ Betti number of $M$. Using Poincaré duality, we can finally deduce $b_1 = b_2 = b_4 = b_5$. All possible values indeed arise, simply take $$M = SO(3) \times \#^k(S^1 \times S^2),$$ and let $SO(3)$ act on the first factor by left translation, and trivially on the second factor.
Did you try to use the double $T$ of the surface $S$? Any fixed point-free action of $\mathbf Z/n$ on $S$ induces a fixed point-free action on the closed orientable surface $T$. Moreover, the induced action commutes with the natural orientation-reversing involution on $T$. By Nielsen's Theorem that you mentioned, you've reduced to study orientation-reversing involutions on $T$ that commute with a given action of $\mathbf Z/n$ on $T$ and whose set of fixed points is homeomorphic to a disjoint union of $b$ circles. Such involutions induce orientation-reversing involutions on the quotient surface $U=T/(\mathbf Z/n)$. Now, orientation-reversing involutions on a closed orientable surface $U$ whose set of fixed points is a disjoint union of circles are easily classified. It should not be too difficult to decide which ones lift to orientation-reversing involutions on $T$ having a set of fixed points composed of $b$ circles.
With this approach you already get conditions on the mere existence of fixed point-free actions of $\mathbf Z/n$ on $S$. Indeed, the double $T$ of $S$ has genus $2g+b-1$. So that $n$ has to divide $$ -\chi(T)=2(2g+b-1)-2=4g+2b-4 $$ in order for a fixed point-free action of $\mathbf Z/n$ on $S$ to exist. For example, if $S$ is a pair of pants, i.e., $g=0$ and $b=3$, the natural number $n$ has to divide $2$. Adopting the above strategy, I think you only get $1$ fixed point free action of $\mathbf Z/2$ on the pair of pants $S$: the one where the nontrivial element of $\mathbf Z/2$ acts on $S$ by exchanging $2$ boundary circles, and by acting antipodally on the third. That case probably was clear anyway.
Best Answer
First, you might want to identify actions that are not conjugated in $\mathrm{Homeo}(M)$. For example, let $\mathbb{Z}\times\mathbb{Z}$ act on $M$ in the following two ways: the first factor acts by a map $f:M\to M$ and the second acts trivially, or the converse. Then the actions are the same only up to an automorphism of the group.
Second, it is not clear to me what you mean by "the same orbit space". Rational rotations of the circle all have homeomorphic orbit spaces, but of course they are not conjugate in any way.
I guess that even in favorable cases, you need the (conjugacy class of the) stabilizer of each orbit to identify the action.
Last, a remark that is not directly linked to your question, but that I like to advertise: there exist (infinite families) of analytic group actions on analytic manifolds that are topologically but not $C^1$ conjugate.