The answer is quite classical when $f \colon X \to Y$ is an unramified double cover.
In this case Riemann - Hurwitz formula gives
$g(X)-1 = 2g(Y)-2$.
Consider the following three natural maps:
$f^* \colon J(Y) \to J(X)$,
$Nm \colon \textrm{Pic}^0(X) \to \textrm{Pic}^0(Y), \quad Nm(\sum a_ip_i):= \sum a_if(p_i)$
$\tau \colon J(X) \to J(X)$,
where $f^*$ is induced by the pull-back of $0$-cycles, $Nm$ is the norm map and $\tau$ is the involution induced by the double cover $f$.
Then
$\textrm{Ker} \; f^*=\langle L \rangle$, where $L$ is a point of order $2$ in $J(Y)$;
the connected component of $Nm^{-1}(0)$ containing the identity coincides with the image of $I-\tau$. It is an Abelian subvariety of $\textrm{Pic}^0(X)$ of dimension $g(Y)-1$, that is denoted by $\textrm{Prym}(X, \tau)$.
Moreover, under the identification of $\textrm{Pic}^0(X)$ with $J(X)$, the principal polarization of $J(X)$ restricts to twice a principal polarization on $\textrm{Prym}(X, \tau)$.
The geometry of Prym varieties is very rich. In particular, Riemann-Hurwitz identity
$K_X =f^*K_Y$
induces subtle relations between the Theta divisor $\Theta$ of $X$ and the Theta divisor $\widetilde{\Theta}$ of $\textrm{Prym}(X, \tau)$.
You can look at [Arbarello-Cornalba-Griffiths-Harris, Geometry of algebraic curves, Appendix C] or at [Birkenhake-Lange, Complex Abelian Varieties, Chapter 12] for further details.
In the general case, it is possible to define the so-called generalized Prym varieties, at least where $f \colon X \to Y$ is a tame Galois branched cover. Look for instance at the paper of MERINDOL
"Varietés de Prym d'un revetement galoisien [Prym varieties of a Galois covering]"
Journal Reine Angew. Math. 461 (1995), 49-61.
Let $X$ be the compact Riemann surface and $H$ a finite subgroup of $G=Aut(X)$. Then you can think of two different constructions.
- One is taking the quotient in the category of complex manifolds $X/H$ (which corresponds to the GIT quotient $X//H$ in the algebraic category). Quotients (of complex manifolds by complex Lie groups) don't always exist, so you have to prove that there actually exists a complex manifold $X/H$ which has the desired quotient properties (which is done in your textbook).
- The other construction is considering the quotient orbifold $[X/H]$. In this case, since $H$ is finite, it has always a meaning, and is -indeed- a 1-dimensional complex-analytic orbifold.
The relation between the two constructions is that (in this case of smooth Riemann surfaces and finite groups) you can think of $[X/H]$ as the Riemann surface $X/H$ decorated, at the ramification points of the quotient map $\pi:X\to X/H$, with the stabilizers.
You say:
"We know that only when an group G act freely and properly discontinuously on a manifold $M$, the quotient space $M/G$ is a manifold. Here the question is that the action of automorphism group of a Riemann surface on itself has the fixed points"
I think the correct statement would be that when $G$ acts freely and properly discontinuously there exists a quotient $M/G$ in the category of manifolds, and the quotient orbifold $[M/G]$ (which is the quotient in the category of orbifolds) happens to be a manifold. But there are cases, like the case of $M=\mathbb{C}\mathbb{P}^1$ and $G=${$1,-1$}, in which the action is not free, nevertheless the quotient manifold $M/G$ does exist (in this case it's $\mathbb{C}\mathbb{P}^1$ itself) but is different from the quotient orbifold $[M/G]$ (in this case $[\mathbb{C}\mathbb{P}^1/H]$ is the Riemann sphere decorated at $0$ and $\infty$ with copies of $\mathbb{Z}/2\mathbb{Z}$).
Best Answer
This (existence and uniqueness) is proved in the paper in much more general setting (in fact, the result is due to E. Picard):
M. Heins, ‘On a class of conformal metrics’, Nagoya Math. J. 21 (1962) 1–50.
For a more recent and simpler proof of the special case that you ask, see
MR1479037 (99d:30009) Zakeri, Saeed On critical points of proper holomorphic maps on the unit disk. Bull. London Math. Soc. 30 (1998), no. 1, 62–66.