The dimension of the subvariety $Z$ of the moduli space $M_3$ of genus 3 curves that have a pair of points $P \ne Q$ with $4P \sim 4Q$ is 5, so it is a divisor in $M_3$.
Indeed, the linear system generated by the divisors $4P$ and $4Q$ defines a morphism
$$
f \colon C \to \mathbb{P}^1
$$
which has ramification index 4 at $P$ and $Q$. By Hurwitz formula it follows that $f$ has at most 8 branch points, so the position of these points depend on $8 - 3 = 5$ parameters. The rest of the ramification data is discrete, hence $\dim(Z) \le 5$.
However, the hyperelliptic locus (which also has dimension 5) is not the only component of $Z$. For instance, the curve
$$
x^3y + xy^3 + z^4 = 0
$$
has $4P \sim K_C \sim 4Q$, where $P = (1,0,0)$ and $Q = (0,1,0)$, and is not hyperelliptic.
Edit: thanks to Ted Shifrin who pointed out an error. Riemann-Hurwitz applied to a degree $2$ map from a genus $g$ surface to $\Bbb{P}^1$ gives:
$$
2g - 2 = 2(-2) + \sum_{P\in X} (e_P - 1)
$$
where $e_P$ is the ramification index at $P\in X$. So, $2g+2$ is the number of ramification points (since they all must have $e_P = 2$). Consequently, I think the problem statement should be modified to "$2g + 2$ ramification points". Note that this generalizes the statement for a genus $1$ surface, $E$, which admits a map $E\to \Bbb{P}^1$ ramified at $4$ points. See here.
Your idea is correct. Given a hyperelliptic curve with map $f:X\to \Bbb{P}^1$ as in the definition, you can define a map $\alpha:X\to X$ given by permuting the points in each fibre $f^{-1}(p) = \{q_1,q_2\}$ of cardinality $2$ and fixing each singleton fibre. $\alpha$ is easily seen to be a bijection of $X$ to itself and it is also clearly fibre preserving. To see that $\alpha$ is holomorphic, you can check locally around each point. There are basically two cases:
- $q\in X$ lies in a fibre of cardinality $2$: i.e. $q\in f^{-1}(p) = \{q,q'\}$. In this case, the holomorphic map $f$ is unramified at both $q$ and $q'$ so that there exists a small open neighborhood $U$ of $p$ such that $f|_{f^{-1}(U)}:f^{-1}(U)\to U$ is equivalent to the map $U\times \{1,2\}\to U$ given by $(x,i)\mapsto x$. $\alpha$ maps $f^{-1}(U)$ into itself and acts by $(x,1)\mapsto (x,2)$ and $(x,2)\mapsto (x,1)$. That is, $\alpha$ is (locally) the "switch sheets of the cover" map. So, $\alpha$ is holomorphic near such $q$.
- $q\in X$ lies in a fibre of cardinality $1$: i.e. $q$ is a ramification point for the map $f$. In local analytic coordinates, $f$ is given $z\mapsto z^2$. In this case, $\alpha$ acts by $z\mapsto -z.$ Hence, it is again holomorphic and we are done.
By definition, the ramification points of $f$ are the fixed points of $\alpha$. Next, given such an involution $\alpha$, we basically want to form the quotient of $X$ by the associated $\Bbb{Z}/2\Bbb{Z}$ action. Long story short is that you can, and the associated map $X\to X/\alpha$ is degree $2$ with ramification points at the $2g + 2$ fixed points. An application of Riemann-Hurwitz shows that $X/\alpha$ is genus $0$, i.e. biholomorphic to $\Bbb{P}^1$.
Here is a sketch proof that $X/\alpha$ is well-defined: You can first consider $X_0 = X\setminus \{p_1,\ldots, p_{2g+2}\}$, i.e. the complement of the fixed points. $\alpha$ still acts on this set and in particular it acts freely. Consequently, $X_0/\alpha$ inherits a Riemann surface structure in a unique way compatible with the map $\pi_0:X_0\to X_0/\alpha$. Now, there is a unique way to complete $X_0/\alpha$ to a compact Riemann surface $Y$ so that $\pi_0:X_0\to X_0/\alpha$ extends to a map $\pi:X\to Y = X/\alpha$. The idea is that you map "holes" in $X_0$ to "holes" in $X_0/\alpha$. More precisely, given a curve $\gamma(t)$ in $X$ such that $\lim_{t\to 0} \gamma(t) = p_i$ for one of the $p_i$, define a curve by $(\pi_0\circ \gamma)(t)$ in $X_0/\alpha$. Define $\pi(p_i)\in Y$ in such a way that $\lim_{t\to 0}(\pi_0\circ \gamma)(t) = \pi(p_i)$. This should be enough that you can fill it in yourself if you don't know this already.
An extra comment: if you learn the equivalence of categories between transcendence degree $1$ field extensions over $\Bbb{C}$ (i.e. finite extensions of $\Bbb{C}(t))$ and Riemann surfaces, you will know that a map $X\to \Bbb{P}^1$ corresponds functorially to a field extension $\mathcal{M}(X)$ of $\Bbb{C}(t) = \mathcal{M}(\Bbb{P}^1)$. The degree of the map corresponds to the degree of the extension of fields. So, a hyperelliptic curve $X\to \Bbb{P}^1$ is nothing more than a degree $2$ field extension $\mathcal{M}(X)\supset \Bbb{C}(t)$. Because $[\mathcal{M}(X):\Bbb{C}(t)] = 2$, $\mathrm{Aut}_{\Bbb{C}(t)}(\mathcal{M}(X)) \cong \Bbb{Z}/2\Bbb{Z}$, but then it follows that $\mathrm{Aut}_{\Bbb{P}^1}(X) = \Bbb{Z}/2\Bbb{Z}$. Consequently, there exists an order $2$ automorphism of $X$ commuting with the map $X\to \Bbb{P}^1$. Such a map takes ramification points to themselves, as needed.
Note, though, that in spite of the fancy "functorial" language, the methods from earlier on in the post are really needed to set up this correspondence. This perspective does result in quick answers to these sorts of questions, though.
Best Answer
$\newcommand\P{\mathbb P}\newcommand\msL{\mathscr L}\newcommand\pull[1]{#1^*}\newcommand\msO{\mathscr O}\newcommand\msF{\mathscr F}$ Say $C$ is hyperelliptic with morphism $f:C\xrightarrow2\P^1$, and let $\msL=\pull f\msO(1)$. In other words $H^0(\msL)=L(P+Q)$ where $P,Q$ are the preimages of $\infty\in\P^1$, as you suggest. Our strategy will be to use $f$ to related cohomology groups on $C$ to those on $\P^1$, and then stare at a diagram until the answer pops out.
We have a short exact sequence $0\to\msO_{\P^1}\to\msO_{\P^1}(1)\to\msO_\infty\to0$. Further, it is a general fact that the morphism $f$ induces a (unique) map $H^\bullet(\P^1,-)\to H^\bullet(C,\pull f(-))$ of $\delta$-functors extending the natural map in degree $0$ (i.e. the map $H^0(\P^1,\msF)=\msF(\P^1)\to (\pull f\msF)(C)=H^0(C,\pull f\msF)$). The upshot is that our short exact sequence induces a homomorphism of long exact sequences $$\require{AMScd}\begin{CD} 0 @>>> H^0(\msO_{\P^1}) @>>> H^0(\msO_{\P^1}(1)) @>>> H^0(\msO_\infty) @>>> H^1(\msO_{\P^1}) @>>> H^1(\msO_{\P^1}(1)) @>>> 0\\ & @VVV @VVV @VVV @VVV @VVV\\ 0 @>>> H^0(\pull f\msO_{\P^1}) @>>> H^0(\pull f\msO_{\P^1}(1)) @>>> H^0(\pull f\msO_\infty) @>>> H^1(\pull f\msO_{\P^1}) @>>> H^1(\pull f\msO_{\P^1}(1)) @>>> 0 \end{CD}$$
This is a bit of a mouthful, so let's rewrite this a little. We know that $\pull f\msO_{\P^1}=\msO_C$, that $\pull f\msO_{\P^1}(1)=\msL$, and that $\pull f\msO_\infty=\msO_{P+Q}$. We also know $h^1(\msO_{\P^1})=g(\P^1)=0$ and $h^1(\msO_{\P^1}(1))=h^0(\msO_{\P^1}(-3))=0$ by Serre duality. Letting $k$ be our ground field, we have
$$\begin{CD} 0 @>>> k @>>> k^{\oplus2} @>>> k @>>> 0 @>>> 0 @>>> 0\\ & @VVV @VVV @VVV @VVV @VVV\\ 0 @>>> k @>>> H^0(\msL) @>>> k^{\oplus2} @>>> k^{\oplus g} @>>> H^1(\msL) @>>> 0 \end{CD}$$ which now has a shorter scroll bar. Now, we're in luck. The map $$k=H^0(\msO_\infty)\to H^0(\msO_{P+Q})=k^{\oplus2}$$ above is injective; it sends a function on the point $\infty\in\P^1$ to the induced function on the subscheme $P+Q\subset C$. Thus, commutativity of the square (with top row $k\to0$) shows that $$\dim\mathrm{im}(H^0(\msL)\to k^{\oplus2})=\dim\ker(k^{\oplus2}\to k^{\oplus g})\ge1$$ and $$\dim\ker(k^{\oplus g}\to H^1(\msL))=\dim\mathrm{im}(k^{\oplus2}\to k^{\oplus g})\le1.$$
Since $k^{\oplus g}\twoheadrightarrow H^1(\msL)$ is surjective, we see that $h^1(\msL)\le g-1$. Hence, $$h^0(\msL)=\chi(\msL)+h^1(\msL)=(3-g)+h^1(\msL)\le2.$$ At the same time, exactness of the bottom row of our diagram + the bound $\dim\mathrm{im}(H^0(\msL)\to k^{\oplus2})\ge1$ shows that $h^0(\msL)\ge2$
Edit: To anyone looking at this answer, user10354138 has a comment on the question giving a much simpler proof that $h^0(\msL)\le2$.