Let me first point out that you can compute the canonical divisor on a plane curve in a pretty straightforward fashion: by example II.8.20.3, for $X$ a plane curve of degree $d$ we have $\omega_X \cong \mathcal{O}_X(d-3)$, or $K_X\sim(d-3)H$. So in your case with $X$ a plane quintic we have $K\sim 2H$, and so $K$ is very ample. Then by proposition IV.5.2, this means $X$ cannot be hyperelliptic.
I'm not familiar enough with rational scrolls to comment on the rest of Sasha's answer, so let me present a different solution method.
First, reinterpret Riemann-Roch as follows: from $l(D)-l(K-D)=\deg D + 1 -g$, we can rearrange a little to get $l(D)-1=\deg D -1 -(g-1-l(K-D))$. Now we interpret $l(D)-1$ as $\dim |D|$ and $g-1-l(K-D)$ as the dimension of the linear span of $D$ under the canonical embedding: the canonical embedding $\varphi_K$ embeds $C$ in to $\Bbb P^{g-1}$, and $\Gamma(X,\mathcal{O}_X(K-D))$ is the space of hyperplanes vanishing on $\varphi_K(D)$. This gives $$\dim |D| = \deg D - 1 - \dim \overline{\varphi_K(D)},$$ or that the dimension of $|D|$ is the difference between the "expected" dimension of the linear span of $\deg D$ points and the actual dimension of the linear span of $D$ under the canonical embedding. (This is sometimes known as "Geometric Riemann-Roch", but I don't know how common this terminology is.)
Why does this help us solve this problem? If we have a divisor $D$ so that $\deg D = 3$ and $\dim |D|=1$, the formula above says that any effective divisor linearly equivalent to $D$ must lie on a line under the canonical embedding. Since the morphism determined by the $g_3^1$ has to be separable, it's generically unramified, so almost all fibers are three points and each of these fibers must lie on a line under the canonical embedding. We'll show that can't happen by finding a global section of the canonical bundle which vanishes on two of these points but not the third (such a global section would correspond to a hyperplane under the canonical embedding which contains two points but not the third, an impossibility if the three points were on a line).
In our case, $K\sim 2H$, so given three points on our curve we can pretty easily find the required global section: take a quadratic form on $\Bbb P^2$ which hits two points and misses the third, for instance by taking a product of two linear forms.
This generalizes: you can show that no plane curve of degree $d>2$ admits a divisor $D$ with $\dim |D|=1$ and $\deg D < d-1$, assuming $\deg D$ is a prime or coprime to the characteristic.
Best Answer
By contrapositive, it is enough to show that if $X$ is hyperelliptic, then there is no map $X\to\Bbb P^1$ of degree 3. Suppose that there were a map of degree three, and let $D$ be the pullback of the hyperplane class along this map. Then $\deg D=3$ and $\dim |D|=1$, so by Riemann-Roch we would have $l(K-D)=1$, which means that there's a hyperplane in $\Bbb P^{g-1}$ which vanishes on the image of $D$ under the canonical map. But for hyperelliptic curves, the canonical map has image a rational normal curve, which in this case is a conic in $\Bbb P^2$, and by Bezout there is no hyperplane which intersects it in three points.
In general, one may use the base-point free pencil trick (see section 3 of this blog post by Akhil Mathew) to show that a curve of genus $\geq 3$ cannot be simultaneously trigonal and hyperelliptic.
(PS: let me point out that it's perhaps more usual to consider projection from a point as the degree 3 map $X\to\Bbb P^1$ for the first part of the question.)