This problem can be done without any calculations whatsoever just by understanding the graph of an ellipse in standard form. However just for fun I provide a calculation using calculus without the use of any transcendental functions.
We are trying to find the points on the ellipse, $ \frac{x^2}{4}+\frac{y^2}{9} = 1$, which are closest to and farthest away from the origin. This means that we must optimize the expression $d^2 = x^2+y^2$ subject to the given constraint. To do this we first introduce the parameterization,
$$ x = 2 \cdot \frac{2t}{1+t^2} \qquad y=3 \cdot \frac{1-t^2}{1+t^2}.$$
Substituting our parameterization into the expression for $d^2$ we get,
$$ d^2 = \frac{16 t^2 + 9(1-t^2)^2}{(1+t^2)^2}.$$
Now we look for critical points by examining $(d^2)^\prime$,
$$ (d^2)' = \frac{d}{dt} \frac{16 t^2 + 9(1-t^2)^2}{(1+t^2)^2} = \frac{40(t)(t^2-1)}{(t^2+1)^3},$$
the above expression is equal to zero when $t=0,1,-1$ or $t=\pm \infty$. To determine which values correspond to the minimum and maximum respectively we simply evaluate $x$ and $y$ for all the different values of $t$. We will manually pick out the values which are closest to and furthest from the origin.
$$x(0) = 0 \quad y(0)=3$$
$$x(1) = 2 \quad y(1)=0$$
$$x(-1) = -2 \quad y(-1)=0$$
$$x(\infty) = 0 \quad y(\infty) = -3$$
$$x(-\infty) = 0 \quad y(-\infty) = -3$$
The points furthest from the origin are $(0,3)$ and $(0,-3)$. The points closest to the origin are $(2,0)$ and $(-2,0)$.
Edit In response to the comment:
$t$ is just a parameter which labels the points on the ellipse. To understand the validity of this parametrization note that,
$$ \frac{1}{4} \left( 2 \cdot \frac{2t}{1+t^2} \right)^2
+ \frac{1}{9} \left( 3 \cdot \frac{1-t^2}{1+t^2}\right)^2
=1.$$
For a proof just google "rational points on the circle".
Its not necessary to do it this way I just happen to enjoy this rational parametrization. What you are suggesting about substituting $y^2$ as a function of $x$ into the $d^2$ equation will work just as well. Also you could parameterize with $x=2\sin \theta$ and $y=3\cos \theta$.
Lagrange multipliers do work for your problem, but that involves solving three simultaneous non-linear equations in three unknowns. There is another way that uses only one variable: parameterization.
Get a parameterization that describes the given curve in terms of only one variable. In the case of your ellipse you can use
$$x=\cos t, \quad y=\frac 17\sin t, \quad 0\le t\le 2\pi$$
Now you want to want to find the extrema of
$$\hat f(t)=4x+y=4\cos t+\frac 17\sin t$$
There are several ways to find the extrema of that, using calculus or just trigonometry. Here is a calculus way:
$$\hat f'(t)=-4\sin t+\frac 17\cos t=0$$
$$28\sin t=\cos t$$
$$\tan t=\frac 1{28}$$
$$\cos t=\sqrt{\frac 1{\tan^2 t+1}}=\pm\frac{28}{\sqrt{785}}$$
$$\sin t=\sqrt{1-\cos^2 t}=\pm\frac{1}{\sqrt{785}}$$
$$x=\cos t=\pm\frac{28}{\sqrt{785}}$$
$$y=\frac 17\sin t=\pm\frac{1}{7\sqrt{785}}$$
where $x$ and $y$ have the same sign. This means the maximum of $f$ is
$$4x+y=4\frac{28}{\sqrt{785}}+\frac{1}{7\sqrt{785}}=\frac{785}{7\sqrt{785}}=\frac{\sqrt{785}}{7}\approx 4.00255$$
and the minimum is the negative of that. This graph confirms the maximum.
Best Answer
I found where it was in my notes. We went over it very briefly, so it was a small section. Here's what you need to do.
$$ \frac{\partial{f}}{\partial{m}} = \sum_{i=1}^{5}2(y_i-mx_i-b)*(-x_i)\\ = -2(\sum x_iy_i-m\sum x_i^2-b \sum 1) = 0\\ \frac{\partial{f}}{\partial{b}} = -2(\sum y_i -mx_i-b)\\ -2(\sum y_i -m\sum x_i-b\sum i) = 0\\ \text{now we do some algebraic manipulations}\\ m\sum x_i^2 + b\sum x_i = \sum x_iy_i\\ m\sum x_i+b\sum1=\sum y_i\\ \text{and now we can solve for parts of these equations}\\ \sum_{i = 1}^{5} 1=5\\ \sum x_i = 17\\ \sum x_i^2 = 63\\ \sum y_i = 17 \\ \sum x_iy_i = 66\\ \text{We now get two equations with two variables we can solve for}\\ Eqn 1: 63m + 17b = 66\\ Eqn 2: 17m + 5b = 17\\ \text{Solve for b in Eqn 2 yields}\\ b = \frac{17}{5}(1-m)\\ \text{Plug into Eqn 1 and solve for m}\\ m = \frac{41}{26} \\ \text{Which makes b}\\ b = \frac{17}{5}-\frac{17*41}{5*26} $$
For the final portion, $F(m,b)$ you just need to plug in all the values and do the addition.