Hmm.
\begin{align*}
y&=\frac{1}{x}+\frac{1}{x^2}-\frac{1}{x^3} \\
&=\frac{x^2+x-1}{x^3} \\
y'&=\frac{x^3(2x+1)-3x^2(x^2+x-1)}{x^6} \\
&=\frac{x(2x+1)-3(x^2+x-1)}{x^4} \\
&=\frac{2x^2+x-3x^2-3x+3}{x^4} \\
&=\frac{-x^2-2x+3}{x^4}.
\end{align*}
Setting this equal to zero is tantamount to solving $x^2+2x-3=0,$ with solutions
$$x=\frac{-2\pm\sqrt{4+4(3)}}{2}=\frac{-2\pm 4}{2}=\{1, -3\}. $$
You can probably use the Second Derivative Test to show which of these is a local min, and which a local max.
You have $Q(x) = \langle x,Ax \rangle$ and the function $g(x) = \langle x,x \rangle -1$, and you want to find extrema of $Q$ when restricted to the level set $g^{-1}(\{0\})$. First, of all, you can express the standard real-inner product using transpose as well:
\begin{equation}
Q(x) = x^t A x \qquad \text{and} \qquad g(x) = x^tx - 1
\end{equation}
Now, you want to set $\nabla (Q - \lambda g)(x) = 0$, and find the points $x$ which satisfy this equation. (your equation (1) left out the "$=0$", and uses $\Delta$ rather than $\nabla$)
As a personal preference, I'd much rather work with the notation of inner products rather than transpose. So, the gradient of $Q$ at $x$ is the following $1 \times n$ matrix:
\begin{equation}
\nabla Q(x) = 2 \left(\langle Ax,e_1\rangle, \dots, \langle Ax,e_n \rangle \right)
\end{equation}
You can derive this either by calculating each $\dfrac{\partial Q}{\partial x_i}(x)$ by writing out $Q(x)$ entirely in terms of components, or in any other way you like, but in any case, you will have to use the fact that $A^t = A$ somewhere. Also, we have that
\begin{equation}
\nabla g(x) = 2 \left(\langle x,e_1\rangle, \dots, \langle x,e_n \rangle \right)
\end{equation}
This can be simplified to $2(x_1, \dots, x_n)$, but leaving it like this makes the next step clearer (step (3)).
The lagrange multipliers method now says to set $\nabla Q(x) - \lambda \nabla g(x) = 0$. If you do this, you will notice that for every $1 \leq i \leq n$, we have
\begin{equation}
\langle Ax - \lambda x, e_i \rangle = 0
\end{equation}
This says that for every $i$, the $i^{th}$ component of $Ax - \lambda x$ is $0$. Thus, it follows that
\begin{equation}
Ax - \lambda x = 0.
\end{equation}
Now, since $x$ lies in the level set $g^{-1}(\{ 0\}),$ it follows that $\lVert x \rVert = 1$, so that in particular, $x \neq 0$. This tells you $x$ is in fact an eigenvector of $A$.
So, as a recap, the maxima and minima of $Q$ on the level set of $g$ are attained when you evaluate $Q$ on an eigenvector of $A$. You can in fact show that if $Q(\xi) = \lambda \xi$ is the maximum value of $Q$ on the level set, then $\lambda$ is the largest eigenvalue of $A$, and likewise for the minimum.
Best Answer
Note that you can factor the derivative as
$$ \frac{\rm d}{{\rm d}x} f = \tfrac{1}{t} ( 3x -2 t) ( 3 t x + 2) = 0 $$
which gives you the $x$ values for the extrema
$$ \begin{gathered} x = \frac{2 t}{3} & x = -\frac{2}{3 t} \end{gathered} $$
For the next part, you can do a variable substitution with $ u = t + \frac{1}{t}$ since the answer you are looking for is in terms of $u$.
$$ f = (x - \sqrt{ u^2-4}) ( 3 x^2 - 4) $$
and
$$ \begin{gathered} x = \frac{\sqrt{u^2-4}-u}{3} & x = \frac{\sqrt{u^2-4}+u}{3} \end{gathered} $$
At this point plug in the extrema $x$ into $f$ and simplify. Take the difference between the two results.