Hint: start with the fact that $f(x) = x^2-a$ has $\sqrt{a}$ as a root.
Use the Newton's Method formula:
$$x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}$$
where $f(x_n) = x_n^2-a$ and $f'(x_n) = 2 x_n$.
Plug these into the above equation and the first result is obtained with a little algebra. For the second result, note that
$$\sqrt{a}-x_{n+1} = \sqrt{a}-\frac{1}{2} \left (x_n + \frac{a}{x_n} \right )$$
What you can do here is multiply that last piece out and factor out $-1/(2 x_n)$. What you should get is 3 terms that fit a pattern of a binomial squared. Look at the result you are trying to show to guide you.
The formulation of tolerance:
$$ |\text{guess} - c/\text{guess}| > \epsilon \cdot \text{guess} $$
is attractive for two reasons besides the relative precision that it imposes as a termination criterion. Bear in mind that the above is the condition to continue iteration. If the condition is too loose, iteration will terminate prematurely, but if too tight, iteration may never terminate.
First: If guess is the current (possibly initial) approximation, note that when guess is above the square root, $c$/guess will fall below the square root (in exact arithmetic). Conversely if guess is below the square root, then $c$/guess will lie above it. For this reason the true square root should be between these, and the difference of those two computed values should overestimate the actual error (but only by a constant factor $\lt 2$ as convergence approaches).
Second: The Newton iteration can be expressed:
$$ \text{guess} \gets (\text{guess} + c/\text{guess})/2 $$
So we haven't wasted a division operation by computing $c/$*guess* if the outcome of the test is to continue the iteration. This result will be used directly in forming the updated approximation to square root.
Now we come to the application of relative precision in deciding to continue or terminate iteration. Any computation of a positive square root is liable to contain a rounding error resulting from normal floating point operations. The absolute size of this error depends (because of the nature of the floating point format) on the size of the value being computed.
If you take the square root of $16664444$, the result is slightly over $4000$ and the rounding error will accordingly be several orders of magnitude greater than the square root of $1E-50$, which is $1E-25$. So if you impose a test for the absolute value of the error, as proposed with:
$$ |\text{guess} - c/\text{guess}| > \epsilon $$
then the same value of $\epsilon$ that does a reasonable job for $16664444$ will be too loose for $1E-50$, and a value that is right for $1E-50$ will be generally impossible to meet for arguments as large as $16664444$.
Therefore testing the relative value of the error as shown at top is the way to balance accuracy over a wide range of floating point arguments. Note that this test is equivalent (in exact arithmetic) to:
$$ \frac{|\text{guess} - c/\text{guess}|}{\text{guess}} > \epsilon $$
(which should make apparent we are comparing relative error in guess to $\epsilon$), but the form of the test trades off a division for a multiplication in the form shown at top. This would typically be computationally faster, if it mattered (and it might since we are doing the check once for each iteration).
Best Answer
I recommend to check $9788 \times 8$ again.