If $x_n$ is a Cauchy sequence, then uniform continuity of $f$ allows us to conclude that the sequence $f(x_n)$ is also Cauchy.
The sequences $a_n=a+\frac{1}{n}$ and $b_n =b-\frac{1}{n}$ are Cauchy, hence so are the sequences $f(a_n)$, $f(b_n)$. Since $\mathbb{R}$ is complete, these sequences converge to some numbers $f_a, f_b$ respectively. Define the function $\overline{f}:[a,b] \to \mathbb{R}$ by $\overline{f}(a) = f_a$, $\overline{f}(b) = f_b$ and $\overline{f}(x) = f(x)$ for $x \in (a,b)$. Clearly $\overline{f}$ is continuous in $(a,b)$, it only remains to show continuity at $a,b$.
Suppose $x_n\in [a,b]$, and $x_n \to a$. We have $|\overline{f}(x_n) - \overline{f}_a| \leq |\overline{f}(x_n) - \overline{f}(a_n)| + | \overline{f}(a_n) - \overline{f}_a|$. Let $\epsilon >0$, then by uniform continuity, there exists $\delta>0$ such that if $|x-y|< \delta$, with $x,y \in (a,b)$, then $|f(x)-f(y)| < \epsilon$. Choose $n$ large enough such that $|x_n-a_n| < \delta$, and $| f(a_n) - f_a| < \epsilon$. If $x_n = a$, then $|\overline{f}(x_n) - \overline{f}_a| = 0$, otherwise we have $|\overline{f}(x_n) - \overline{f}_a| \leq |f(x_n) - f(a_n)| + | f(a_n) - f_a| < 2 \epsilon$. Consequently $\overline{f}(x_n) \to \overline{f}_a$, hence $\overline{f}$ is continuous at $a$. Similarly for $b$.
For the second case, take $f(x) = \frac{1}{x-a}$. Then $f$ is continuous on $(a,b)$, but the domain cannot be extended to $[a,b]$ while keeping $f$ continuous, and $\mathbb{R}$ valued. To prove this, take the sequence $a_n$ above, then $f(a_n) = n$, and clearly $\lim_n f(a_n) = \infty$. If $f$ could be continuously extended to $\overline{f}$, then $\overline{f}(a) \in \mathbb{R}$, which would be a contradiction.
Let $g:A\to\mathbb{R}$ and $*$ be the unique point added to $A$, i.e. $B=A\cup\{*\}$.
"$\Rightarrow$" Let $g':B\to\mathbb{R}$ be the extension of $g$. Put $c:=g'(*)$. We will show that $g-c$ vanishes.
So pick $\epsilon > 0$ and define
$$U=\{b\in B\ |\ |g'(b)-c| < \epsilon\}$$
So $U=f^{-1}(-\infty, \epsilon)$ for $f(b)=|g'(b)-c|$. Since $g'$ is continous then so is $f$ and thus $U$ is open in $B$.
Therefore $C:=B\backslash U$ is compact (being a closed subset of a compact space). Note that $*\not\in C$ since $*\in U$ because $|g'(*)-c|=0<\epsilon$. Thus $C=A\cap C$ and therefore $A\cap C$ is a compact subset of $A$ that satisfies desired properties. This completes the proof. $\Box$
"$\Leftarrow$" Let $c\in\mathbb{R}$ be such that $g-c$ vanishes. Define
$$g':B\to\mathbb{R}$$
$$g'(x)=\begin{cases}
g(x) & \mbox{ if } x\in A \\
c & \mbox{ if } x= *
\end{cases}$$
Obviously $g'$ extends $g$. All that is left is to prove that $g'$ is continous. Can you complete the proof?
So it is enough to prove that if $c\in(a,b)$ for some $a,b\in\mathbb{R}$ then $g'^{-1}(a,b)$ is open in $B$. Let $x\in g'^{-1}(a,b)$. If $x\neq *$ then there exists an open neighbourhood $U$ of $x$ in $A$ (and thus in $B$) such that $g'(U)=g(U)\subseteq (a,b)$ (by continuity of $g$). Therefore the only thing that we actually have to prove is the existence of a neighbourhood $U$ of $*$ such that $g'(U)\subseteq (a,b)$.
Take $\epsilon := \min\big(|b-c|, |a-c|\big) > 0$. Since $g-c$ vanishes then there exists a compact subset $C\subset A$ such that
$$g(A\backslash C)\subseteq (c-\epsilon, c+\epsilon)\subseteq (a,b)$$
By the definition of the one point compactification $U:=A\backslash C\cup\{*\}$ is open in $B$ and since $g'(*)=c$ then $g'(U)\subseteq (a,b)$ which completest the proof. $\Box$
Best Answer
This function cannot be extended continuously to the entire plane $\mathbb{R}^2$, because there are points on the line $x=0$ such that your function $f$ approaches different limits from different directions.
For example, consider the point $P=(0,0)$, which does not lie in the domain of $f$, and consider the limit of $f(x,y)$ as $(x,y)$ approaches this point along the two lines $y=0$ and $y=x$. Clearly $f$ is defined at all points on these lines except $P$.
On the line $y=0$, $f(x,y)=f(x,0)=\frac{1}{x}sin(\frac{x^3}{x^2+0})=\frac{1}{x}sin(x)=\frac{sin(x)}{x}$.
So using l'Hopitals rule, we see that $lim_{x\rightarrow 0}\frac{sin(x)}{x}=lim_{x\rightarrow 0}\frac{cos(x)}{1}=cos(0)=1$.
Similarly, on the line $y=x$, $f(x,y)=f(x,x)=\frac{1}{x}sin(\frac{x^3}{x^2+x^2})=\frac{1}{x}sin(\frac{1}{2}x)$,
and again, using l'Hopital's rule, $lim_{x\rightarrow 0}\frac{sin(\frac{1}{2}x)}{x}=lim_{x\rightarrow 0}\frac{\frac{1}{2}cos(\frac{1}{2}x)}{1}=\frac{1}{2}cos(0)=\frac{1}{2}$.
This proves that the limit of $f$ as $(x,y)$ approaches $P$ is undefined, and so $f$ cannot be extended continuously to the whole plane.