Given function $f\colon \mathbb{R}\to \mathbb{R}$ be defined by
$$
f(x)=\begin{cases}
e^{-\frac{1}{x^2}} & \text{if $x > 0$;}\\
0 & \text{if $x \leq 0$.}
\end{cases}
$$
a) Show that $g(x)=f(x-a)f(b-x)$ is a smooth function, positive on $(a,b)$ for $a<b$ and zero else where then
$$
h(x)=\frac{\displaystyle\int_{-\infty}^{x} g(t)\,dt}{\displaystyle\int_{-\infty}^{\infty} g(t)\,dt}
$$
is smooth function satisfying $h(x)=0$ for $x<a$, $h(x)=1$ for $x>b$ and $0<h(x)<1$ for $x \in (a,b)$.
b) Construct a smooth function on $\mathbb{R}^k$ that equals 1 on the ball of radius $a$ and zero outside the ball of radius $b$ and is strictly between $0$ and $1$ at intermediate points $(0<a<b)$
Here is what I got so far
First I need to show that $f(x)$ is smooth.
For $x < 0$, $f(x)$ is differentiable. For $x>0$, same thing happen. For $x=0$
$\displaystyle\lim _{h\to0^-} \frac{e^{-\frac{1}{h^2}}}{h}=0$ also by L'Hospital rule
$\displaystyle\lim _{h\to0^+} \frac{e^{-\frac{1}{h^2}}}{h}=0$ by L'Hospital rule
So for $x=0$, $f(x)$ also differentiable. Now we need to check if $f'$ is continuous.
For $x>0$ and $x<0$, f(x) is continuous, by calculus rule. At $x=0$ we have
$f'(x)= -\frac {2}{x^3} e^{-\frac{1}{x^2}}$
so by L'Hospital rule
$\displaystyle \lim _{x\to0^+} -\frac {2}{x^3} e^{-\frac{1}{x^2}}=0=\lim _{x\to0^-} -\frac {2}{x^3} e^{-\frac{1}{x^2}}$
so $f'$ is continuous at $x=0$. Hence $f(x)$ is smooth. Since $f(x)$ is smooth $g(x)$ is smooth.
Now I'm stuck
Best Answer
I will try to not only give you a solution but also motivate it a bit:
a) Smoothness of $f$: To show that $f$ is smooth, we have to show that all derivatives $f^{(n)}$ exist. You have already found out that $f'(x) = \frac{-2}{x^3} e^{-1/x^2}$. For $x > 0$, we then get $$f''(x) = \frac6{x^4} e^{-1/x^2} + \frac{-2}{x^3} \cdot \frac{-2}{x^3} e^{-1/x^2} = \frac{6x^2 + 4}{x^6} e^{-1/x^2}.$$ So we have, for $x > 0$, \begin{align*} f(x) &= r_0(x) \cdot e^{-1/x^2} \quad \text{with $r_0(x) = 1$},\\ f'(x) &= r_1(x) \cdot e^{-1/x^2} \quad \text{with $r_1(x) = \frac{-2}{x^3}$},\\ f''(x) &= r_2(x) \cdot e^{-1/x^2} \quad \text{with $r_2(x) = \frac{6x^2 + 4}{x^6}$}. \end{align*} The $r_i$ are rational functions (quotients of polynomials) and it is easy to see that the derivative of a rational function is again a rational function whose denominator has no additional zeros.
In the above equations there clearly is a pattern. So let us conjecture the following: For all $n \ge 0$ and all $x > 0$, $f^{(n)}(x)$ exists and has the form $f{(n)}(x) = r_n(x) \cdot e^{-1/x^2}$ where $r_n$ is a rational function whose denominator is some power of $x$.
Lets prove that by induction on $n$: The base case $n = 0$ is clear. Now assume that for some $n$ the assertion is correct. Then we get for all $x > 0$: \begin{align*} f^{(n+1)}(x) = (f^{(n)})'(x) = \left(r_n(x) \cdot e^{-1/x^2}\right)' = r_n'(x) \cdot e^{-1/x^2} + r_n \cdot \frac{-2}{x^3} e^{-1/x^2} = \left(r_n'(x) - \frac{2 r_n(x)}{x^3}\right) e^{-1/x^2}. \end{align*} Define $r_{n+1}(x)$ to be the left factor in this term. Then it is not too hard to check that $r_{n+1}$ is a rational function whose denominator is a power of $x$. In the following, let us write $r_n(x) = p_n(x) / x^{k_n}$ for some polynomial $p_n$.
So far we have shown that $f$ is smooth for every $x > 0$. Also, it is trivial that $f$ is smooth for $x < 0$ and that all derivatives are zero there. Thus it remains to be shown that $f$ is also smooth at $0$. So let us conjecture that $f^{(n)}(0)$ exists and is equal to $0$. This is true for $n = 0$. Now suppose it is true for some $n$. Then $$f^{(n+1)}(0) = \lim_{h \to 0} \frac{f^{(n)}(h)}{h} = \lim_{h \to 0} \frac{r_n(h)}{h \cdot e^{1/x^2}} = \lim_{h \to 0} \frac{p_n(h)}{h^{1+k_n} \cdot e^{1/x^2}} = 0.$$ The last step is l'Hospital's rule (applied multiple times).
Hence $f$ is smooth.
The properties of $g$ are clear. The properties of $h$ are also not too hard to see. Not that the infinite integral $\int_{\infty}^{\infty} g(x) dx$ is actually finite, since $g$ is zero outside of $(a, b)$.
b) Lets first consider the case $k = 1$. Take $F = 1 - h$ on [a, b]. Then $F$ is a smooth function with $F(x) = 1$ for $x = a$, $1 > F(x) > 0$ for $x \in (a, b)$ and $F(x) = 0$ for $x = b$. Thus $F$ has the desired properties for $x \in [a, b]$. We extend $F$ by saying $F(x) = 1$ for $|x| < a$, $F(x) = 0$ for $|x| > b$ and $F(x) = F(|x|)$ for $x \in [-b, -a]$.
Check that this $F$ is smooth. Then certainly $F$ is the function we are looking for.
Now, for a general $k$, we can try a similar thing. On $\Bbb R^k$ we have the norm $\|x\|$ which is smooth for $x \ne 0$. Using this norm and the function $F$, can you find a solution for b)?