Suppose that both $f$ and $g$ are in $L^{1}$. Then $\hat{f}$ and $\hat{g}$ are bounded and, therefore, $\hat{f}g$, $f\hat{g}$ are in $L^{1}$. And,
\begin{align}
\int_{-\infty}^{\infty}\hat{f}(s)g(s)ds & =\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}f(t)e^{-ist}dt g(s)ds \\
& = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}f(t)\int_{-\infty}^{\infty}e^{-ist}g(s)ds \\
& = \int_{-\infty}^{\infty}f(t)\hat{g}(t)dt.
\end{align}
Suppose $f,g \in \mathcal{C}_{c}^{\infty}(\mathbb{R})$ (i.e., are compactly supported $C^{\infty}$ functions.) Then $f,g,\mathcal{F}f,\mathcal{F}g$ are in $L^{1}$, and the above gives
$$
\int_{-\infty}^{\infty}\mathcal{F}f\overline{\mathcal{F}g}ds=
\int_{-\infty}^{\infty}\mathcal{F}f\mathcal{F^{-1}}\overline{g}ds
= \int_{-\infty}^{\infty}f\mathcal{F}\mathcal{F^{-1}}\overline{g}dt
= \int_{-\infty}^{\infty}f\overline{g}dt
$$
Therefore, $\|\mathcal{F}f\|=\|f\|$. Replace $g$ by $\mathcal{F}^{-1}g$ in the above:
$$
(\mathcal{F}f,g)=(f,\mathcal{F}^{-1}g).
$$
Therefore $\mathcal{F}$, $\mathcal{F}^{-1}$ extend by continuity to isometries on $L^{2}(\mathbb{R})$, and the last equality similarly extends, which gives $\mathcal{F}^{\star}=\mathcal{F}^{-1}$ and $(\mathcal{F}^{-1})^{\star}=\mathcal{F}$. So $\mathcal{F}$ is unitary.
While removing the singularity at the origin could give a better decrease at infinity for the Fourier transform, it depends on the details. The issue is that multiplying by a characteristic function (which is rough) also introduces relatively slow decay for the Fourier transform. The rate of decay reflects the geometry of the discontinuity, i.e. of the boundary of the ball.
In the case of $\| \cdot \|_{\infty}$, where the ball is a box with flat sides, the Fourier transform decays on the order of $| \omega |^{-1}$ in the directions perpendicular to the sides of the box. Other directions decay on the order of $| \omega |^{-d}$, though not uniformly.
If you use $\| \cdot \|_2$, so that the ball is round, the Fourier transform decays on the order of $| \omega |^{-(d+1)/2}$ in all directions.
So depending on the value of $\alpha$ and the direction of $\omega$, this may be better or worse than the initial decay of $| \omega |^{-(d - \alpha)}$.
Without going into a ton of detail, one way you can show these decay rates is to first write the distribution as $G = \psi G + (1 - \psi) G$, where $\psi$ is a smooth cutoff function that's $1$ on a neighborhood of the ball (whichever norm is being used to define the ball). By using a dyadic partition of unity, you can show that $\mathcal{F}((1 - \psi) G)$ decays rapidly, i.e. on the order of $| \omega |^{-N}$ for any $N$.
For the Fourier transform of $\psi G$, we can express it as an integral since $\psi G$ is given by an integrable (bounded and compactly supported) function:
$$\mathcal{F}(\psi G)(\omega) = \int e^{-i \omega \cdot x} \psi(x) G(x) \, dx,$$
or whatever your preferred normalization is.
Now fix a particular direction for $\omega$, and integrate out in $x$ along hyperplanes orthogonal to $\omega$. This leaves a one-dimensional integral of the form
$$\int e^{-i |\omega| t} h(t) \, dt,$$
where $h$ is compactly supported.
The smoothness properties of $h$ are important to note here.
- In the case of $\| \cdot \|_{\infty}$ and $\omega$ orthogonal to one of the faces of the box, $h$ is actually discontinuous at $t = \pm 1$.
- In the case of $\| \cdot \|_{\infty}$ and $\omega$ not orthogonal to one of the faces of the box, $h$ is $C^{d-2}$.
- In the case of $\| \cdot \|_{2}$, we have that $h$ "looks like" $(1 - t^2)^{(d-1)/2}$ near $t = \pm 1$. That is, the failure of $h$ to be $C^{\infty}$ is owing to a "kink" with properties like that of $(1 - t^2)^{(d-1)/2}$.
The final step for getting the claimed decay rates is then (possibly repeated) integration by parts in the one-dimensional integral.
Edit:
To see that $\mathcal{F}((1 - \psi) G)$ decays rapidly, let's first set up notation for the dyadic partition of unity. We can write this as
$$1 =
\Psi_0(x) + \sum_{k = 1}^{\infty} \Psi(2^{-k} x).$$
Here $\Psi_0$ is 1 close to the origin (it's convenient to make it so that $\Psi_0 \equiv 1$ on the set where $1 - \psi \neq 1$), and $\Psi$ is 1 on a dyadic annulus. (I'm leaving things a little vague intentionally, as some details could differ between the case of $\| \cdot \|_{\infty}$ and $\| \cdot \|_2$.)
So now we'll have
$$\mathcal{F}((1 - \psi) G)
= \mathcal{F}(\Psi_0 (1 - \psi) G)
+ \sum_{k = 1}^{\infty} \mathcal{F}(\Psi(2^{-k} \cdot) G).$$
Note that we're able to drop the $1 - \psi$ from the terms in the infinite sum since $1 - \psi \equiv 1$ in the relevant regions. Also, each function on the right that we're taking the Fourier transform of is $C_c^{\infty}$, so the individual Fourier transforms are Schwartz functions with good decay. The remaining piece is to see that the good decay isn't messed up by the infinite summation.
With that setup out of the way, the key is to use the homogeneity of $G$ to express everything in terms of dilates of a single function:
$$\Psi(2^{-k} x) G(x)
= 2^{-\alpha k} \Psi(2^{-k} x) G(2^{-k} x)
= 2^{-\alpha k} G_0(2^{-k} x).$$
Now the infinite sum can be expressed in terms of $\mathcal{F}(G_0)$ (which has good decay), taking into account how the Fourier transform interacts with dilations.
Best Answer
The Fourier transform can be diagonalised. Let $h_n$ be the Hermite functions, then we have $$ \mathcal F[h_n] = (-i)^n h_n. $$ So a good definition for a square root of the Fourier transform would be $$ \mathcal G[f] = \sum_{n=0}^\infty \langle f, h_n \rangle \mathcal G[h_n ] = \sum_{n=0}^\infty \langle f, h_n \rangle (-i)^{n/2} h_n. $$ A closed form expression can be found in the other answers.