Assume that $\mathbf{J} = \{ J \}_{ij}$ are centered independent standard Gaussian with variance $1/n$ random variables for $i, j = 1, \dots, n$. Why is the function $\|\mathbf{J}\|_{\infty}$ then $1/n$-Lipschitz w.r.t to the Euclidean norm, that is,
$$
\left|
\sup_{\|u\| = 1} \langle u, \mathbf{J}u \rangle
– \sup_{\|v\| = 1} \langle v, \mathbf{J}v \rangle
\right|
\leq
\frac{1}{n} \|u – v\|_2 .
$$
Thus, it has a sub-Gaussian tail which is
$$
P(\|\mathbf{J}\|_{\infty} – E[\|\mathbf{J}\|_{\infty}] \geq x)
\leq
e^{-n x^2 / 2}.
$$
The original statement is as follows.
Moreover, it is easy to see that
$$
\|\mathbf{J}\|_\infty = \sup_{\|u\| = 1} \langle u, \mathbf{J} u \rangle
$$
has sub-Gaussian tail.
Indeed, is is a Lipschitz function of the Gaussian entries of $\mathbf{J}$ (with respect to the Euclidean norm) with constant bounded by $N^{-1/2}$ so that by Herbst argument (see [24, 25]), we have the concentration inequality
$$
P(\|\mathbf{J}\|_\infty \geq E[\|\mathbf{J}\|_\infty] + x)
\leq
e^{-\frac{1}{2} N x^2} .
$$
Best Answer
$\|\mathbf{J}\|_{\infty}$ can be considered as a function of $\{\mathbf{J}_{i,j}\}_{i,j\in [n]^2}$, denoted it by $f(\mathbf{J}_{1,1},\dots,\mathbf{J}_{n,n})$. Let $\mathbf{J}_{vec}=(\mathbf{J}_{1,1},\dots,\mathbf{J}_{n,n})\in \mathbb{R}^{n^2}$ represents the unrolled version of $\mathbf{J}$. Based on Theorem 5.8 in lecture note [1], if we manage to show that $\|\mathbf{J}\|_{\infty}$ is $1$-Lipschitz in the sense that $$ \|\mathbf{J}\|_{\infty} - \|\mathbf{J'}\|_{\infty} \leq \| \mathbf{J}_{vec}- \mathbf{J'}_{vec} \|_2 $$
Then, we are done. Consider
$$ \sup_{||u||=1}<u, \mathbf{J}u>-\sup_{||v||=1}<v, \mathbf{J'}v> \\ \leq \sup_{||u||=1}<u, \mathbf{J}u>-<u, \mathbf{J'}u> $$ Then, $\sup_{||u||=1}<u, \mathbf{J}u>-<u, \mathbf{J'}u>=\sup_{||u||=1}<u, (\mathbf{J}-\mathbf{J'})u> = \|\mathbf{J}-\mathbf{J'}\|_2$
where $\|\mathbf{J}-\mathbf{J'}\|_2$ is the matrix $\ell_2$ norm. Then, by a standard result (for a proof see [2]) we have $\|\mathbf{J}-\mathbf{J'}\|_2 \leq \|\mathbf{J}-\mathbf{J'}\|_F$ where $\|\mathbf{J}-\mathbf{J'}\|_F$ is the Frobenius norm. Finally note that $\|\mathbf{J}-\mathbf{J'}\|_F = \| \mathbf{J}_{vec}- \mathbf{J'}_{vec} \|_2$
[1]https://www.stat.cmu.edu/~arinaldo/Teaching/36755/F16/Scribed_Lectures/LEC0914.pdf
[2] Why is the Frobenius norm of a matrix greater than or equal to the spectral norm?