The delta distribution has the property $f(x)\delta(x)=f(0)\delta(x)$. Thus it doesn't matter whether you use $E$ or $E'$, since in either case you have $2\sqrt E\delta(E-E')=2\sqrt{E'}\delta(E-E')$.
[Edit in response to the comment:]
You seem to be missing a number of factors here.
First, since $E=p^2/2m$ and $p=\hbar k$, it should be $k=\sqrt{2mE}/\hbar$, with $\hbar$ where you had $h$.
Then you dropped all the factors in converting the delta functions. The relationship you need is
$$\delta\left(\frac{\sqrt{2mE}}\hbar-\frac{\sqrt{2mE'}}\hbar\right)=\sqrt{\frac{2E}m}\hbar\delta(E-E')\;.$$
Finally, you're missing a factor of $2\pi$ from the normalization of the momentum eigenfunctions themselves,
$$
\int_{-\infty}^\infty\mathrm e^{\mathrm i(k-k')x/\hbar}\,\mathrm dx=2\pi\delta(k-k')\;.
$$
Putting this all together, we have
$$
\begin{align}
\langle E'|E\rangle
&=
\int_{-\infty}^\infty \psi_{E'}^*(x)\psi_E(x)\,\mathrm dx
\\
&=
N(E')^*N(E)\int_{-\infty}^\infty \mathrm e^{\mathrm i(k-k')x}\,\mathrm dx
\\
&=
N(E')^*N(E)2\pi\delta(k-k')
\\
&=
|N(E)|^22\pi\sqrt{\frac{2E}m}\hbar\,\delta(E-E')\;.
\end{align}
$$
If you want this to be $\delta(E-E')$, you need
$$
\begin{align}
|N(E)|
&=
\left(\frac m{2E}\right)^{1/4}\frac1{\sqrt{2\pi\hbar}}
\\
&=
\left(\frac m{2Eh^2}\right)^{1/4}\;.
\end{align}$$
$\newcommand{\+}{^{\dagger}}
\newcommand{\angles}[1]{\left\langle\, #1 \,\right\rangle}
\newcommand{\braces}[1]{\left\lbrace\, #1 \,\right\rbrace}
\newcommand{\bracks}[1]{\left\lbrack\, #1 \,\right\rbrack}
\newcommand{\ceil}[1]{\,\left\lceil\, #1 \,\right\rceil\,}
\newcommand{\dd}{{\rm d}}
\newcommand{\down}{\downarrow}
\newcommand{\ds}[1]{\displaystyle{#1}}
\newcommand{\expo}[1]{\,{\rm e}^{#1}\,}
\newcommand{\fermi}{\,{\rm f}}
\newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,}
\newcommand{\half}{{1 \over 2}}
\newcommand{\ic}{{\rm i}}
\newcommand{\iff}{\Longleftrightarrow}
\newcommand{\imp}{\Longrightarrow}
\newcommand{\isdiv}{\,\left.\right\vert\,}
\newcommand{\ket}[1]{\left\vert #1\right\rangle}
\newcommand{\ol}[1]{\overline{#1}}
\newcommand{\pars}[1]{\left(\, #1 \,\right)}
\newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
\newcommand{\pp}{{\cal P}}
\newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,}
\newcommand{\sech}{\,{\rm sech}}
\newcommand{\sgn}{\,{\rm sgn}}
\newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}}
\newcommand{\ul}[1]{\underline{#1}}
\newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}
\newcommand{\wt}[1]{\widetilde{#1}}$
$$
\mbox{In spherical coordinates,}\quad
\delta\pars{\vec{r}}={\delta\pars{r}\delta\pars{\cos\pars{\theta}}\delta\pars{\phi} \over r^{2}}\quad
\mbox{such that}
$$
\begin{align}
\color{#66f}{\large\int_{{\mathbb R}^{3}}\delta\pars{\vec{r}}\,\dd^{3}\vec{r}}
&=\int_{0^{-}}^{\infty}\dd r\,r^{2}\int_{0}^{\pi}\dd\theta\,\sin\pars{\theta}
\int_{0}^{2\pi}\dd\phi\,{\delta\pars{r}\delta\pars{\cos\pars{\theta}}\delta\pars{\phi} \over r^{2}}
\\[3mm]&=\underbrace{\bracks{\int_{0^{-}}^{\infty}\delta\pars{r}\,\dd r}}
_{\ds{=\ 1}}\
\underbrace{\bracks{%
\int_{0}^{\pi}\delta\pars{\cos\pars{\theta}}\sin\pars{\theta}\,\dd\theta}}
_{\ds{=\ 1}}\
\underbrace{\bracks{\int_{0}^{2\pi}\delta\pars{\phi}\,\dd\phi}}_{\ds{=\ 1}}\
\\[3mm]&=\ \color{#66f}{\Large 1}
\end{align}
$$\mbox{Note that}\quad
\int_{{\mathbb R}^{3}}\delta\pars{\vec{r} - \vec{r}_{0}}\,\dd^{3}\vec{r}
=\int_{{\mathbb R}^{3}}\delta\pars{\vec{r}}\,\dd^{3}\vec{r}
$$
Best Answer
Firstly, the integrals $\newcommand\dif{\mathop{}\!\mathrm{d}}$ \begin{equation} \int_0^\infty \delta(x)\dif x \end{equation} and \begin{equation} \int_0^\infty \delta(x^2-x)\dif x \end{equation} are not defined in the context of distribution theory. Here is one way to justify this: Let $\delta_\epsilon$ be a nascent dirac delta, then the limit \begin{equation} \lim_{\epsilon\to 0}\int_0^\infty \delta_\epsilon(x)\dif x \end{equation} is not independent of the chosen sequence and may even not exist (here is the proof).
That being said, one may still assign a value to these integrals: As explained in Mathematics for Physicists we can define such integrals by restricting ourselves to "appropriate" representations of the $\delta$-function, e.g. the Gaussian representation \begin{equation} \delta_\epsilon(x)=\frac{\exp(-x^2/\epsilon^2)}{\epsilon\sqrt\pi} \end{equation} or the Lorentzian representation: \begin{equation} \delta_\epsilon(x)=\frac{1}{\pi}\frac{\epsilon}{\epsilon^2+x^2} \end{equation} Then we obtain the following results: \begin{equation} \lim_{\epsilon\to 0}\int_0^\infty \delta_\epsilon(x)\dif x=\frac{1}{2} \end{equation} and \begin{equation} \lim_{\epsilon\to 0}\int_0^\infty \delta_\epsilon(x^2-x)\dif x=1 \end{equation}
Sketch of the proof
More generally, we can consider integrals of the form \begin{equation} \int_A\delta(g(x))\dif x:=\lim_{\epsilon\to 0}\int_A\delta_\epsilon(g(x))\dif x \end{equation} To be precise, we consider the following setting: $g$ is defined on an open set $D\subset\mathbb R$ and $A\subset D$ such that $g$ has finitely many zeros $$x_1<\ldots<x_n$$ in $A$. $A$ is not necessarely open. Furthermore, we assume that $g$ is a diffeomorphism on a neighbourhood of each zero. Now let $U_1,\ldots,U_n$ be a list of disjoint subsets in $A$ which are open w.r.t. to the relative topology and with $x_i\in U_i$. Then: \begin{align} \forall \epsilon:\lim_{\epsilon\to 0}\int_A\delta_\epsilon(g(x))\dif x=\lim_{\epsilon\to 0}\int_{\bigcup_i U_i}\delta_\epsilon(g(x))\dif x+\lim_{\epsilon\to 0}\int_{A\setminus\bigcup_i U_i}\delta_\epsilon(g(x))\dif x\\=\sum_i\lim_{\epsilon\to 0}\int_{U_i}\delta_\epsilon(g(x))\dif x+\underbrace{\int_{A\setminus\bigcup_i U_i}\underbrace{\lim_{\epsilon\to 0}\delta_\epsilon(g(x))}_{=0}\dif x}_{=0}\\=\sum_i\lim_{\epsilon\to 0}\int_{g(U_i)}\frac{\delta(y)}{|(g'\circ g^{-1})(y)|}\dif y=\sum_i\frac{c_i}{|g'(x_i)|} \end{align} with $c_i=1$ if $g(x_i)$ is an inner point of $g(U_i)$ which is equivalent to $x_i$ being an inner point of $D$ since $g|_{U_i}$ is a diffeo (I am not 100% sure regarding the equivalence) and $c_i=1/2$ otherwise (i.e. if $x_i$ is a boundary point of $U_i$).
Not let us apply the result above to your example: In our case, we choose some $0<\epsilon<1/2$ such that $g(x)=x^2-x$ is a diffeo on $[0,\epsilon]$ and on $[1-\epsilon,1]$. Note that $g(\epsilon)=\epsilon^2-\epsilon<0$ and $g(1-\epsilon)=-\epsilon(1-\epsilon)<0$ and we obtain: \begin{align} \forall\epsilon:\int_0^1 \delta_\epsilon(x^2-x)\dif x=\int_0^\epsilon \delta_\epsilon(g(x))\dif x+\int_{1-\epsilon}^1 \delta_\epsilon(g(x))\dif x\\=\int_{\epsilon^2-\epsilon}^0 \frac{\delta_\epsilon(y)}{|(g'\circ g^{-1})(y)|}\dif y+\int_{-\epsilon(1-\epsilon)}^0 \frac{\delta_\epsilon(y)}{|(g'\circ g^{-1})(y)|}\dif y \end{align} and thus: \begin{equation} \lim_{\epsilon\to 0}\int_0^1 \delta_\epsilon(x^2-x)\dif x=\frac{1}{2}\frac{1}{|g'(0)|}+\frac{1}{2}\frac{1}{|g'(1)|}=1 \end{equation}