Coming from a measure-theoretic point of view, one can write the following:
$\int_{-\infty}^{+\infty}\delta(x-a)dx = \int_{\mathbb R} 1 d \mu_a(x)$, where $\mu_a$ denotes the point measure with mass $1$ at $a$ and $0$ everywhere else.
Now, if we want to integrate over an interval $I$, we integrate the characteristic function $\chi_I(x)$ over $\mathbb R$:
$$\int_I \delta(x-a)dx= \int_{\mathbb R}\chi_I(x)d\mu_a(x).$$
And here comes a important distinction which we dont need to make when using the Riemann integral: If $a$ is a border point of the interval $I$, do we consider the open interval $I_1 =(-\infty,a)$, or the interval $I_2=(-\infty,a]$ ? Because the mass of $\mu_a$ lies at $a$, this is an important distinction:
$$\int_{(-\infty,a)} \delta(x-a)dx= \int_{\mathbb R}\chi_{(-\infty,a)}(x)d\mu_a(x) = 0 $$
while
$$\int_{(-\infty,a]} \delta(x-a)dx= \int_{\mathbb R}\chi_{(-\infty,a]}(x)d\mu_a(x) = 1.$$
So we have to be very careful with what we mean by $\int_{-\infty}^a\delta(x-a)dx.$
Summing up: The fact that it's 'infinitely thin' translates to the measure having a point mass, and this means that the inclusion or the exclusion of single points will change the value of the integral.
The $\delta$-function is not actually a function - it's a distribution. The idea of finding an integral by using lots of thin rectangles doesn't work here. This is a more abstract form of integration. In fact, lots of integrals from Quantum Mechanics do not converge in the classical sense.
The $\delta$-function has the property that $\delta(x) = 0$ for all $x \neq 0$.
So, for example, $\delta(2) = 0$ and $\delta(-3.4)=0$.
The value of $\delta(0)$ is not well-defined, but we do know that
$$\int_{-\infty}^{\infty}\delta(x)~\mathrm dx = 1$$
Since $\delta(x) = 0$ for all $x \neq 0$, the values of $x$ away from zero contribute nothing to the integral:
$$\int_{-\varepsilon}^{\varepsilon} \delta(x)~\mathrm dx = 1$$
for any $\varepsilon > 0$, as small as you like!
If $S$ is some open subset of the real numbers then
$$\int_S \delta(x)~\mathrm dx \ \ = \ \ \left\{ \begin{array}{ccc} 1 & : & 0 \in S \\ 0 & : & 0 \notin S \end{array}\right.$$
In fact, you can make even stronger statements, e.g. $S$ doesn't need to be open, but you need to be careful how you word it.
In your example
$$H(x) := \int_{-\infty}^{x} \delta(\tau)~\mathrm d\tau$$
the set $S$ is the interval $(-\infty,x)$. If $x < 0$ then $0 \notin S$ and so $H(x) = 0$ for all $x < 0$. If $x>0$ then $0 \in S$ and so $H(x) = 1$ for all $x > 0$. What happens when $x=0$ depends on whom you speak to.
Best Answer
$$\int_{-\infty}^t \delta(x) \, dx = \begin{cases} 1 & t \ge 0 \\ 0 & t < 0\end{cases} = u(t)$$ This is a function of $t$ because there is a variable $t$ in the upper limit of integration; for each value of $t$, there an integral to consider.
Taking $t \to \infty$ yields $\int_{-\infty}^\infty \delta(x) \, dx = 1$. This is a number because it is a single definite integral.