Given a real number $x$, the notation $[x]$ (also often seen as $\lfloor x\rfloor$) literally means "the greatest integer less than or equal to $x$". For example, $1.2$ is not an integer. The integers are
$$\ldots,\;-4,\;-3,\;-2,\;-1,\;0,\;1,\;2,\;3,\;4,\ldots$$
We throw away all the integers that are not less than or equal to $1.2$:
$$\ldots,\;-4,\;-3,\;-2,\;-1,\;0,\;1\hphantom{,\;2,\;3,\;4,\ldots}$$
and take the largest one of them, which is $1$. Therefore, the greatest integer less than $1.2$ is $1$, and we write
$$\lfloor 1.2\rfloor =1.$$
(By the way, take a look at the relevant Wikipedia article.)
Here is a graph of $\lfloor x\rfloor$, from $x=-3$ to $x=3$:
Here is a version where I've added a graph of just the function $x$ itself (in red) and lines indicating the integers (in green).
You can see on this graph that, as we expected, $\lfloor 1.2\rfloor =1$.
Here is a graph of $\lfloor \sqrt{x}\rfloor$, from $x=0$ to $x=25$:
Here is a version where I've added a graph of just the function $\sqrt{x}$ itself (in red) and lines indicating the integers (in green).
Here's a check for this graph: using a calculator, we know that
$$\sqrt{20}\approx 4.47214$$
and so the largest integer that's less than or equal to $\sqrt{20}$ will be $4$.
Using the fact that, for any numbers $a<b<c$,
$$\int_a^c f(x)\,dx=\int_a^bf(x)\,dx+\int_b^cf(x)\,dx,$$
we can break up the range we're integrating over into pieces where $\lfloor\;\;\rfloor$ is constant, and we do know how to integrate constants (note that the endpoints don't contribute the value of the integral; integrating over the set of $x$'s for which $a\leq x\leq b$ will give the same answer as integrating over the set of $x$'s for which $a\leq x<b$, and we denote both of these operations by $\int_a^b$). For example,
$$\int_1^3\lfloor x\rfloor\,dx=\int_1^2\lfloor x\rfloor\,dx+\int_2^3\lfloor x\rfloor\,dx=\int_1^21\,dx+\int_2^32\,dx=1+2=3.$$
Your way is fine, it's just a matter of recognising a differentiated geometric series in that:
$$\begin{align}
\sum_{n=0}^\infty \frac{n(e-1)}{e^{n+1}} &= \sum_{n=0}^\infty \frac{e-1}{e^2} \cdot \frac{n}{e^{n-1}}\\
&= \frac{e-1}{e^2} \sum_{n=0}^\infty n\left(\frac{1}{e}\right)^{n-1}\\
&= \frac{e-1}{e^2} \frac{1}{\left(1-\frac{1}{e}\right)^2}\\
&= \frac{e-1}{(e-1)^2}\\
&= \frac{1}{e-1}.
\end{align}$$
In a similar way, when you have a series
$$\sum_{n=0}^\infty p(n)\cdot x^{n+d}$$
with a polynomial $p$ of degree $k$, then you can write that as a linear combination of derivatives of the geometric series with ratio $x$ of orders $\leqslant k$.
Best Answer
For each $x\in[0,2]$,$$\lfloor x^2-x+1\rfloor=\left\{\begin{array}{l}0&\text{ if }x\in(0,1)\\1&\text{ if }x=0\text{ or }x\in\left[1,\frac{1+\sqrt5}2\right)\\2&\text{ if }x\in\left[\frac{1+\sqrt5}2,2\right)\\3&\text{ if }x=2.\end{array}\right.$$Therefore\begin{align}\int_0^2\lfloor x^2-x+1\rfloor\,\mathrm dx&=\frac{1+\sqrt5}2-1+2\left(2-\frac{1+\sqrt5}2\right)\\&=\frac{5-\sqrt5}2.\end{align}