I would suggest you to have a look at this answer where I have discussed the Fundamental Theorem of Calculus for functions which are not necessarily continuous.
Next note that the integrand here has a discontinuity at $x=1$ and as far as the interval of integration is concerned the discontinuity is removable and hence we just remove it by changing value of integrand at $x=1$ to $0$. Doing this has no impact on the value of the integral (because the value of a Riemann integral does not depend on the value of the integrand at a finite number of points in the interval under consideration) and thus the desired integral is equal to $\int_{1/2}^{1}0\,dx=0$.
In general if we have to evaluate an integral of the form $\int_{a} ^{b} f(x) \, dx$ where $f$ has a finite number of discontinuities at $x_{1},x_{2},\dots,x_{n}$ in $[a, b] $ with $$a\leq x_{1}\leq x_{2}\leq \dots\leq x_{n} \leq b$$ such that left and right hand limits of $f$ exist at each point of discontinuity then we split the integral as follows $$\int_{a} ^{b} f(x) \, dx=\int_{a} ^{x_{1}}f(x)\,dx+\int_{x_{1}}^{x_{2}}f(x)\,dx+\cdots+\int_{x_{n-1}}^{x_{n}}f(x)\,dx+\int_{x_{n}}^{b}f(x)\,dx$$ Each integral on right can be evaluated by observing that the integrand has removable discontinuity at the end points $x_{i}$ and the integrand can be redefined at these points to make it continuous on that interval.
You can use this technique to evaluate $\int_{1}^{2}[x^{2}]\,dx$ as $$\int_{1}^{2}[x^{2}]\,dx=\int_{1}^{\sqrt{2}}1\,dx+\int_{\sqrt{2}}^{\sqrt{3}}2\,dx+\int_{\sqrt{3}}^{2}3\,dx$$
Also to answer your question about evaluating the integral under consideration via second part of fundamental theorem of calculus, note that there is no anti-derivative of $[x] $ on interval $[1/2,1]$ (why? perhaps you should answer this yourself, but let me know if you feel issue here) and hence we can't use fundamental theorem of calculus here.
Also note that continuous functions are guaranteed to have an anti-derivative, but continuity is not necessary. There are discontinuous functions which possess anti-derivative and if we have a function $f$ which is Riemann integrable on $[a, b] $ and possesses an anti-derivative $F$ (note that existence of anti-derivative is not guaranteed and hence we are assuming its existence here as a part of the hypotheses) such that $F'(x) =f(x) $ for all $x\in[a, b] $ then we have $\int_{a}^{b}f(x)\,dx=F(b)-F(a)$.
Best Answer
Say $f(t)=\cos(t^2+t)$ and an antiderivative is $F(t)$. The integral in question is, by the fundamental theorem of calculus, $$F(x)-F(0)$$ $F(0)$ is a constant and disappears upon differentiating with respect to $x$, whereas $F(x)$ becomes $f(x)$ once again. Thus, after differentiation we must have the RHS as $\cos(x^2+x)$.