This is a good question! It is all a matter of how to define an improper integral. The traditional way is to write
$$\int_a^b f(t)\, dt = \lim_{x \to a} \lim_{y \to b} \int_x^y f(t)\, dt$$
whenever one of the bounds is worrisome. In particular, we can write
$$\int_a^b f(t) \,dt = \lim_{x \to a} \lim_{y \to b} \left(\int_x^1 f(t) \, dt + \int_1^y f(t)\, dt \right)$$
or
$$\lim_{x \to a} \lim_{y \to b} \left(\int_x^1 f(t) \,dt \right) + \lim_{x \to a} \lim_{y \to b} \left( \int_1^y f(t) \,dt \right) $$
so that
$$\int_a^b f(t)\, dt = \lim_{x \to a} \left(\int_x^1 f(t) \,dt \right) + \lim_{y \to b} \left(\int_1^y f(t)\, dt \right).$$
Now convergence means that the integral is finite and the only way for the sum of two numbers to be finite is if both are finite. Hence your professor's comment. All of the above assumes $a < 1 < b$ and that there is no trouble with $f$ near $t=1$.
But you are right that you can get cancelation if you take the limits differently. Instead of taking the limit to a and then the limit to b sequentially, if you took them at the same time--that is to say you combined them--you could possibly get cancellation.
Here's an example. If define instead the improper integral as
$$\int_{-\pi/2}^{\pi/2} \tan t\, dt = \lim_{x \to \pi/2} \int_{-x}^x \tan t\,dt$$
then
$$\int_{-\pi/2}^{\pi/2} \tan t\, dt = \lim_{x \to \pi/2} \left(- \ln \cos(x) + \ln \cos(-x) \right) = 0.$$
What you are thinking of is a useful concept and it often comes up with the term Principal Value in more advanced calculus courses.
Best Answer
This does converge conditionally. By what you wrote above have $$ \int_0^{\infty} (\sqrt{x+\cos x}-\sqrt{x}) \, dx = \int_0^{\infty} \frac{\cos x}{\sqrt{x+\cos x}+\sqrt{x}} \, dx $$ and since $$ \frac{1}{\sqrt{x+\cos x}+\sqrt{x}} $$ is a decreasing function on $(0,\infty)$ tending to zero as $x \to \infty$ while $\cos x$ is uniformly bounded and has average value equal to zero, the improper integral converges by Dirichlet's test for integrals.