They're not special. They're just convenient. It's relatively easy to tell what happens to a matrix when you apply an elementary row operation to it, and this isn't quite as true for more complicated types of operations.
In the language of group theory, elementary matrices form a set of generators for the group of invertible square matrices. You could choose a different set of generators if you wanted to, but again, the elementary matrices are convenient.
There are an infinite number of functions out there, and you can put an integral sign in front of any of them. Some of those functions are pretty strange and/or ugly. There is no reason why such an integral should have a representation in the very limited set of elementary functions. In fact when you get such a representation, you could consider yourself lucky.
For practical purposes, any convergent integral can be evaluated to any desired degree of accuracy. There are numerous numeric methods available with which to do that. And even if an integral should have some elementary function representation, that most often is a lot harder to evaluate than just running a numeric method on the integral. And you wouldn't be more accurate if the answer were cos(28.34) and you had to evaluate that.
Similarly, if you are going to use the integral in a further computation, it might be easiest to leave it as an integral, rather than tangling yourself up in things that have a lot of terms like sec(log^{-1}(x^{2})) -- or much worse.
So why do calculus classes teach you to find anti-derivatives? First, if the problem is going to resolve itself into an easy anti-derivative you might as well use it. Second, to familiarize you better with the underlying concepts, such as the chain rule and the product rule of derivatives. Third to give you some experience with actual answers so you have some idea what the answer ought to look like. If your numeric method evaluates to 2034.86, and you know the answer can't be larger than 80, then you know you made a mistake in your computation.
Of course, nowadays, all those integrals can be accurately evaluated online; but still, you should have some knowledge of what kind of answer to expect and what it all means.
Best Answer
As Sivaram Ambikasaran mentioned the description on Wikipedia is fine.
I believe the class of elementary functions, $E$, is commonly thought of as a construction of the form
All polynomials are in $E$
The exponential and the logarithm function is in $E$
The sine and cosine functions are in $E$.
$E$ is closed under addition, subtraction, multiplication, division and composition (finitely many operations of these).
$E$ is the smallest set with the properties 1-4.
This applies to both real or complex valued functions.
Edit 1:
Some examples of functions that are elementary
Some examples of functions that are not elementary
Edit 2:
The nomenclature elementary function is of course used since the functions that are elementary can be deduced using finitely many applications of elementary operations on basic "high school functions".
Edit 3:
Convinced by comments - removed closed under inversion in step 5. Added log in step 2.
Addendum (2013-02-28):
To see that the square root function $s(x)=\sqrt{x}$ is elementary using the above definition just note that $$ x\mapsto \frac{1}{2}\log x = \log x^{1/2}=\log\sqrt{x}$$ and hence $$s(x)=\sqrt{x}=\exp(\log\sqrt{x})$$ is elementary too. Perhaps $\arctan$ is a bit harder to deduce from the other steps in the construction of $E$, first note that \begin{eqnarray} \tan x=\frac{\sin x}{\cos x}= \frac{e^{ix}-e^{-ix}}{i(e^{ix}+e^{-ix})}= -i\frac{e^{2ix}-1}{e^{2ix}+1}=\\ -i\frac{e^{2ix}+1 -2}{e^{2ix}+1} = -i\frac{e^{2ix}+1 }{e^{2ix}+1}-i\frac{-2}{e^{2ix}+1}=\\-i +i\frac{2}{e^{2ix}+1} \end{eqnarray}
Hence if we solve for, $e^{2ix}$ in the above identity we get \begin{eqnarray} e^{2ix}=\frac{2}{1-i\tan x}-1= \frac{2-(1-i\tan x)}{1-i\tan x}= \frac{1+i\tan x}{1-i\tan x} \end{eqnarray} and taking the complex logarithm (more precisely the principal branch of the logarithm) we get $$x=\frac{1}{2i}\log\frac{1+i\tan x}{1-i\tan x}$$ That is $$t\mapsto \arctan t =\frac{1}{2i}\log\frac{1+it}{1-it}$$ belongs to $E$.