1.) What are poles and zeros of linear system {A,B,C,D} exactly? What does it mean for a system to have a pole at a certain value, or a zero at certain value?
Intuitively, I do not know exactly what poles or zeros are. All I know is that the poles are roots of the denominator of the transfer function, or the eigenvalues of the $A$ matrix, like the one in your question. Poles show up explicitly in the solutions of ordinary differential equations, and an example of this can be seen here:
http://www.math.oregonstate.edu/home/programs/undergrad/CalculusQuestStudyGuides/ode/laplace/solve/solve.html
So what kind of question can we answer using information about poles?
i) Is the system stable?
ii) If it is stable, is the response of the system oscillatory, is it like a rigid body?
iii) If it is unstable, is it possible to stabilize this system using output feedback? (you need information about zeros here)
Now, let's talk about zeros. Zeros show up in literature because it has an effect on the behavior of control systems.
i) They impose fundamental limitations on the performance of control systems.
ii) In adaptive control systems, zeros can cause your adaptive controller to go unstable.
iii) They tell you about the "internal stability" of a control system.
As far as I can tell, zeros are more subtle than poles. I cannot say I fully understand them.
2.) The author writes about 'poles of a transfer function matrix H(s)'. What is a transfer function matrix? The only thing I know is how to compute it and that it describes some relation between input/output of the system. But why do we need tranfser function matrices?
Taking the Laplace transform of a differential equation that has a single-input and a single-output yields a transfer function. An example of this is in the link above. A transfer function describes the relationship between a single output and a single input. So if you have a system of differential equations that has, say, 2 inputs and 3 outputs, then a transfer matrix is a matrix of transfer functions that contains 6 elements. Each individual element describing the relationship between one of the inputs and one of the outputs. (The superposition principle plays a big role here)
But why would one want a transfer matrix. I believe it is because calculating zeros for a multi-input multi-output system is not easy. Here is an article that talks about all the different kinds of zeros and why they are important:
http://www.smp.uq.edu.au/people/YoniNazarathy/Control4406/resources/HoaggBernsteinNonMinimumPhaseZero.pdf
3.) To calculate the poles and zeros, the author says that we need the Smith and Smith-McMillan Forms. These are matrices that have only diagonal entries. What is exactly the algorithm to calculate the Smith-(McMillan)-form of a transfer matrix?
Sorry. I don't have much on this one.
4.) What is the relation between the poles of a system and the controllability, observability, stability and stabilizability ? The same for a zero ?
For me, poles and zeros are important to transfer functions, which describe the relationship between inputs and outputs, and they can tell you about stabilizability and stability. However, concepts like controllability and observability are state space concepts (At least for me). If you write a transfer function in state space form, as you have written in your question, then there is a very simple test for controllability and observability. You can find more about this in almost any course, for example in Stephen Boyd's introductory control course at Stanford.edu.
5.) What is an invariant zero polynomial of the system {A,B,C,D} ?
A SISO system just has one kind of zero. A MIMO system has many kinds of zeros, one of which is an invariant zero. The roots of the invariant zero polynomial gives you invariant zeros. It makes me kind of sad that I do not know very much about zeros of MIMO systems.
6.) What is 'a realization of a system'?
Let's say you start off with a differential equation. Then you take its Laplace transform, and obtain a transfer function. Then, for this transfer function, there are an infinite number of state space representations. That is, there are an infinite number of matrices $A, B, C, D$ that yield the same input-output relationship as the original transfer function. These representations are called realizations. We can go from one realization to another using "Similarity Transformations".
7.) Where can I find more good information about this subject?
If you are a mathematician, then you should probably look for a more mathematical text on control systems. Most engineers use a classical control book ( like the one by Ogata ) in undergrad, which is mostly about transfer functions, zeros, poles, and various stability tests. Then, in grad school, engineers take a course called "Linear Systems Theory", where they learn about State Space theory of control systems. The book I used was by "Chen", but I did not like it very much.
Best Answer
The zeros are more fundamental than the poles in the following sense: while poles can be assigned by feedback, the zeros can only be canceled. Therefore, an unstable zero cannot be moved: you have to live with whatever effect it has on the performance of your system, even after closing feedback loops.
And what are those effects? The zeros create a "notch" in the frequency response. The gain at those frequencies is attenuated. Those are the frequencies at which you cannot expect the system to respond. If you need good response at such frequencies, feedback control will not help - you need to redesign the system itself.
Another important property of the zeros can be visualized using the root-locus diagram. For high feedback gains, the closed-loop poles move towards the zeros. Stable zeros pull the roots towards the stable side of the complex plain; unstable zeros, which define non minimum-phase systems, pull the roots to the unstable half-plane, limiting the use of high feedback gains. For minimum-phase systems, some form of high-gain feedback control may be stabilizing (although such methods are often of dubious practical value).