1.) What are poles and zeros of linear system {A,B,C,D} exactly? What does it mean for a system to have a pole at a certain value, or a zero at certain value?
Intuitively, I do not know exactly what poles or zeros are. All I know is that the poles are roots of the denominator of the transfer function, or the eigenvalues of the $A$ matrix, like the one in your question. Poles show up explicitly in the solutions of ordinary differential equations, and an example of this can be seen here:
http://www.math.oregonstate.edu/home/programs/undergrad/CalculusQuestStudyGuides/ode/laplace/solve/solve.html
So what kind of question can we answer using information about poles?
i) Is the system stable?
ii) If it is stable, is the response of the system oscillatory, is it like a rigid body?
iii) If it is unstable, is it possible to stabilize this system using output feedback? (you need information about zeros here)
Now, let's talk about zeros. Zeros show up in literature because it has an effect on the behavior of control systems.
i) They impose fundamental limitations on the performance of control systems.
ii) In adaptive control systems, zeros can cause your adaptive controller to go unstable.
iii) They tell you about the "internal stability" of a control system.
As far as I can tell, zeros are more subtle than poles. I cannot say I fully understand them.
2.) The author writes about 'poles of a transfer function matrix H(s)'. What is a transfer function matrix? The only thing I know is how to compute it and that it describes some relation between input/output of the system. But why do we need tranfser function matrices?
Taking the Laplace transform of a differential equation that has a single-input and a single-output yields a transfer function. An example of this is in the link above. A transfer function describes the relationship between a single output and a single input. So if you have a system of differential equations that has, say, 2 inputs and 3 outputs, then a transfer matrix is a matrix of transfer functions that contains 6 elements. Each individual element describing the relationship between one of the inputs and one of the outputs. (The superposition principle plays a big role here)
But why would one want a transfer matrix. I believe it is because calculating zeros for a multi-input multi-output system is not easy. Here is an article that talks about all the different kinds of zeros and why they are important:
http://www.smp.uq.edu.au/people/YoniNazarathy/Control4406/resources/HoaggBernsteinNonMinimumPhaseZero.pdf
3.) To calculate the poles and zeros, the author says that we need the Smith and Smith-McMillan Forms. These are matrices that have only diagonal entries. What is exactly the algorithm to calculate the Smith-(McMillan)-form of a transfer matrix?
Sorry. I don't have much on this one.
4.) What is the relation between the poles of a system and the controllability, observability, stability and stabilizability ? The same for a zero ?
For me, poles and zeros are important to transfer functions, which describe the relationship between inputs and outputs, and they can tell you about stabilizability and stability. However, concepts like controllability and observability are state space concepts (At least for me). If you write a transfer function in state space form, as you have written in your question, then there is a very simple test for controllability and observability. You can find more about this in almost any course, for example in Stephen Boyd's introductory control course at Stanford.edu.
5.) What is an invariant zero polynomial of the system {A,B,C,D} ?
A SISO system just has one kind of zero. A MIMO system has many kinds of zeros, one of which is an invariant zero. The roots of the invariant zero polynomial gives you invariant zeros. It makes me kind of sad that I do not know very much about zeros of MIMO systems.
6.) What is 'a realization of a system'?
Let's say you start off with a differential equation. Then you take its Laplace transform, and obtain a transfer function. Then, for this transfer function, there are an infinite number of state space representations. That is, there are an infinite number of matrices $A, B, C, D$ that yield the same input-output relationship as the original transfer function. These representations are called realizations. We can go from one realization to another using "Similarity Transformations".
7.) Where can I find more good information about this subject?
If you are a mathematician, then you should probably look for a more mathematical text on control systems. Most engineers use a classical control book ( like the one by Ogata ) in undergrad, which is mostly about transfer functions, zeros, poles, and various stability tests. Then, in grad school, engineers take a course called "Linear Systems Theory", where they learn about State Space theory of control systems. The book I used was by "Chen", but I did not like it very much.
Best Answer
Eigenvalues and poles only tells you at what "frequency" it moves, with frequency I am referring to the complex value $\lambda$ such that its time evolution can be characterized with $e^{\lambda\,t}$. A mode considers this as well but also the mode's shape. For a state space model the shape of a mode refers to the eigenvector corresponding to the eigenvalue $\lambda$.
In finite element analysis modes are used as well, called modal analysis in which the eigenvectors also have a physical meaning. Namely these vectors show the normalized displacement of the approximated system when only that mode/eigenvalue is excited. But for state space models in general you can apply any similarity transformation and still obtain the same outputs. Under these transformations the eigenvalues remain unchanged, but the eigenvectors can change, so the shape of a mode does not have to be meaningful.