UPD Equivallent formulation — how do you find all the independent isometric invariants of a tensor?
In what follows $V$ is a real inner product space.
I want to understand how does one find all (scalar) invariants of tensors under isometries. Wikipedia has a relevant article, but it confines itself only with tensors of type $V^* \otimes V$ or equivalent.
What I mean by an invariant of a tensor under the isometry. Invariant of a $V \otimes V^*$ tensor is a function $I : V \otimes V^* \to \mathbb R$ if
$$I\left( Q(A) \right) = I(A)$$
for any isometry $Q$ (transformation) properly lifted to act on tensors.
I've asked a question about the invariants of tensor of type $V$, but couldn't get the intuition to extend the reasoning to higher rank tensors
So far as I understand vectors have invariants of the form
$$f(\langle v, v \rangle)$$
whereas tensors in $V^* \otimes V$ have
$$f(\operatorname{Tr} A, \operatorname{Tr} A^2, \operatorname{Tr} A^3)$$
$f$ is meant to be an arbitrary function to $\mathbb R$ of appropriate type.
I don't even see why there are three independent invariants for rank two tensors.
One might be be puzzled of the importance of the question and why I'm asking it. The above mentioned wikipedia article provides a good example. Potential energy (scalar) in an elastic material is a function of the strain tensor. The coupling between them should be natural in the sense that it should not involve arbitrary tensors or vectors (because there are simply none), one have to construct a scalar just from the strain tensor. Theories in continuum mechanics abounds in dozens of various tensor fields of different ranks and one need to seek the forms of natural couplings between them.
Best Answer
This seems too large for a comment, so I'll post it as an answer.
First of all for matrices (aka tensors of type $V\otimes V^*$). If the tensor/matrix is symmetric, then it is diagonalizable by an isometry (this is spectral theorem), so any isometric invariant is a function of eigenvalues. Moreover as permutations are isometries, it has to be a symmetric function of eigenvalues. If one further restricts to polynomial symmetric functions then all such are well understood - they are all expressible interms of either $a_1+a_2+\ldots a_n$, $a_1^2+a_2^2+\ldots a_n^2$,$\ldots$, $a_1^n+a_2^n+\ldots a_n^n$ - these are the traces in your question - or in terms of coefficients of $(x-a_1)(x-a_2)\cdots (x-a_n)$ - those are the invariants in the wikipedia article (see also http://en.wikipedia.org/wiki/Elementary_symmetric_polynomial). When written not in terms of eigenvalues, but as traces or coefficients of the characteristic polynomial they become invariants of the matrix entries.
For not-necessarily symmetric matrices there could be other invariants - indeed there must be, for the space of all matrices is of dimension $n^2$ and the space of isometries is of dimension $n(n-1)/2$, so the quotient is of dimension $n(n+1)/2>n$. In particular in 3-D one expects 6 invariants). I don't know the answer, but what one is doing is taking the space (variety) of all matrices and quotienting it by the action of the group of isometries, and then asking what are the functions on the resulting space. In the symmetric case, the quotient is actually $\mathbb{R}^n$ coodinatized in this funny way. In the general case I'm not sure, but I suspect people who do invariant theory or representation theory know this.
Note: In the scalar case the quotient $V$ divided by isometries is $\mathbb{R}_+$ coordinatized by the square length.
Note 2: For skew-symmetric matrices one can bring them to canonical block form by isometric transformations. http://en.wikipedia.org/wiki/Skew-symmetric_matrix so the the invariants are given in terms of squares of the eigenvalues, well expressed as invariants of the $A^2$ - so one again gets either traces of $A^2$, $A^4$ and so on or coefficients of the characteristic polynomial of $A^2$. Of course when $n=3$ there is only one invariant - trace $A^2$.
For "higher" tensors things seem less obvious still. You agin take some vector space (of tensors) and quotient it by the action of the group of isometries and ask for the functions on the resulting quotient.
One other comment - if I understand correctly in some theories you get scalars that depend not only on the tensor itself, but also on its derivatives of various orders. If you allow this generalization the problem becomes harder.