PCA already finds the best lineair combination of features (the principal components) to explain the amount of variance. If you start combining the principal components, you will not explain more variance, since PCA already found the best lineair combinations. Essentially you are trying to perform PCA and then again PCA. You could try performing PCA twice, but you will find the exact same result.
What you could do is maybe the following. Let's say you have a dataset with 2 features, one feature is in m and the other in km. It is justified to change the units so all features share common units. This way you could inflate the amount of variance explained.
However, be careful. If you take this to the extreme case, you could say feature 1 is much more important than feature 2. You could arrive at the conclusion that feature 2 is useless, and multiply it by zero. Now feature 1 will explain all variance (100%). But this becomes meaningless, since you deleted feature 2. So don't start reweighting all your features to just increase the amount of variance explained ;) Note that you should do this before PCA.
Finally, you could ask yourself if it is nececary to keep a lot of variance for your goal. Maybe if your goal is classification, you could consider another method (find a lineair combination of features that tries to seperate the classes as much as possible for example). Otherwise it might be wise to take the 3D dataset and look for a method that can work with multidimensional datasets.
Most of these things are covered in my answers in the following two threads:
- Relationship between SVD and PCA. How to use SVD to perform PCA?
- What exactly is called "principal component" in PCA?
Still, here I will try to answer your specific concerns.
Think about it like that. You have, let's say, $1000$ data points in $12$-dimensional space (i.e. your data matrix $X$ is of $1000\times12$ size). PCA finds directions in this space that capture maximal variance. So for example PC1 direction is a certain axis in this $12$-dimensional space, i.e. a vector of length $12$. PC2 direction is another axis, etc. These directions are given by columns of your matrix $V$. All your $1000$ data points can be projected onto each of these directions/axes, yielding coordinates of $1000$ data points along each PC direction; these projections are what is called PC scores, and what I prefer to simply call PCs. They are given by the columns of $US$.
So for each PC you have a $12$-dimensional vector specifying the PC direction or axis and a $1000$-dimensional vector specifying the PC projection on this axis.
"Reducing dimensionality" means that you take several PC projections as your new variables (e.g. if you take $6$ of them, then your new data matrix will be of $1000\times 6$ size) and essentially forget about the PC directions in the original $12$-dimensional space.
Most websites about PCA say that I should choose some principal components, but isn't it more correct to choose principal directions/axes since my objective is to reduce dimensionality?
This is equivalent. One column of $V$ corresponds to one column of $US$. You can say that you choose some columns of $V$ or you can say that you choose some columns of $US$. Doesn't matter. Also, by "principal components" some people mean columns of $V$ and some people mean columns of $US$. Again, most of the time it does not matter.
I have seen that my matrix V consists of 12 column vectors, each with 12 elements. If I choose 6 of these columns vectors, each vector still has 12 elements - but how is this possible if I have reduced the dimensionality?
You chose 6 axes in the 12-dimensional space. If you only consider these 6 axes and discard the other 6, then you reduced your dimensionality from 12 to 6. But each of the 6 chosen axes is originally a vector in the 12-dimensional space. No contradiction.
Besides, there are 12 column vectors of US, representing the principal components (scores), but each column vector has an awful lot of elements. What does it mean?
As I said, these are the projections on the principal axes. If your data matrix had 1000 points, then each PC score vector will have 1000 points. Makes sense.
Best Answer
If your $\newcommand{\X}{\mathbf X}\X$ is a $n\times d$ matrix with column means subtracted and $\newcommand{\W}{\mathbf W}\W_L$ is a $d \times L$ matrix consisting of $L$ principal directions (eigenvectors of the covariance matrix), then reconstruction error is given by $$\mathrm{Error}^2 = \|\X - \X\W_L^\vphantom{\top}\W_L^\top\|^2.$$ Note that this is not a mean squared error, it is the sum of squared errors. Here $\X\W_L$ is what you called $\mathbf T_L$ in your question and $\|\cdot\|$ denotes Frobenius norm.
The proportion of explained variance in PCA can be defined as $$\text{Proportion of explained variance} = \frac{\|\X\W_L\|^2}{\|\X\|^2},$$ i.e. it is a ratio of scores' sum of squares to the overall sum of squares (or equivalently, a ratio of scores' total variance to the overall total variance).
To see the connection between them, we need the following identity: $$\|\X\|^2 = \|\X - \X\W_L^\vphantom{\top}\W_L^\top\|^2 + \|\X\W_L^\vphantom{\top}\W_L^\top\|^2 = \|\X - \X\W_L^\vphantom{\top}\W_L^\top\|^2 + \|\X\W_L\|^2.$$ It might look mysterious, but is actually nothing else than Pythagoras theorem; see an informal explanation in my answer to Making sense of principal component analysis, eigenvectors & eigenvalues and a formal explanation in my answer to PCA objective function: what is the connection between maximizing variance and minimizing error?
We now see that $$\text{Proportion of explained variance} = 1-\frac{ \text{Error}^2}{\|\X\|^2}.$$
If you know the covariance matrix $\mathbf C = \frac{1}{n-1}\X^\top \X$, then you can compute the total variance $\operatorname{tr}\mathbf C$. The squared norm $\|\X\|^2$ is then given by the total variance multiplied by $n-1$. So we finally obtain
$$\text{Proportion of explained variance} = 1-\frac{\text{Error}^2}{(n-1)\operatorname{tr}\mathbf C}.$$