Using PCA for feature selection (removing non-predictive features) is an extremely expensive way to do it. PCA algos are often O(n^3). Rather a much better and more efficient approach would be to use a measure of inter-dependence between the feature and the class - for this Mutual Information tends to perform very well, furthermore it's the only measure of dependence that a) fully generalizes and b) actually has a good philosophical foundation based on Kullback-Leibler divergence.
For example, we compute (using maximum likelihood probability approx with some smoothing)
MI-above-expected = MI(F, C) - E_{X, N}[MI(X, C)]
where the second term is the 'expected mutual information given N examples'. We then take the top M features after sorting by MI-above-expected.
The reason why one would want to use PCA is if one expects that many of the features are in fact dependent. This would be particularly handy for Naive Bayes where independence is assumed. Now the datasets I've worked with have always been far too large to use PCA, so I don't use PCA and we have to use more sophisticated methods. But if your dataset is small, and you don't have the time to investigate more sophisticated methods, then by all means go ahead and apply an out-of-box PCA.
I think there are two ways to look at the question whether SVD/PCA helps in general.
Is it better to use PCA reduced data instead of the raw data?
Often yes, but there are situations where PCA is not needed.
I'd in addition consider how well the bilinear concept behind PCA fits with the data generation process. I work with linear spectroscopy, which is governed by physical laws that mean that my observed spectra $\mathbf X$ are linear combinations of the spectra $\mathbf S$ of the chemical species I have, weighted by their respective concentrations $c$: $\mathbf X = \mathbf C \mathbf S$.This fits very well with the PCA model of scores $\mathbf T$ and loadings $\mathbf P$: $\mathbf X = \mathbf T \mathbf P$
I don't know of any example where PCA has hurt a model (except gross errors in setting up a combined PCA-whaterver model)
Even if the underlying relationship in your data doesn't suit that well to the bilinear approach of PCA, PCA in the first place is only a rotation of your data which would usually not hurt. Discarding higher PCs leads to the dimension reduction, but due to set up of the PCA, they carry only small amounts of variance - so again, chances are that even if it is not all that suitable, it won't hurt that much, neither.
This is also part of the bias-variance trade-off in the context of PCA as regularization technique (see @usεr11852's anwer).
Is it better to use PCA instead of some other dimension reduction technique?
The answer on this will be application specific. But if your application suggests some other way of feature generation, these features may be far more powerful than some PCs, so this is worth considering.
Again, my data and applications happen to be of a nature where PCA is a rather natural fit, so I use it and I cannot contribute a counter-example.
But: having a PCA hammer does not imply that all problems are nails...
Looking for counterexamples, I'd start maybe image analyses where objects in question can appear anywhere in the picture. The people I know who deal with such tasks usually develop specialized features.
The only task I routinely have that comes close to this is detecting cosmic ray spikes in my camera signals (sharp peaks somewhere caused by cosmic rays hitting the CCD). I also use specialized filters to detect them, although they are easy to find after PCA as well. However, we describe that rather as PCA not being robust against spikes and find it disturbing.
Best Answer
Your question is implicitly assuming that reducing explained variation is necessarily a bad thing. Recall that $R^2$ is defined as: $$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} $$ where $SS_{res} = \sum_{i}{(y_i - \hat{y})^2}$ is a residual sum of squares and $SS_{tot} = \sum_{i}{(y_i - \bar{y})^2}$ is a total sum of squares. You can easily get an $R^2 = 1$ (i.e. $SS_{res} = 0$) by fitting a line that passes through all of the (training) points (though this, in general, requires more flexible model as opposed to a simple linear regression, as noted by Eric), which is a perfect example of overfitting. So reducing explained variation isn't necessarily bad as it could result in a better performance on unseen (test) data. PCA can be a good preprocessing technique if there are reasons to believe that the dataset has an intrinsic lower-dimensional structure.