The standard deviation is the square root of the variance.
The standard deviation is expressed in the same units as the mean is, whereas the variance is expressed in squared units, but for looking at a distribution, you can use either just so long as you are clear about what you are using. For example, a Normal distribution with mean = 10 and sd = 3 is exactly the same thing as a Normal distribution with mean = 10 and variance = 9.
Reporting standard deviations instead of variances
I think you are right in that standard deviation of each PC can perhaps be a more reasonable or a more intuitive (for some) measure of its "influence" than its variance. And actually it even has a clear mathematical interpretation: variances of PCs are eigenvalues of the covariance matrix, but standard deviations are singular values of the centered data matrix [only scaled by $1/\sqrt{n-1}$].
So yes, it is completely fine to report it. Moreover, e.g. R does report standard deviations of PCs rather than their variances. For example running this simple code:
irispca <- princomp(iris[-5])
summary(irispca)
results in this:
Importance of components:
Comp.1 Comp.2 Comp.3 Comp.4
Standard deviation 2.0494032 0.49097143 0.27872586 0.153870700
Proportion of Variance 0.9246187 0.05306648 0.01710261 0.005212184
Cumulative Proportion 0.9246187 0.97768521 0.99478782 1.000000000
There are standard deviations here, but not variances.
Explained variance
A PC that contains 95% of the data variance might contain only 80% of the variation in the data as measured in standard deviations: isn't the latter a better descriptor?
However, note that after presenting standard deviations, R does not display a "proportion of standard deviation", but instead a proportion of variance. And there is a very good reason for that.
Mathematically, total variance (being a trace of covariance matrix) is preserved under rotations. This means that the sum of variance of original variables is equal to the sum of variances of PCs. In case of the same Fisher Iris dataset, this sum is equal to $4.57$, and so we can say that PC1, having a variance of $2.05^2=4.20$ explains $92\%$ of the total variance.
But the sum of standard deviations is not preserved! The sum of standard deviations of original variables is $3.79$. The sum of standard deviations of PCs is $2.98$. They are not equal! So if you want to say that PC1 with standard deviation $2.05$ explains $x\%$ of the "total standard deviation", what would you take as this total? There is no answer, because it simply does not make sense.
The bottom line is that it is completely fine to look at the standard deviation of each PC and even compare them between each other, but if you want to talk about "explained" something, then only "explained variance" makes sense.
Best Answer
Robert's and Bey's answers do give part of the story (i.e. moments tend to be regarded as basic properties of distributions, and conventionally standard deviation is defined in terms of the second central moment rather than the other way around), but the extent to which those things are really fundamental depends partly on what we mean by the term.
There would be no insurmountable problem, for example, if our conventions went the other way -- there's nothing stopping us conventionally defining some other sequence of quantities in place of the usual moments, say $E[(X-\mu)^p]^{1/p}$ for $p=1,2,3,...$ (note that $\mu$ fits into both the moment sequence and this one as the first term) and then defining moments -- and all manner of calculations in relation to moments -- in terms of them. Note that these quantities are all measured in the original units, which is one advantage over moments (which are in $p$-th powers of the original units, and so harder to interpret). This would make the population standard deviation the defined quantity and variance defined in terms of it.
However, it would make quantities like the moment generating function (or some equivalent relating to the new quantities defined above) rather less "natural", which would make things a little more awkward (but some conventions are a bit like that). There's some convenient properties of the MGF that would not be as convenient cast the other way.
More basic, to my mind (but related to it), is that there are a number of basic properties of variance that are more convenient when written as properties of variance than when written as properties of standard deviation (e.g. the variance of sums of independent random variables is the sum of the variances).
This additivity is a property that's not shared by other measures of dispersion and it has a number of important consequences.
[There are similar relationships between the other cumulants, so this is a sense in which we might want to define things in relation to moments more generally.]
All of these reasons are arguably either convention or convenience but to some extent it's a matter of viewpoint (e.g. from some points of view moments are pretty important quantities, from others they're not all that important). It may be that the "at a deep level" bit is intended to imply nothing more than kjetil's "when developing the theory".
I would agree with kjetil's point that you raised in your question; to some extent this answer is merely a hand-wavy discussion of it.