The basic issue here is true and fairly well known in statistics. However, his interpretation / claim is extreme. There are several issues to be discussed:
First, power doesn't change very fast with changes in $N$. (Specifically, it changes as a function of $\sqrt N$, so to halve the standard deviation of your sampling distribution, you need to quadruple your $N$, etc.) However, power is quite sensitive to effect size. Moreover, unless your estimated power is $50\%$, the change in power with a change in effect size isn't symmetrical. If you are trying for $80\%$ power, power will decrease more rapidly with a decrease in Cohen's $d$ than it will increase with an equivalent increase in Cohen's $d$. For example, when starting from $d = .5$ with $N = 128$, if you had 20 fewer observations, power would drop by $\approx 7.9\%$, but if you had 20 more observation, power would increase by $\approx 5.5\%$. On the other hand, if the true effect size were $.1$ lower, then power would be $\approx 16.9\%$ lower, but if it were $.1$ higher, it would only be $\approx 12.6\%$ higher. This asymmetry, and the differing sensitivity, can be seen in the figures below.
![enter image description here](https://i.stack.imgur.com/VS5S2.png)
If you are working from effects estimated from prior research, say a meta-analysis or a pilot study, the solution to this is to incorporate your uncertainty about the true effect size into your power calculation. Ideally, this would involve integrating over the entire distribution of possible effect sizes. This is probably a bridge too far for most applications, but a quick and dirty strategy is to calculate the power at several possible effect sizes, your estimated Cohen's $d$ plus or minus 1 and 2 standard deviations, and then get a weighted average using the probability densities of those quantiles as the weights.
If you are conducting a study of something that has never been studied before, this doesn't really matter. You know what effect size you care about. In reality, the effect is either that large (or larger), or it is smaller (even possibly 0). Using the effect size you care about in your power analysis will be valid, and will give you an appropriate test of your hypothesis. If the effect size that you care about is the true value, you will have (say) an $80\%$ chance of 'significance'. If, due to sampling error, the realized effect size in your study is smaller (larger) your result will be less (more) significant, or even non-significant. That is the way it is supposed to work.
Second, regarding the broader claim that power analyses (a-priori or otherwise) rely on assumptions, it is not clear what to make of that argument. Of course they do. So does everything else. Not running a power analysis, but just gathering an amount of data based on a number you picked out of a hat, and then analyzing your data, will not improve the situation. Moreover, your resulting analyses will still rely on assumptions, just as all analyses (power or otherwise) always do. If instead you decide that you will continue to collect data and re-analyze them until you get a picture you like or get tired of it, that will be much less valid (and will still entail assumptions that may be invisible to the speaker, but that exist nonetheless). Put simply, there is no way around the fact that assumptions are being made in research and data analysis.
You may find these resources of interest:
Kraemer, H.C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J.A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals, Archives of General Psychiatry, 63, 5, pp. 484-489.
Uebersax, J.A. (2007). Bayesian Unconditional Power Analysis. http://www.john-uebersax.com/stat/bpower.htm
You're correct that a power analysis for the effect size you actually found is irrelevant. You already know the study wasn't powerful enough to detect that. But you should still include a power analysis for effect size(s) that were plausible a priori. Eg, calculate that the study had X% power to detect an effect size of 0.5 at alpha = 0.95. That should reassure the reviewer that the null result you've got really does provide some evidence the true effect size is small, rather than just being a small-sample fluke. (Or, if you find your sample was underpowered to detect a reasonable effect, then the reviewer has found a real problem.)
Regarding multiple variables, if your study was focusing on a small set of variables but included the others in order to be efficient and provide added value, then you could only do the power analysis for the important variables and describe the other results as exploratory. Or you could include power analysis for all of them to be complete.
If your study was powerful enough to detect a large effect but not a medium effect (or a medium but not a small), then you might need to give some justification for why you chose the sample size you did. Reasons of cost/time/effort usually work well enough, provided you acknowledge any limitations those cause.
The reviewer probably just wants to confirm that the study was large enough that it would've detected a effect if the effect was large enough to be practically meaningful.
Best Answer
Assume for simplicity that your model is defined by only one parameter $\theta$. The power is the function $\theta \mapsto \Pr(\text{reject } H_0 \mid \theta)$, which depends on the sample size $n$.
In Retrospective Power Analysis, you only plug in your estimate $\theta$: you look at the value $\Pr(\text{reject } H_0 \mid \hat\theta)$ at the power function at $\theta=\hat\theta$, with the same sample size $n$. It answers the question: "what would be the probability that I would obtain significant results if $\theta$ were $\hat\theta$" ? As said in your text this question is rather useless because there is a one-to-one correspondance between the $p$-value and the restrospective power $\Pr(\text{reject } H_0 \mid \hat\theta)$.
For instance consider a binomial experiment with proportion parameter $\theta \in [0,1]$ and the hypothesis $H_0\colon\{\theta=0\}$. Obviously the power increases when $\theta$ increases. And obviously the $p$-value decreases when $\hat\theta$ increases. Consequently the lower $p$-value, the higher RP (retrospective power). A couple of years ago I wrote a R code for the case of Fisher tests in classical Gaussian linear models. It is here. There's a code using simulations for the one-way ANOVA example and a code for the general model provding an exact calculation of RP in function of the $p$-value and the design parameters. I called my function
PAP()
because "Puissance a posteriori" is the French translation of RP and PAP is also an acronym for "Power Approach Paradox". The cause of the decreasing correspondence between $p$ and RP for Gaussian linear models is intuitively the same as for the binomial experiment: if $\theta$ is "far from $H_0$" then the power at $\theta$ is high, and if $\hat\theta$ is "far from $H_0$" then the $p$-value is small. Theoretically this is a consequence of the fact that the noncentral Fisher distributions are stochastically increasing in the noncentrality parameter (see this discussion about noncentral $F$ distributions in Gaussian linear models). In fact here the noncentrality parameter plays the role of $\theta$ (is it the so-called effect size ? I don't know).I claimed "RS is rather useless because of the correspondence with $p$" because this decreasing correspondence with $p$ means that having a high RP is equivalent to having a small $p$, and vice-versa. But the more serious problem is the misinterpretation of RP; for instance, I have found such claims in the literature:
$H_0$ is not rejected and RP is high, so the decision of the test is significant.
$H_0$ is not rejected, it is not surprising because RP is low.
$H_0$ is rejected (so the decision is significant) and RP is high, so the decision is even more significant.
Respectively replace "RP is high" and "RP is low" with "$p$ is low" and "$p$ is high" in the three claims above and you will see that they are either useless, wrong, or puzzling.
From a more "philosophical" perspective, RP is useless because why would we mind about the probability that rejection of $H_0$ occurs once the experiment is done ?
See also here a funny but clever retrospective power online calculator ;-)
The paragraph A Posteriori Power Analysis says nothing about the choice of $\theta$, but it emphasizes the main difference with the retrospective power: here the goal is to use the information issued from your first experiment to evaluate the power of a future experiment, focusing on the sample size. A sensible approach to evaluate this power is to consider your estimate $\hat\theta$ as a "guess" of the true $\theta$ and also to consider the uncertainty about this estimate. There is a natural way to do so in Bayesian statistics, namely the predictive power, which consists to average the possible values of $\Pr(\text{reject } H_0 \mid \theta)$ for various values of $\theta$, according to some distribution (the posterior distribution in Bayesian terms) representing the knowledge and the uncertainty about $\theta$ resulting from your first experiment. In the frequentist framework you could consider the values of the power evaluated at the bounds of your confidence interval about $\theta$.