NOTE: This question has originally been posted in MSE, but it did not generate any interest. It was first posted there, because the question itself is a pure matrix-algebra question.
Nevertheless, since the motive has to do with statistics and econometrics, I am posting the question on Cross Validated also, in the hope that some statistics/matrix algebra savvy brain will have something to contribute.
The framework is as follows: We have a cross-sectional i.i.d. sample $\{\mathbf y, \mathbf X\}$, where $\mathbf y$ is a $N \times 1$ column vector, and $\mathbf X$ is a $N\times K$ matrix. We postulate a linear relationship between $\mathbf y$ and $\mathbf X$,
$$\mathbf y = \mathbf X \beta + \mathbf u $$
where $\mathbf u$ is white-noise with variance $\sigma^2$, and exogenous to the regressors in the $\mathbf X$ matrix, and $\beta$ is a $K\times 1$ column vector of unknown constant coefficients. Under this assumption, the OLS estimator is unbiased and consistent.
Now assume $N$ is "large", say $O \left( 10^{4} \right)$ or more (samples that large have started to appear in the econometrics field also). Then a researcher could conceivably entertain the following two options:
$A$) Run one OLS regression using the whole sample. This tactic can be thought of as appealing to the consistency property of the OLS estimator. Call this estimator $\hat \beta$.
$B$) Divide the sample into $m$ disjoint sub-samples (for simplicity, assumed of equal length, and note that their union equals the whole sample), run $m$ regressions, and calculate the average of the $m$ coefficient estimates she will thus obtain. This tactic can be thought of as appealing to the unbiasedness property of the OLS estimator. Call this averaging estimator $\bar b_m$.
(Note that tactic $B$ does not fall into any re-sampling approach, like bootstrap, subsampling, or jackknife -to be exact, it has been considered as a marginal case in applying jackknife in time series, but it is not trully a jackknife method).
I have derived a nice-looking (to me) result that shows that the variance of the whole-sample estimator is always smaller than the variance of the averaging estimator:
$$\text{Var}\left(\bar b_m\right) > \text{Var}\left(\hat \beta\right) $$
I say it is nice-looking because the result uses the Arithmetic-Harmonic mean inequality for PD matrices, proven here:
Specifically, writing $Z_l= \left(X_l'X_l\right)^{-1}$ for the inverse moment matrix of the regressors from the l-th sample, l=$1,…,m$, denoting by $A_m$ the arithmetic mean and by $H_m$ the harmonic mean of these $Z$ matrices, it is not hard to arrive at the following:
$$\text{Var}\left(\bar b_m\right) = \frac1m\sigma^2A_m > \frac1m\sigma^2H_m =\text{Var}\left(\hat \beta\right) $$
…the inequality holding in the matrix sense. Note that $H_m$ is the harmonic mean of $\left(X_1'X_1\right)^{-1},…,\left(X_m'X_m\right)^{-1} $in the true matrix sense, not a matrix containing the harmonic means of the corresponding elements of the matrices it averages.
So the averaging estimator $\bar b_m$ is always less efficient than the whole sample estimator $\hat \beta$.
My question: Are there any known bounds for the difference between Arithmetic-Harmonic means for matrices?
For real numbers they are (see the wiki article and the original resources here and here ).
Why? because it will be helpful in order to move to the next step and compare estimators that may be neither unbiased nor consistent, and so one is left with a criterion such as Minimum Squared Error to compare them.
Any suggestion, link or reference will be really appreciated.
Best Answer
Yes, indeed there is. Please see the work by Mond and Pec̆arić here. They established the AM-GM inequality for positive semi-definite matrices. Here is a link to the paper that contains the proof:
https://www.sciencedirect.com/science/article/pii/0024379595002693
After downloading the paper, the proof is on pages 450-452, in the Main Result section.
Here is a citation in case you need it:
Mond, B., and Pec̆arić, J. E. (1996), “A mixed arithmetic-mean-harmonic-mean matrix inequality,” Linear Algebra and its Applications, Linear Algebra and Statistics: In Celebration of C. R. Rao’s 75th Birthday (September 10, 1995), 237–238, 449–454. https://doi.org/10.1016/0024-3795(95)00269-3.
I hope this helps you.
Best, =K=