The following not-yet-published paper is, in my opinion, an excellent introduction and answer to the problem you bring up:
http://polmeth.wustl.edu/media/Paper/FixedversusRandom_1_2.pdf
To summarize, you can still proceed with the random effects approach, but you must first modify the model to account for the fact that the within-cluster and between-cluster effects differ (i.e., what the Hausman test indicates). You can do this by adding the cluster means of your predictor as a separate predictor in the model, and then optionally also applying within-cluster centering to the original predictor. The details of this procedure and the resulting interpretations are discussed at some length in the paper linked above.
First for your question about the variance-covariance and s.e. relationship: the variance-covariance matrix is a symmetric matrix which contains on the off-diagonal elements the covariances between all your betas in the model. The main diagonal elements contain the variance of each beta. If you take the square root of the main diagonal entries, you get the standard error of your betas.
Now to Hausman.
Since random effects is a matrix weighted average of the within and between variation in your data it is more efficient (i.e. has lower variance) than the fixed effects estimator which only exploits the within variation. If you want to test the difference between both models, you can write the test statistic as
$$H = (\beta_{FE}-\beta_{RE})'[Var(\beta_{FE})-Var(\beta_{RE})]^{-1}(\beta_{FE}-\beta_{RE})$$
Given that RE is more efficient the difference in the variances is positive definite - or at least it should be. If you use different variance estimators in the two regressions then $H$ might as well be negative. Often this is a sign of model miss-specification but this is a tricky discussion as there can be other instances for which the test statistic may be negative. Let's not consider those for the moment for simplicity.
If you now increase the sample size, you correctly said that your estimators become more efficient. Consequently $[Var(\beta_{FE})-Var(\beta_{RE})]^{-1}$ becomes smaller. Note that this difference is the denominator of a fraction, so as the denominator becomes smaller the fraction becomes bigger.
Maybe this is more intuitive if we consider the case when you are interested in a single variable (call it $k$) only. In this case the test statistic can be written as
$$H =\frac{(\beta_{FE,k}-\beta_{RE,k})}{\sqrt{[se(\beta_{FE,k})^{2}-se(\beta_{RE,k})^{2}]}}$$
To give a numerical example let's start first with the small sample. Let's say the difference in coefficients is 100 and their standard errors in FE and RE are 10 and 5, respectively:
$$H_{small} =\frac{(100)}{\sqrt{[10^{2}-5^{2}]}} = 11.547$$
Then you increase the sample size and suppose the standard errors reduce by one half:
$$H_{large} =\frac{(100)}{\sqrt{[5^{2}-2.5^{2}]}} = 23.094$$
Now you see how the test statistic becomes larger for a larger sample (as the denominator decreases in size thanks to the smaller standard errors). The intuition for the test statistic in matrix notation is the same.
Best Answer
Yes. See the following taken from a Princeton slide:
see: https://dss.princeton.edu/training/Panel101R.pdf