In An Introduction to Empirical Bayes Data Analysis by George Casella (1985), it is given that
\begin{align}
X|\theta &\sim N(\theta,\sigma^2) \\
\theta &\sim N(\mu,\tau^2) \\
\theta|X &\sim N\left(\frac{\theta\tau^2 + \mu\sigma^2}{\sigma^2+\tau^2},\frac{\sigma^2\tau^2}{\sigma^2 + \tau^2}\right) \\
X &\sim N(\mu,\sigma^2 + \tau^2)
\end{align}
It is my understanding that empirical Bayes estimation involves estimating the parameters $\mu$ and $\tau$ that $p(\theta)$ depends on using the samples $\mathcal{D} = \{x_1,x_2,\dots,x_N\}$ obtained from $p(x)$.
Using the law of total expectation,
\begin{align}
E[X] &= E_{p(\theta)}[E[X\mid\theta]] \\
&= E_{p(\theta)}[\theta] \\
&= \mu
\end{align}
and using the law of total variance,
\begin{align}
\text{Var}(X) &= E_{p(\theta)}[Var(X\mid\theta)] + \text{Var}(E[X\mid\theta]) \\
&= E_{p(\theta)}[\sigma^2] + \text{Var}(\theta) \\
&= \sigma^2 + \tau^2
\end{align}
Now suppose that we compute the sample mean and variance of $X$ from $\mathcal{D}$ such that
\begin{align}
\hat{\mu}_\mathcal{D} &\approx E[X] \\
\hat{\sigma}^2_\mathcal{D} &\approx \text{Var}(X)
\end{align}
Substituting these estimates into the equations above yields
\begin{align}
\mu &\approx \hat{\mu}_\mathcal{D} \\
\sigma^2 + \tau^2 &\approx \hat{\sigma}^2_\mathcal{D}
\end{align}
Re-arranging the second expression yields
\begin{align}
\mu &\approx \hat{\mu}_\mathcal{D} \\
\tau^2 &\approx \hat{\sigma}^2_\mathcal{D} – \sigma^2
\end{align}
which are the empirical Bayes estimates of the parameters $\mu$ and $\tau^2$ that we are looking for. This, by the way, is a special case of the method of moments.
Since
\begin{align}
X &\sim N(\mu,\sigma^2 + \tau^2) \\
\theta &\sim N(\mu,\tau^2)
\end{align}
I can see that empirical Bayes estimation was only possible when both $X$ and $\theta$ are dependent on $\mu$ and $\tau$. Moreover, an estimate of $\mu$ using the sample mean of $\mathcal{D}$ and an estimate of $\sigma^2 + \tau^2$ using the sample variance of $\mathcal{D}$ will yield information about $\mu$ and $\tau$, which are the parameters of $p(\theta)$.
However, is empirical Bayes estimation still possible if $X$ and $\theta$ are not dependent on the same parameters? For example, suppose that
$$
\begin{align}
X &\sim N(\mu,\sigma^2) \\
\theta &\sim N(\tau,\eta^2)
\end{align}
$$
Would empirical Bayes estimation still be possible in this case?
Best Answer
The confusion stems from considering marginals rather than joint. When $X$ depends on $\theta$ and $\theta$ depends on $\eta$ then marginally $X$ depends on $\eta$, which makes empirical Bayesian estimation feasible.
To consider that \begin{align} X &\sim N(\mu,\sigma^2) \\ \theta &\sim N(\tau,\eta^2) \end{align} means that $X$ does not depend on $\theta$ hence that $\theta$ cannot be estimated based on the observation of $X$.