regressionterminology
This is a very basic question, but I haven't been able to get an answer with my google-fu.
In the following sentences:
We regress W on Z
We performed a regression of W on Z
I'm unsure what of and on denote. Do I read that as:
$W \tilde{} \alpha + \beta Z + \epsilon $
or
$Z \tilde{} \alpha + \beta W + \epsilon $
If you assume a uniform prior on $\beta$, then the posterior for $\beta$ is $$ \beta|y \sim N(\hat{\beta},V_\beta). $$ with $$\hat{\beta} = [X'\Sigma^{-1}X]X'y \quad \mbox{and} \quad V_\beta =[X'\Sigma^{-1}X]^{-1}.$$ To find the predictive distribution, we need more information. If $\tilde{y} \sim N(\tilde{X}\beta,\tilde{\Sigma})$ and is conditionally independent of $y$ given $\beta$, then $$ \tilde{y}|y \sim N(\tilde{X}\hat{\beta}, \tilde{\Sigma} + V_\beta).$$ But typically for these types of models, $y$ and $\tilde{y}$ are not conditionally independent, instead, we typically have $$ \left( \begin{array}{c} y \\ \tilde{y} \end{array} \right) \sim N\left( \left[\begin{array}{c} X\beta \\ \tilde{X}\beta \end{array}\right], \left[ \begin{array}{cc} \Sigma & \Sigma_{12} \\ \Sigma_{21} & \tilde{\Sigma} \end{array} \right]\right). $$ If this is the case, then $$ \tilde{y}|y \sim N(\tilde{X}\hat{\beta} + \Sigma_{21}\Sigma^{-1}(y-X\hat{\beta}), \tilde{\Sigma} - \Sigma_{21}\Sigma^{-1}\Sigma_{12}). $$ This assumes $\Sigma, \Sigma_{12},$ and $\tilde{\Sigma}$ are all known. As you point out, typically they are unknown and need to be estimated. For the common models that have this structure, e.g. time series and spatial models, there generally won't be a closed form for the predictive distribution.
Below is a geometric viewpoint similar to an answer to a different question: Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression
The regression is a perpendicular projection onto the vectors in the columns of $X$ and $Z$. What you are basically doing is defining a different vector $\bar{X}$ such that the coordinates associated with the projection remain the same.
This alternative vector is drawn in the image with a red on the right side.
The vector $\bar{X}$ is perpendicular to $Z$ and that is why all those coefficients $\beta$ turn out to be the same.
If $Z$ and $\bar{X}$ are perpendicular then
The regression $$Y \sim \beta_1 \bar{X} + \beta_2 Z$$ and $$Y \sim \beta_1^\prime \bar{X} $$ will be the same in the sense $\beta_1 = \beta_1^\prime$
The regression $$Y \sim \beta_1 \bar{X} + \beta_2 Z$$ and $$Y \sim \beta_1^{\prime\prime} (\bar{X} + a Z) + \beta_2 Z$$ with $a$ some constant, will be the same in the sense $\beta_1 = \beta_1^{\prime\prime}$.
Note that we can write $X = \bar{X} + a Z$. The difference between $X$ and $\bar{X}$ is some multiple of $Z$.
Best Answer
The "regression of W on Z" means you're predicting W from Z. Using your equations, this corresponds to the first one, namely $W=\alpha+\beta Z+\epsilon$