Bayesian – How to Compare Predictions from MLE-based Regression Vs. Bayesian Regression

bayesianmodel selectionpredictive-models

Say I have two linear regression models that I want to use for predictions.

Linear regression:
\begin{equation}
\mathbf{y} \sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y)
\end{equation}

Bayesian linear regression:
\begin{align}
\mathbf{y | b} &\sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y) \notag\\
\mathbf{b} &\sim \mathcal{N}(0, \Sigma_b)
\end{align}

Predictions

For the linear regression, I find the coefficients by MLE:
\begin{equation}
\mathbf{b}_{\text{MLE}} = \mathbf{(X X^T)^{-1} X y}
\end{equation}

The predictions for the the test set are:
\begin{equation}
\mathbf{y_{test}} = \mathbf{X_{test}^T b_\text{MLE}}
\end{equation}

For the bayesian linear regression, I find the predictive posterior:
\begin{equation}
p(\mathbf{y_{test}} | \mathbf{X_{test}} , \mathbf{y}, \mathbf{b}) =
\int p(\mathbf{y_{test}} | \mathbf{X_{test}}, \mathbf{b}) p(\mathbf{b} |\mathbf{y}) \text{d}\mathbf{b}
\end{equation}
(Thought this integral could be analytically derived, I find it by Gibbs sampling since in my actual model the different $b_i$ come from a mixture of gaussians.)

The predictions are plotted here, where for the bayesian prediction I also plotted the variance of every $y$ (computed from the traces of the Gibbs sampler)
benchmark

Questions

  • How should I compare the two models? Should I use $R^2$ (and then all the bayesian machinery is for nothing?)

  • (forgetting about MLE) How to evaluate the performance of the bayesian estimators? I've found a lot about model selection, but what about the accuracy of a single model?

Best Answer

How should I compare the two models?

Form a Bayesian point of view you do not have two models to compare: MLE is just a single 'best' guess. (Instead of 'MAP' I would you the expectation of the model prediction; because MAP also is only a 'best' guess.) To compare the two models in a Bayesian framework, you would need their evidences - which is essentially zero for the MLE 'model'.

How to evaluate the performance of the bayesian estimators? I've found a lot about model selection, but what about the accuracy of a single model?

From a Bayesian viewpoint it makes only sense to compare a model if you have alternative competing models. The 'root means square deviation' (RMSD) is a measure I would refrain from using (the only exception being if everything is nice and Gaussian). By definition, the likelihood defines how your model relates to the data. How can you justify to switch to the RMSD to assess the 'quality of your model'?

Of course, you can use the tools mentioned. However, you should not call it Bayesian then.

Related Question