Regression – Difference Between Logistic Regression and Bayesian Logistic Regression

Question

I'm a bit confused whether these two are the same concept. If they are different what's the difference?

Thanks!

Answer 1

The other answers are good. However, to clarify the intuition as well as give some further details:

• In logistic regression, you maximize the likelihood function $p(y|\beta_{0},\beta_{1},x)$ (find MLE). That is, you find the weights $\beta_{0},\beta_{1}$ that maximizes how likely your observed data is. There is no closed form solution to the MLE, so you need to use iterative methods. This gives you a single point estimate of our weights.
• In bayesian logistic regression, you start with an initial belief about the distribution of $p(\beta_{0},\beta_{1})$. Then $p(\beta_{0},\beta_{1}|x,y)\propto p(y|\beta_{0},\beta_{1},x)p(\beta_{0},\beta_{1})$. That is, the posterior, which is our updated belief about the weights given evidence, is proportional to our prior (initial belief) times the likelihood. We can't evaluate the closed form posterior, but can approximate it by sampling or variational methods. This gives us a distribution over the weights. For instance, if we use a normal approximation for both $\beta_{0}$ and $\beta_{1}$ using variational methods, then we'll get a mean and variance for $\beta_{0}$, and one for $\beta_{1}$ as well.

For further detail on both techniques, these scribe notes of a lecture are excellent http://web.cse.ohio-state.edu/~kulis/teaching/788_sp12/scribe_notes/lecture6.pdf.