Recently I read some papers about the Bayesian neural network (BNN) [Neal, 1992], [Neal, 2012], which gives a probability relation between the input and output in a neural network. Training such a neural network is through MCMC which is different from the traditional back-propagation algorithm.
My question are: What is the advantage of using such a neural network? More specifically, could you provide some examples which better fit BNN rather than NN?
Best Answer
Bayesian neural nets are useful for solving problems in domains where data is scarce, as a way to prevent overfitting. They often beat all other methods in such situations. Example applications are molecular biology (for example this paper) and medical diagnosis (areas where data often come from costly and difficult expiremental work). Actually, Bayesian nets are univerally useful and can obtain better results for a vast number of tasks but they are extremely difficult to scale to large problems.