deep learningregression
there is a lot of research where deep learning works so well with classification but not in regression field
SVR, tree-based approach is still good and I couldn't find good architecture about regression
well there is some scheme you have to follow when implementing deep regression but I want to know why it doesn't work well as classification
any good theory or explanation why deep learning (mlp) doesn't work as well as classification?
I want to dig into this problem
The neural networks used in typical deep learning models have a very large number of nodes with many layers, and therefore many parameters that must be estimated. This requires a lot of data. A small neural network (with fewer layers and fewer free parameters) can be successfully trained with a small data set - but this would not usually be described as "deep learning".
Typically, people suggest that the deeper the network the better
At some point, adding more layers hurt performances.
Example 1: Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. "FractalNet: Ultra-Deep Neural Networks without Residuals." arXiv preprint arXiv:1605.07648 (2016). https://arxiv.org/abs/1605.07648
Example 2: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. "Deep Residual Learning for Image Recognition". arXiv:1512.03385. https://arxiv.org/abs/1512.03385
Practitioners simply try and see (and in between buy more GPUs).
(Similar question has been asked before 2 years ago, and I think it is a good time to bring it up again: Guideline to select the hyperparameters in Deep Learning)
As far as I know, not much progress has been done since then. There have been some new papers, but using known hyperparameter optimization techniques, e.g.:
There is one original work though that one of my lab mate made me aware of:
Miconi, Thomas. "Neural networks with differentiable structure.", arXiv (2016): the author introduce the concept of differentiable network structure to include it in the objective function.
Best Answer
In my opinion, the following might be one -but not the only- reason for the relatively low popularity of Deep Learning in regression problems:
Much of the success of modern Neural Networks comes from their ability to exploit the compositional nature of the world. This is, in perception problems such as image or audio analysis, features have an order (spatial or temporal) and local patterns aggregate to form higher level concepts and objects (e.g, a picture of a car is made of wheels and other parts, which are made of lower level visual features, which are made of basic shapes like edges, circles and lines, etc.). Modern Neural Networks such as Convolutional Neural Networks take advantage of this by learning increasingly abstract features in the deeper layers. These ideas are explained in more detail in these slides by Hinton et al. [1]
Conversely, classical regression problems consist of a number of non-ordered features, and the target value can be predicted fairly well with a shallow linear/nonlinear model of the input features. In some sense, this compositional property present in problems such as image classification or speech recognition is not present in problems such as "Predict the income of an individual based on their sex, age, nationality, academic degree, family size...".
This might explain why some of the regression problems where Deep Learning is more popular are those based on images (e.g., Age prediction based on face photography).
Of course, these are only some intuitive ideas, and a more formal analysis of this problem is certainly a nice research topic. Deep Learning is certainly a field where more theoretical guarantees and insights are needed.
[1] https://www.iro.umontreal.ca/~bengioy/talks/DL-Tutorial-NIPS2015.pdf