The structure of this question is as follows: at first, I provide the concept of ensemble learning, further I provide a list of pattern recognition tasks, then I give examples of ensemble learning algorithms and, finally, introduce my question. Those who don't need all the supplemental information might just look at the headlines and go straight to my question.
What is ensemble learning?
According to Wikipedia article:
In statistics and machine learning, ensemble methods use multiple
learning algorithms to obtain better predictive performance than could
be obtained from any of the constituent learning algorithms alone.
Unlike a statistical ensemble in statistical mechanics, which is
usually infinite, a machine learning ensemble refers only to a
concrete finite set of alternative models, but typically allows for
much more flexible structure to exist among those alternatives.
Examples of pattern recognition tasks:
- Optical character recognition
- Barcode recognition
- License plate recognition
- Face detection
- Speech recognition
- Image recognition
- Document classification
Examples of ensemble learning algorithms:
The following ensemble learning algorithms used for PR tasks (according to Wiki):
Ensemble learning algorithms (supervised meta-algorithms for combining
multiple learning algorithms together):
Boosting (a machine learning ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms which convert weak learners to strong ones)
Bootstrap aggregating ("bagging") (a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression).
Ensemble averaging (the process of creating multiple models and combining them to produce a desired output, as opposed to creating just one model. Frequently an ensemble of models performs better than any individual model, because the various errors of the models "average out.")
Mixture of experts, hierarchical mixture of experts
Different implementations
- Ensembles of neural networks (a set of neural network models taking a decision by averaging the results of individual models).
- Random forest (an ensemble learning method for classification, regression and other tasks, that operate by constructing a multitude
of decision trees at training time and outputting the class that is
the mode of the classes (classification) or mean prediction
(regression) of the individual trees).- AdaBoost (the output of the other learning algorithms ('weak learners') is combined into a weighted sum that represents the final
output of the boosted classifier).
Additionally:
- Methods that use one neural network to combine different classifiers
- Competence areas method
My question
Which of the ensemble learning algorithms is considered to be state-of-the-art nowadays and is actually used in practice (for face detection, vehicle registration plates recognition, optical character recognition, etc.) by enterprises and organizations? Using ensemble learning algorithms is supposed to increase recognition accuracy and lead to a better computational efficiency. But, do matters stand this way in reality?
Which ensemble method, potentially, can show better classification accuracy and performance in the pattern recognition tasks? Perhaps, some of the methods are outdated now, or have shown to be ineffective. It is also possible that ensemble methods now tend to not be used anymore on the strength of some new algorithms. Those who have experience in the area or have sufficient knowledge in this field, can you help to clarify the matters?
Best Answer
State-of-the-art algorithms may differ from what is used in production in the industry. Also, the latter can invest in fine-tuning more basic (and often more interpretable) approaches to make them work better than what academics would.
Example 1: According to TechCrunch, Nuance will start using "deep learning tech" in its Dragon speech recognition products this september.
Example 2: Chiticariu, Laura, Yunyao Li, and Frederick R. Reiss. "Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems!." In EMNLP, no. October, pp. 827-832. 2013. https://scholar.google.com/scholar?cluster=12856773132046965379&hl=en&as_sdt=0,22 ; http://www.aclweb.org/website/old_anthology/D/D13/D13-1079.pdf
With that being said:
One of the state-of-the-art systems for image classification gets some nice gain with ensemble (just like most other systems I far as I know): He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." arXiv preprint arXiv:1512.03385 (2015). https://scholar.google.com/scholar?cluster=17704431389020559554&hl=en&as_sdt=0,22 ; https://arxiv.org/pdf/1512.03385v1.pdf