When I started studying machine learning in 2002, Neural Networks were on their way out and Support Vector Machines were becoming more and more popular. At the time my understating was that SVM could do anything that a NNet could, and they were based on a more solid theoretical grounding (Vapnik–Chervonenkis theory).
Now NNets have made a come back with Deep Neural Nets and nobody seems to use SVM anymore.
So what can DNN do that SVM can't ? Is there any reason other than hype to choose DNN over SVM?
Best Answer
I will list a few areas where I am fairly confident DNNs perform better than SVMs, and it's not just the "hype". I'm sure there are more, just as I'm sure there are places where SVMs would do better. In particular, I've found that lots of people who ask questions like these are thinking only about Fully Connected networks (a.k.a. feed forward networks, or multi-layer perceptrons, or ANNs, or ...), typically being applied to more standard "tabular" style data. In these cases, I have not seen incredible results from DNNs, and if that is your only experience it may be easy to believe it is just hype.
Where DNNs really shine is with Convolutional Neural Networks, but also their ability to handle sequential data, their ability to generate data, and reinforcement learning (e.g. learning to play Go or Atari games). I'll go into detail on a few below.
Images
DNNS, in particular Convolutional Neural Networks (CNNs), are the clear state of the art on almost every image processing task. I'm not sure if anyone is seriously suggesting that SVMs reach comparable performance on classification datasets like ImageNet, Cifar10, or even MNIST.
This goes doubly for "dense" image predictions, e.g. given a 500x500 image of a CT scan classify exactly which pixels are a tumor (see U-Net: Convolutional Networks for Biomedical Image Segmentation by Olaf Ronneberger, Philipp Fischer, Thomas Brox for one of the earlier works in this area). I'm not sure what it would even look like for an SVM to do that.
Note that while images are the poster child project for CNNs, they can also be readily applied to other tasks where signal processing may historically have been used, and in my personal experience have reached better performance levels.
Data Generation
There are a number of data generation tasks that are currently areas of research. For example, using Generative Adversarial Networks to generate new images (in practice, though, it is typically just used as unsupervised feature learning). There is also interesting work being done to try and generate art/music (https://magenta.tensorflow.org/). Similar to dense image predictions, I'm not sure what it would look like for SVMs to do this. Maybe there are people doing fascinating work in this area with SVMs, I'm not going to claim to be an expert on that, but my impression is that is not happening.
I will note that data generation is not just purely academic, there are uses in voice synthesis (think Siri, Cortana, or Google Assistant), and likely other areas.
Reinforcement Learning
Researchers have been able to train DNNs to learn how to play Atari games using only the raw pixel data and the scores as input (https://deepmind.com/research/dqn/). Perhaps this is just my inexperience, but this is a feat I would struggle to achieve with SVMs.