Neural Networks Speed Comparison – Is Convolutional Neural Network (CNN) Faster Than Recurrent Neural Network (RNN)?

Question

I am doing a research about NLP and I am using RNN (Recurrent Neural Network) or CNN (Convolutional Neural Network) to encode a sentence into a vector. When using CNN, the training time is significantly smaller than RNN. It is natural to me to think that CNN is faster than RNN because it does not build the relationship between hidden vectors of each timesteps, so it takes less time to feed forward and back propagate. However, this question was asked recently by another person and I do not have any evidence to support my intuition.

Is there any evidence or results about comparing the speed of CNN and RNN, especially on NLP tasks?

Answer 1

I have done some projects on text classification and relation extraction using CNN and RNN (specifically, LSTM and GRU): CNNs tend to be much faster (~5 times faster) than RNN.

It's hard to draw fair comparisons:

• CNN and RNN have different hyperparameters (filter dimension, number of filters, hidden state dimension, etc.)
• there exist many sort of RNNs
• the running time depends on the implementation, especially RNNs.
• CNNs run faster with CuDNN + CNMeM. RNNs benefit less from them.
• etc.

Nvidia has historically focused much more on CNN than RNN, as computer vision mostly employs CNN.

One benchmark: https://github.com/baidu-research/DeepBench

FYI:

• Benchmarks based on neural networks libraries to compare the performance between different GPUs
• Why doesn't training RNNs use 100% of the GPU?