Recurrent Neural networks are recurring over time. For example if you have a sequence
x = ['h', 'e', 'l', 'l']
This sequence is fed to a single neuron which has a single connection to itself.
At time step 0, the letter 'h' is given as input.At time step 1, 'e' is given as input. The network when unfolded over time will look like this.
![RNN](https://i.stack.imgur.com/r7L0U.jpg)
A recursive network is just a generalization of a recurrent network. In a recurrent network the weights are shared (and dimensionality remains constant) along the length of the sequence because how would you deal with position-dependent weights when you encounter a sequence at test-time of different length to any you saw at train-time. In a recursive network the weights are shared (and dimensionality remains constant) at every node for the same reason.
This means that all the W_xh weights will be equal(shared) and so will be the W_hh weight. This is simply because it is a single neuron which has been unfolded in time.
This is what a Recursive Neural Network looks like.
![Recursive Neural Network](https://i.stack.imgur.com/ngG5S.png)
It is quite simple to see why it is called a Recursive Neural Network. Each parent node's children are simply a node similar to that node.
The Neural network you want to use depends on your usage. In Karpathy's blog, he is generating characters one at a time so a recurrent neural network is good.
But if you want to generate a parse tree, then using a Recursive Neural Network is better because it helps to create better hierarchical representations.
If you want to do deep learning in c++, then use CUDA. It has a nice user-base, and is fast. I do not know more about that so cannot comment more.
In python, Theano is the best option because it provides automatic differentiation, which means that when you are forming big, awkward NNs, you don't have to find gradients by hand. Theano does it automatically for you. This feature is lacked by Torch7.
Theano is very fast as it provides C wrappers to python code and can be implemented on GPUs. It also has an awesome user base, which is very important while learning something new.
I have never worked with recurrent networks, but from what I know, in practice, some RNN and TDNN can be used for the same purpose that you want: Predict time series values. However, they work different.
It is possible with TDNN:
- Predict process' values
- Find a relationship between two processes.
Some RNN, like NARX also allow you to do that, and it is also used to predict financial time series, usually better than TDNN.
A TDNN looks more like a feedforward network, because time aspect is only inserted through its inputs, unlike NARX that also needs the predicted/real future value as input. This characteristic makes TDNN less robust than NARX for predicting values, but requires less processing and is easier to train.
If you are trying to find a relationship between a process $X(t)$ and a process $Y(t)$, NARX requires you to have past values of $Y$, while TDNN does not.
I recommend reading Simon Haykin's Neural Networks: A Comprehensive Foundation (2nd Edition) and this FAQ. There are lots of neural networks architectures and variations. Sometimes they have many names or there is no consensus about their classification.
Best Answer
I have done some projects on text classification and relation extraction using CNN and RNN (specifically, LSTM and GRU): CNNs tend to be much faster (~5 times faster) than RNN.
It's hard to draw fair comparisons:
Nvidia has historically focused much more on CNN than RNN, as computer vision mostly employs CNN.
One benchmark: https://github.com/baidu-research/DeepBench
FYI: