natural languagerecursive-modeltensorflow
Is there any available recursive neural network implementation in TensorFlow TensorFlow's tutorials do not present any recursive neural networks. Most TensorFlow code I've found is CNN, LSTM, GRU, vanilla recurrent neural networks or MLP. I am most interested in implementations for natural language processing.
Example of a recursive neural network:
The tree structure may vary between different inputs. I.e., I do not want the tree structure to be fixed.
Recurrent Neural networks are recurring over time. For example if you have a sequence
x = ['h', 'e', 'l', 'l']
This sequence is fed to a single neuron which has a single connection to itself.
At time step 0, the letter 'h' is given as input.At time step 1, 'e' is given as input. The network when unfolded over time will look like this.
A recursive network is just a generalization of a recurrent network. In a recurrent network the weights are shared (and dimensionality remains constant) along the length of the sequence because how would you deal with position-dependent weights when you encounter a sequence at test-time of different length to any you saw at train-time. In a recursive network the weights are shared (and dimensionality remains constant) at every node for the same reason.
This means that all the W_xh weights will be equal(shared) and so will be the W_hh weight. This is simply because it is a single neuron which has been unfolded in time.
This is what a Recursive Neural Network looks like.
It is quite simple to see why it is called a Recursive Neural Network. Each parent node's children are simply a node similar to that node.
The Neural network you want to use depends on your usage. In Karpathy's blog, he is generating characters one at a time so a recurrent neural network is good.
But if you want to generate a parse tree, then using a Recursive Neural Network is better because it helps to create better hierarchical representations.
If you want to do deep learning in c++, then use CUDA. It has a nice user-base, and is fast. I do not know more about that so cannot comment more.
In python, Theano is the best option because it provides automatic differentiation, which means that when you are forming big, awkward NNs, you don't have to find gradients by hand. Theano does it automatically for you. This feature is lacked by Torch7.
Theano is very fast as it provides C wrappers to python code and can be implemented on GPUs. It also has an awesome user base, which is very important while learning something new.
Embeddings are dense vector representations of the characters. The rationale behind using it is to convert an arbitrary discrete id, to a continuous representation.
The main advantage is that back-propagation is possible over continuous representations while it is not over discrete representations. A second advantage is that the vector representation might contain additional information based on its location compared to the other characters.
This is still a hot area of research. If you are interested in learning more, check out the word2vec algorithms: vector embeddings are learned for words where interesting relationships are learned. For example an interesting write-up here: https://deeplearning4j.org/word2vec.html
Best Answer
Take a look at Tensorflow Fold:
https://github.com/tensorflow/fold
TensorFlow Fold is a library for creating TensorFlow models that consume structured data, where the structure of the computation graph depends on the structure of the input data.