machine learningneural networksrecurrent neural networksequence analysistheano
What exact kind of architecture of neural networks do I need for a sequence binary/multiclass classification? The sequences can be of different length and are to be discriminated by a certain occurrence of smaller subsequences in it.
It would be great if you can provide Lasagne/Keras layers setup and their parameters.
Thanks.
As you mentioned, most neural networks are based on general simple abstractions of the brain. Not only are they lacking in mimicking characteristics like plasticity, but they do not take into account signals and timing as real neurons do.
There's a fairly recent interview, that I felt was appropriate for your specific question, Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts, and I quote:
But it’s true that with neuroscience, it’s going to require decades or even hundreds of years to understand the deep principles. There is progress at the very lowest levels of neuroscience. But for issues of higher cognition—how we perceive, how we remember, how we act—we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like. So we are not yet in an era in which we can be using an understanding of the brain to guide us in the construction of intelligent systems.
Your data basically cannot be learned with an RNN trained that way. Your input is $\sin(t)$ is $2\pi$-periodic $\sin(t) = \sin(t+2\pi)$
but your target $\sin(t/2)$ is $4\pi$-periodic and $\sin(t/2) = -\sin(t+2\pi)$
Therefore, in your dataset you'll have pairs of identical inputs with opposite outputs. In terms of Mean Squared Error, it means that the optimal solution is a null function.
These are two slices of your plot where you can see identical inputs but opposite targets
Best Answer
I recommend to use a combination of CNN layers and a RNN layer (e.g. long short-term layer LSTM or gated recurrent units).
It depends on your sequence classification problem, I tried to solve a typical application (e.g. in bioinformatics) and failed.
Then I used this network architecture - which I found by very experienced data scientist - and suddenly it worked very well.