Can anyone provide a brief overview of the most popular training algorithms for perceptrons?
I am currently training my perceptron using standard stochastic gradient descent (online gradient descent) with a fixed learning rate.
There seems to be hundreds of different algorithms which claim to perform better, but I have found it difficult to find out which one would be most promising to implement (with complexity and time to code up/understand being costs).
Also, I am using theano on python.
Best Answer
@Will, you are cooking. You are asking "what is the best spice to go in my soup".
Can you give an idea of the scale of the inputs, the nature of the data, the topology of the network, the types of transfer functions, and the problem with the convergence in the rates that you have tried? What is the dimension and nature of the outputs.
The behavior different training methods depend on those sorts of things.
I'm going to assume that you pre-conditioned your data. You scaled and centered the result so that you are modeling variation from the expected value.
If you are using a single layer single neuron perceptron without feedback, then least squares might not be a bad idea. I say this to point out a case where one method essentially non-iterative and best, but in nearly all other cases it can be worse than useless.
UPDATE 1:
Would you mind reducing the number of hidden-layer neurons by an order of magnitude and re-running for a bit and describing how the error evolves over time?
The higher number of neurons makes the floor lower, but makes getting there slower.