I am a new to deep learning and neural networks, and I need to know if there is a good weights initialization method to use if the activation function is Softmax like Tanh, ReLU and Sigmoid. Related answer.
Solved – Softmax weights initialization
deep learningmachine learningneural networkssigmoid-curvesoftmax
Related Solutions
You should not use a non-linearity for the last layer before the softmax classification. The ReLU non-linearity (used now almost exclusively) will in this case simply throw away information without adding any additional benefit. You can look at the caffe implementation of the well-known AlexNet for a reference of what's done in practice.
Rectified Linear Units (ReLUs) only make the output of the neurons to be non-negative. The parameters of the network, however, can, and will, become positive or negative depending on the training data.
Here are two reasons I can think of right now that justifies (intuitively) why some parameters would become negative:
the regularization of the parameters (a.k.a. the weight decay); the variation in the parameter values makes prediction possible, and if the parameters are centered around zero (i.e. their mean is close to zero), then their $\ell 2$ norm (which is a standard regularizer) is low.
although the gradients of the output of a layer with respect to the layer parameters depend on the input to the layer (which are always positive assuming that the previous layer passes its outputs through a ReLU), however, the gradient of the error (which comes from the layers closer to the final output layers) may be positive or negative, making it possible for SGD to make some of the parameter values negative after taking the next gradient step. More specifically, let $I$, $O$, and $w$ denote the input, output, and parameters of a layer in a neural network. Also, let $E$ be the final error of the network induced by some training sample. The gradient of the error with respect to $w$ is computed as $\frac{\partial E}{\partial w} = \left( \sum_{k=1}^K\frac{\partial E}{\partial O_k} \right) \cdot \frac{\partial O_k}{\partial w}$; note that $O_k = O, \forall k$ (see picture below):
Best Answer
For:
When initializing the weights with a normal distribution, all these methods use mean 0 and variance σ²=scale/fan_avg or σ²=scale/fan_in. The fan_in is the layer's number of inputs, the fan_out is the layer's number of outputs (=number of neurons), fan_avg is the average between the two =½(fan_in+fan_out). Specifically:
When initializing the weights with a uniform distribution, all these methods just use the range [-limit, limit] where limit = sqrt(3 * σ²).
If you have consecutive ReLU layers with very different sizes, you may prefer using fan_avg rather than fan_in. In Keras, you can use something like this: