I have been randomly initializing weights for my neuron layers. I did some calculations on the paper and realized at least for the initial few iterations the probabilities might not add up while summing the results from the output layer. Is it acceptable or there should be weight distribution from the initial iteration itself so that the probabilities add up to one. If so how to initialize the weights for the neural networks.
Solved – How to initialize the weights of neural networks so that sum of probabilities at output layer adds up to 1
neural networks
Related Solutions
Sparse representations are expected in hierarchical models. Possibly, what you are discovering is a problem intrinsic to the hierarchical structure of deep learning models. You will find quite a few scientific papers on "sparse representations", especially in memory research.
I think you would benefit from reading about "receptor fields" in visual cortex. Not only are there ON and OFF cells in the mammal brain, but also RF cells that fire both during ON and OFF. Perhaps the edge/sparsity problem could be circumvented by updating the model to reflect current neuroscience on vision, especially in animal models.
I think its about saturation of the neurons. Think about you have an activation function like sigmoid.
If your weight val gets value >= 2 or <=-2 your neuron will not learn. So, if you truncate your normal distribution you will not have this issue(at least from the initialization) based on your variance. I think thats why, its better to use truncated normal in general.
Related Question
- Solved – Neural networks – how can I interpret what a hidden layer is doing to the data
- Solved – In Neural Network back propagation, how are the weights for one training examples related to the weights for next training examples
- Solved – What are the theoretical/practical reasons to use normal distribution to initialize the weights in Neural Networks
- Solved – Why does each convolution layer require activation function and weight initialization
Best Answer
You don't need to hack the initial weights. You may need to revisit the architecture of the neural network. For example, add a softmax layer as a final layer so that the output can be interpreted as probabilities for classes.