In a simple neural network with two neurons in series carrying weights w1 and w2 and just one input, what is limitation behind starting off with both weights being same?
I am referring to patrick winston's MIT lectures and to quote him ".. By the way what would happen if you started off with all the weights being the same? Nothing, because it would always stay the same"
Since the two neurons are in series i fail to see why they would necessarily remain same
Thank you in advance
Best Answer
Firstly, see the answer to this post
Furthermore, to expand on that answer, if all hidden neurons and subsequently all output neurons have the same activation values, there would only be $k$ different updates for the weights between the hidden and the output layer where $k$ is the number of output neurons. This is easily understandable by looking at the update rule.
This is bad because the power of neural networks comes from their ability to associate different weights to different neurons (parameters) and by initializing the network with equal weights, you are forcing it to yield groups of same-valued weights.