In Simple Neural Network back propagation, we normally use one round of forward and back propagation in every iteration. Let's assume, we have one training example for any arbitrary dimensions, and some initial weights. Then using forward propagation, we calculate the predicted output. This predicted output is then used to calculate the total error which is the back propagated to Re-calculate the weights. After recalculating the weights for all the layers, we update the weights for all the layers all at once. It's not like first we update the weights of one layer and then the other, but instead we first recalculate the weights of all layers( layer by layer ) and then update all at once. We can conclude that "
Re-calculating of the weights layer by layer and then updating the
weights with recalculated weights all at one for all the layers".
Does this makes sense? Is it the right way of weight update using back propagation?
Now Let's assume, I have "m" examples instead of just one example. In case of "m" examples, each of these small gradient steps will be taken after one back propagation iteration over all examples "m".
I am confused that in case of "m" examples, this back propagation
works on these examples one by one. Like, it first takes the first
example and update the weights. Then it takes the second example and
calculate the weight again. then it takes the third example and
calculate the weight and so on. Then in the last when it has run over
all the examples, only then it takes the single step towards optimum
point. If that is the case, is there any relation between weights for
one example to the weights for another example?? As the BP is is
recalculating the weights for each examples in sequence?
Best Answer
A batch of data is taken for feed-forward and "Back-propagation" is performed on the number of examples in that batch. Wights and bias are updated on the basis of change of average error/batch. Then change in weights are updated in the previous wights before performing feed-forward on the next batch of data. A detailed explanation is given in the following book:
http://neuralnetworksanddeeplearning.com/chap2.html