Solved – Neural Networks: Is an epoch in SGD the same as an epoch in mini-batch

machine learningneural networks

In SGD an epoch would be the full presentation of the training data, and then there would be N weight updates per epoch (if there are N data examples in the training set).

If we now do mini-batches instead, say in batches of 20. Does one epoch now consist of N/20 weight updates, or is an epoch 'lengthened' by 20 so that it contains the same number of weight updates?

I ask this as in a couple of papers learning seems to be too quick for the number of epochs stated.

Best Answer

In the neural network terminology:

  • one epoch = one forward pass and one backward pass of all the training examples
  • batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.
  • number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).

Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.

Related Question