[Math] How many parameters does the neural network have

Question

We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , …, ℎ𝑙−1 nodes respectively and an output layer of ℎ𝑙 nodes.
How many parameters does the network have?

Answer 1

Suppose the network has $784$ inputs, $16$ nodes in $2$ hidden layers and $10$ nodes in the output layer.

The amount of parameters (meaning weights and bias that make up the cost function) is then:

For the weights: $$784\times 16+16\times16+16\times10=12960$$

For the bias components:

We have $32$ neurons in the hidden layers and $10$ in the output, so we have $$32+10 = 42$$ biases.

So in total, the amount of parameters in this neural network is $13002$.