Solved – the difference between “learnable parameters” and “weights” in neural networks

Question

I think the learnable parameters are the set of all weights and biases in the neural network, but is it true in general?

Answer 1

Learnable parameters usually means weights and biases, but there is more to it - the term encompasses anything that can be adjusted (i.e. learned) during training.

There are weights and biases in the bulk matrix computations; when thinking e.g. about a Conv2d operation with its number of filters and kernel size.

There are also parameters that are learned in layers such as BatchNorm, which are referred to as parameters (not just weights) perhaps because they are often discussed in terms of simple equations, where they look like coefficients in an equation. The parameters of this equation (the scale and shift) are learnt during training, so are included in the number of learnable parameters, but are a bit different from normal weights and biases. Have a look at the original paper (Ioffe, Szegedy, 2015) for more information/interpretation.