Suppose that we have a trained CNN, there is 5 conv layers and 3 fully connected layers. We take the first 5 conv layers as it is (with their parameter settings: like kernel size, activation etc) and their weights and biases which are trained before by using another dataset.
If we want to profit from this knowledge in our new model (which has the same 5 conv layers at the beginning but later is different), do we:
- Continue to train the whole model (at the beginning: 5 conv layers is initialized to the parameters found in the previous training and the later layers some other initialization (e.g. he initialization for weights and 0 for biases)) with our data?
or
- We keep the first 5 conv layers as in the test mode (no more update) and only update/train the parameters in later layers?
Which one is understood/done when we talk about transfer learning?
Note: I am not very familiar with all the terminology used in deep learning. However, I hope that I could at least explain my question.
Best Answer
Stanford University lists three types of transfer learning.
Note that the last type really only differs in that the pre-trained model is typically large and publicly available.
When to use which approach depends on how much data you have and how different it is from the pre-trained model (for more details see the link).
\begin{array} {|l|l|l|} \hline \bf \text{Amount of data}& \bf \text{Similarity of data}& \bf \text{Procedure} \\ \hline Low & Low & \text{Unfreeze more layers of pretrained model}\\ \hline Low & High & \text{Unfreeze fewer layers of pretrained model}\\ \hline High & Low & \text{Initialize with pretrained weights and train completely} \\ \hline High & High & \text{Fine-tune completely} \\ \hline \end{array}
CS231n Convolutional Neural Networks for Visual Recognition