I want to use the encoder of my autoencoder for feature extraction in an image anomaly detection framework.
For that reason, I thought that pretraining the autoencoder on a large dataset and then fine-tuning it on my target dataset would be a good idea. This idea crossed my mind because many anomaly detection approaches use CNN architectures like VGG, ResNet etc. as a feature extractor, which are pretrained on ImageNet.
I did not find papers regarding this matter and therefore my question if transfer learning is even really used on autoencoders?
Best Answer
This approach often used in NLP models, so called via tuning and it is what you called transfer learning. A recent debate on this so called Foundation models actually revolves around this concept.
Yes, it is possible, as these new foundation models approach training a large vision models or autoencoders. However, VGG or Resnet are very tiny models to be used as a foundational model compare to language models like GPT-3 or similar in size and training data coverage.