What is the difference between a Convolutional Neural Network (CNN) and an ordinary Neural Network (NN)? What does convolution mean in this context?
Solved – What makes neural network a *convolutional* neural network
conv-neural-networkconvolutiondeep learningneural networksterminology
Related Solutions
If I understand you correctly, the question is how to train the net if you have pooling layers? Well, the weights in pooling layers are not that different from the ones in "normal" layers. Imagine you have a max pooling layer with grid size 3x3. Imagine further that for a given training example, pixel number 5 (that is, in position (2,2) ) has had the max value in forward propagation, i.e. its value has been passed through the max pooling layer. When doing backprop for that sample, the weight between your pixel number 5 and the output of the pooling is simply one, while for the other eight pixels it is zero. And since the max pooling does not do any further transformation, the error used is that from the layer that came after the max pooling layer. For a more mathematical formulation, there is a nice website: http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/
I'll first try to share some intuition behind CNN and then comment the particular topics you listed.
The convolution and sub-sampling layers in a CNN are not different from the hidden layers in a common MLP, i. e. their function is to extract features from their input. These features are then given to the next hidden layer to extract still more complex features, or are directly given to a standard classifier to output the final prediction (usually a Softmax, but also SVM or any other can be used). In the context of image recognition, these features are images treats, like stroke patterns in the lower layers and object parts in the upper layers.
In natural images these features tend to be the same at all locations. Recognizing a certain stroke pattern in the middle of the images will be as useful as recognizing it close to the borders. So why don't we replicate the hidden layers and connect multiple copies of it in all regions of the input image, so the same features can be detected anywhere? It's exactly what a CNN does, but in a efficient way. After the replication (the "convolution" step) we add a sub-sample step, which can be implemented in many ways, but is nothing more than a sub-sample. In theory this step could be even removed, but in practice it's essential in order to allow the problem remain tractable.
Thus:
- Correct.
- As explained above, hidden layers of a CNN are feature extractors as in a regular MLP. The alternated convolution and sub-sampling steps are done during the training and classification, so they are not something done "before" the actual processing. I wouldn't call them "pre-processing", the same way the hidden layers of a MLP is not called so.
- Correct.
A good image which helps to understand the convolution is CNN page in the ULFDL tutorial. Think of a hidden layer with a single neuron which is trained to extract features from $3 \times 3$ patches. If we convolve this single learned feature over a $5 \times 5$ image, this process can be represented by the following gif:
In this example we were using a single neuron in our feature extraction layer, and we generated $9$ convolved features. If we had a larger number of units in the hidden layer, it would be clear why the sub-sampling step after this is required.
The subsequent convolution and sub-sampling steps are based in the same principle, but computed over features extracted in the previous layer, instead of the raw pixels of the original image.
Related Question
- Solved – In convolutional neural network, what does fully-connected layer mean
- Solved – How does a Stacked AutoEncoder increases performance of a Convolutional Neural Network in image classification tasks
- Solved – the intuition behind convolutional neural network
- Solved – Convolutional neural networks backpropagation
Best Answer
Starting from the Neural Network perspective:
I would say that the base Neural Network has all neurons interconnected between layers. The convolutional version simplifies this model using two hypotheses:
The first asumption is expressed by setting to zero the weights leading to a hidden neuron, except for a region of interest/patch from the input.
Shift invariance is obtained by sharing the same weights across all the patches. In order to capture features anywhere in the image, it is simpler to pave the input with patches only slided by one pixel.
Those simplifications drastically reduce the number of parameters and lead to much simpler computations which 'happen' to take the form of a convolution, hence the C in CNN.
Note 1: the fixed feature size hypothesis is alleviated by the use of multiresolution and/or by using separate networks with different patch sizes.
Note 2: equivariance is usually not as useful as invariance, so the latter is often emulated with additional pooling layers.
Alternative approach
Before deep learning, a popular problem solving method was to extract features and feed them to a classifier. For images, the features were often extracted using expertly chosen filters such as Gabor filters/wavelets. On can view CNN as a parameterized filtering function, where parameters are trained using methods for Neural Networks