conv-neural-networkmachine learningneural networks
I have a sequence of images of shape $(40,64,64,12)$. If I apply conv3d with 8 kernels having spatial extent $(3,3,3)$ without padding, how to calculate the shape of output.
If the next layer is max pooling with $(2,2,2)$, what will be the output shape?
You already understand that the dimension of a single kernel is 3x3x3 and there are 5 kernels. So each kernel is a 2D window of 3x3 pixels and there are 3 components in each kernel, one corresponding to each color channel (R,G,B). When the kernel is placed at a particular location over the input image its 3 components are multiplied (dot product) with the corresponding channel's pixel data to produce a single scalar number for every component (or channel). So you get 3 scalars, one for each channel. Then these scalars are summed up and another scalar representing the bias of the filter is added to the sum. The end result is a single scalar.
You can view an animated demo under the convolution demo section on this page. Use the Toggle Movement button to pause the animation and look at how the output is being generated.
In the screenshot above (taken at a random instance) the portions of the image channels to which convolution is being applied are outlined in Blue. The three components of filter W1 are outlined in Red. If you take the dot product of each 3x3 Blue rectangle with it's component of the filter you get a scalar. The 3 scalars obtained for each channel are summed up along with the bias of Filter W1, shown as a single cell below the filter components. The result is a single scalar, the number 2, outlined green in the last column.
The text also describes how convolution is being carried out.
When you convolve a layer of size 128x128x3 by 3 kernels, you actually end up with a 128x128x3 tensor. This is because each kernel is a 3x3x3 tensor, so you don't end up with exponential growth you described. Each kernel compresses all 3 of the "feature maps" in the last layer into one value. If you only had one kernel, a 128x128x3 input would result in a 128x128x1 output.
Mathematically, if single channel convolution was
$$Y_{i,j} = \sum_{\delta_i, \delta_j} X_{i+\delta_i, j+\delta_j} \cdot K_{\delta_i, \delta_j} $$
Then with multiple channels and one kernel we have
$$Y_{i,j} = \sum_{\delta_i, \delta_j,k} X_{i+\delta_i, j+\delta_j,k} \cdot K_{\delta_i, \delta_j,k} $$
Second, yes, flattening the tensor at the end of the convolutional layers into a vector which is input into fully connected layers is standard practice. Alternatively, you can also average across each feature map, so that a 8x8x1024 tensor is transformed into a 1024 dimension vector.
Best Answer
The convolution formula is the same as in 2D and is well-described in CS231n tutorial:
$$Out = (W−F+2P)/S+1$$
... where $W$ is the input volume size, $F$ is the receptive field size, $S$ is the stride, and $P$ is the amount of zero padding used on the border. In particular, when $S=1$ and $P=0$, like in your question, it simplifies to
$$Out=W-F+1$$
So, if you input the tensor $(40,64,64,12)$, ignoring the batch size, and $F=3$, then the output tensor size will be $(38, 62, 62, 8)$.
Pooling layer normally halves each spatial dimension. This corresponds to the local receptive field size
F=(2, 2, 2)and strideS=(2, 2, 2). Hence, the input tensor $(38, 62, 62, 8)$ will be transformed to $(19, 31, 31, 8)$.But you set the stride
S=(1, 1, 1), it'll reduce each spatial dimension by 1: $(37, 61, 61, 8)$.The last dimension doesn't change.