Solved – Convolutional neural network – Using absolute of tanh on convolution output

conv-neural-networkfeature-engineeringmachine learningneural networks

I've watched an online lecture regarding CNN (https://www.youtube.com/watch?v=wORlSgx0hZY) that confused me a bit. At roughly 8:35 in the lecture it was stated that it is important to use the absolute value of the tanh function on the output from the convolutional layer. I've never heard that before and I can't quite understand it just based on the lecture. Can anyone expand on that?

The lecture connects this to how "the polarity of features is often irrelevant to recognize objects" and "the rectification eliminates cancellations between neighboring filter outputs when combined with average pooling".

I've attached an excerpt of the slide used in the lecture below:
enter image description here

Best Answer

When you are trying to recognize objects, edges, etc., the algorithms generally look for deviation from nearby or summed/averaged elements. The deviation could be negative or positive, but in most cases it is a "recognition" regardless of the polarity. The algorithms impart polarity due to the mathematics, but the interpretation doesn't care if it is a deviation up or down.

However, some filters and algorithms do carry information in the polarity. If you combine one where polarity matters with one where polarity does not, then you want to be sure you are adding a "recognition" properly, so the filter is augmented appropriately and not inappropriately pushed more negative or more positive.