I've just started reading ML papers and occasionally I encounter the word 'train NLL' and 'test NLL', which I cannot figure out the meaning. What does they mean?
Solved – What does ‘train NLL’ and ‘test NLL’ mean
machine learningterminology
Related Solutions
Intuition
A saturating activation function squeezes the input.
Definitions
- $f$ is non-saturating iff $ (|\lim_{z\to-\infty} f(z)| = +\infty) \vee (|\lim_{z\to+\infty} f(z)| = +\infty) $
- $f$ is saturating iff $f$ is not non-saturating.
These definitions are not specific to convolutional neural networks.
Examples
The Rectified Linear Unit (ReLU) activation function, which is defined as $f(x)=max(0,x)$ is non-saturating because $\lim_{z\to+\infty} f(z) = +\infty$:
The sigmoid activation function, which is defined as $f(x) = \frac{1}{1 + e^{-x}}$ is saturating, because it squashes real numbers to range between $[0,1]$:
The tanh (hyperbolic tangent) activation function is saturating as it squashes real numbers to range between $[-1,1]$:
(figures are from CS231n, MIT License)
Let's say you want to achieve a certain objective. For instance, you want to find $z_i$ that minimize the sum of squares: $$\sum_i(y_i-z_i)^2$$ where $y_i$ are given. In this case obviously $z_i=y_i$ would minimize the sum, in fact it would make it zero.
Now, what if we wanted to impose some other condition on $z_i$? For example, I'd like to penalize the length of the piece-wise linear curve that goes through $z_i$. The longer the curve, the higher is the penalty.
Here's how I could do it: change the objective to the following: $$\sum_i(y_i-z_i)^2+\sum_i \sqrt{1+(z_i-z_{i-1})^2}$$
Now if you make $z_i\ne y_i$ then you can reduce the second term of the above objective a little more than increasing the first term, and the net effect would be lower objective than when $z_i=y_i$.
This is a general idea, and you can apply it to many situations such as SVD, where you're minimizing some kind of function too. You add a penalty to it, and get a different solution.
Best Answer
NLL is an abbreviation for "negative log likelihood".
It serves the same purpose as the Maximum Log Likelihood. However, since most optimization problems typically attempt to minimize a function, the negative log likelihood is what is preferred to the Maximum Log likelihood.