hyperparameterloss-functionsmachine learning
I would like to know if exists a rule of thumb to set the $\gamma$ parameter in Focal Loss when we have very imbalanced classes.
The focal loss first appeared in Focal Loss for Dense Object Detection by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár
Huber loss will clip gradients to delta for residual (abs) values larger than delta. You want that when some part of your data points poorly fit the model and you would like to limit their influence. Also, clipping the grads is a common way to make optimization stable (not necessarily with huber).
Set delta to the value of the residual for the data points you trust.
See how the derivative is a const for abs(a)>delta
import numpy as np
import matplotlib.pyplot as plt
def huber(a, delta):
value = np.where(np.abs(a)<delta, .5*a**2, delta*(np.abs(a) - .5*delta))
deriv = np.where(np.abs(a)<delta, a, np.sign(a)*delta)
return value, deriv
h, d = huber(np.arange(-1, 1, .01), delta=0.2)
fig, ax = plt.subplots(1)
ax.plot(h, label='loss value')
ax.plot(d, label='loss derivative')
ax.grid(True)
ax.legend()
this guide lists the typical values for max_depth of xgboost as 3-10 - how is this range decided on as typical?
You guessed it correct. Typical means in most problems, this parameter is chosen something in between 3-10. This is based on experimentation, trial and error and a just rough guide. Another guide could have said 3-12, and it wouldn't be a wrong guide. This is problem and data dependent. And, there are computational considerations as well. Maybe the business problem is not that sensitive and you wouldn't want to wait a whole day just to get a 0.5% improvement.
However, although there aren't set in stone rules for choosing max depth candidates (at least to the best of my knowledge), you can choose reasonable values/ranges based on your number of features, and dataset size. For instance, if your dataset has around 1000 samples, and if each split is assumed to split the space into half, in 10 splits, your leaves would be left with only one or two samples, which seems like overfitting (not necessarily but it looks like). You can come up with some bounds following this kind of logic.
Best Answer
The parameter $\gamma$ smoothly adjusts the rate at which easy examples are down-weighted and that is quite dataset and application dependent. In the paper, the focal loss is actually given as: $−\alpha (1 − p_t)^\gamma \log(p_t)$ which is a reformulated view of the standard cross-entropy loss and the class imbalance itself is "controlled" by $\alpha$ rather than $\gamma$. People often treat the focal loss as a tool to primarily address class imbalance whether it is actually a tool to primarily address information asymmetry during learning; i.e. focal loss can very relevant when training with a balanced set where one of the two classes is easy to distinguish. But to bring us back: there is no good rule of thumb aside setting $\gamma=2$ (as the paper suggests) and then adjusting it based on our evaluation criteria. This point highlights the difference between a loss function and an evaluation metric; CV.SE has thread on Loss function and evaluation metric if you want to explore this distinction further but the main point to carry here is that $\gamma$ (i.e. a hyper-parameter of our loss) needs to be adjust in relation to our evaluation criteria and "on it's own" is often meaningless.