Solved – Should I use Binary Cross Entropy to evaluate model with Unbalanced Dataset

Question

My dataset has a really high ratio between positive and negative class (neg/pos≈1000).

My current objective is making prediction of positive class as high as possible.

Is it reasonable to use Binary Cross Entropy to evaluate my classification model with this unbalanced dataset ?

Answer 1

Binary cross entropy is probably not the best metric for you to use if you want your model to have high recall (ratio of correctly labeled positive samples). You could modify it to a weighted version:

$-\frac{1}{N}\sum_n wt_n\log(y_n) + (1-t_n)\log(1-y_n)$

where $w$ is the weight of positive samples (in your case it could be 1000).

Also, F-score is a measure designed for balancing high recall and high specificity.