Assuming nothing special in your particular case, I think there is a good argument for either using the default (Mean Square Error) or use the mean of the error of the logs, or even the chi-squared error.
The purpose of the cost function is to express how "upset" you are with wrong predictions, specifically what "wrongness" bothers you most. This is particularly important for binary responses, but can matter in any situation.
Mean Square Error (of responses)
$C = \frac{1}{n}\sum_i (Y_i-\hat Y_i)^2$
Using the MSE you are equally sensitive to errors from above and below and equally sensitive for large and small predictions. This is a pretty standard thing to do, and so I don't think would be frowned on in most situations.
Mean Square Error (of log responses)
$C = \frac{1}{n}\sum_i (\ln Y_i-\ln \hat Y_i)^2$
Because you are working with count data, it could be argued that you are not symmetric nor size indifferent. Being out by 10 counts for a prediction of 10 is very different from a prediction of 1000. This is a somewhat "canonical" cost function, because you have matched the costs up to the link function. This ensures that that costs match the variance distribution being assumed in the model.
Chi-Squared Error
$C = \frac{1}{n}\sum_i \frac{(Y_i-\hat Y_i)^2}{\hat Y_i}$
A third way would be to use the chi-squared error. This could be particularly appealing if you are comparing your GLM to other count based models - particularly if there are factors in your GLM. Similar to the error log responses, this will scale with size, but it is symmetric around the predicted count. You are now evaluating goodness of fit based on percentage error.
On The Discreteness
The question cites the documentation example where they have a binary response variable, so use a different cost function. The issue for a binary response is that the GLM will forecast a real number between 0 and 1, even though the response is always exactly 0 or 1. It is perfectly valid to say that the closer that number is to the correct response the better the forecast, but often people don't want this. The reasoning being that one often must act either as though it is 0 or 1, and so will take anything less than 0.5 as a forecast for 0. In that case, it makes sense simply to count the number of "wrong" forecasts. The argument here is that for a True/False question you can only ever be right or wrong - there is no gradation of wrongness.
In your case you have count data. Here it is far more common to accept predictions that are not on the same support as the response. A prediction of 2.4 children per family for example, or 9.7 deaths per year. Usually one would not try to do anything about this because it is not about being "right" or "wrong", just as close as you can get. If you really must have a prediction that is an integer though, perhaps because you have a very very low count rate, then there is no reason you can't round the prediction first and count the "whole number" or error. In this case, the three expressions above still apply, but you simply need to round $\hat Y$ first.
OK, No answers to my post. But I think I got the answer. All credits go to @Feng Mai. He wrote a post here: What is the cost function in cv.glm in R's boot package? and thanks to it here is my answer to my question:
For a cutoff value of 0.2, I think that I could I apply the following cost function:
mycost <- function(r, pi){
weight1 = 1 #cost for getting 1 wrong
weight0 = 1 #cost for getting 0 wrong
c1 = (r==1)&(pi<0.2) #logical vector - true if actual 1 but predict 0
c0 = (r==0)&(pi>0.2) #logical vecotr - true if actual 0 but predict 1
return(mean(weight1*c1+weight0*c0))
}
And then I would use the cv.glm function with the fitted model and mycost function:
cv.glm(data, glmfit, cost=mycost, K)
Hopefully this might work. Am I right?
Best Answer
r is a vector that contains the actual outcome, pi is a vector that contains the fitted values.
This is saying $cost = \sum|r_i - pi_i|$. You can define your own cost functions. In your case for binary classification you can do something like this
and put mycost as an argument in the cv.glm function.