The problem which I have at hand consists of a logistic regression model for risk evaluation that has been made on some credit card data of Quarter-1'12 (Jan'12 – Mar'12). Now I use the same model to evaluate risk for the data of Quarter-2'12(Apr'12-Jun'12). I want to devise an accuracy score that gives an insight into how "accurately" my model fits the new data (Quarter -2 data).
I have used Hosmer Lemeshow Statistic and Balanced Accuracy Method till now but none served the purpose. What further can be done in this regard?
Best Answer
If I understand correctly, the situation is that you have an estimated logistic regression model that you use in production. So you need a system of quality control that can tell you if the model start to perform worse.
One possibility could be to calculate a running Brier score, you could on the same plot indicate the expected Brier score assuming the model is correct and calibrated. That should give a reference value. There might be some ideas in this stored search.
The Brier score is a proper scoring rule, you could also choose to use some other proper scoring rule.