Solved – Weight variables for predictive model

logisticpredictive-modelsvalidation

I received a question today that I wasn't exactly sure how to answer.

I have built a predictive model using a fairly basic logistic regression that works pretty well and fits our business needs. Recently, we purchased a CRM tool that allows us to build "probability" scores, but only allows the end users to give integer weights to various factors. Said differently, one can arbitrarily assign a weight of 10 points to one factor and -5 points to another with the sum of all weights representing the "probability" for a given entity in our database.

What I am looking to do is translate my model to this new format such that the resulting score equals the calculated probability from my logistic model. This is not out of desire, but business needs.

Admittedly I am not sure how to use the calculated coefficients and "adjust" them to these requirements. What is the best approach, if any? General thoughts on how to assign statistically valid integer weights to business criteria given these constraints?

Any thoughts or insight will be very much appreciated.

Best Answer

Unfortunately you're not going to be able to create the exact solution you're looking for. The company's existing system depends on linear relationships between the factors and the final score, which is a proxy for probability. Your logistic model, on the other hand, depends on S-shaped curves rather than linear relationships between factors and the probabilities. The latter are bounded at 0 and 1; if you were to try to use linear weights to compute probabilities, you would no doubt have to assign to certain cases probabilities less than zero or greater than one. This is one of the classic reasons why logistic regression is preferred over linear regression when the outcome variable is binary.

Your best bet, from a statistical point of view, is to create the best logistic model you can and to use that instead of the existing linear weights system. This will give you the best predictive accuracy while also keeping all predicted probabilities in a reasonable range.

Related Solutions

Solved – Kappa for Predictive Model

It might be useful to consider Cohen's $\kappa$ in the context of inter-rater-agreement. Suppose you have two raters individually assigning the same set of objects to the same categories. You can then ask for overall agreement by dividing the sum of the diagonal of the confusion matrix by the total sum. But this does not take into account that the two raters will also, to some extent, agree by chance. $\kappa$ is supposed to be a chance-corrected measure conditional on the baseline frequencies with which the raters use the categories (marginal sums).

The expected frequency of each cell under the assumption of independence given the marginal sums is then calculated just like in the $\chi^2$ test - this is equivalent to Witten & Frank's description (see mbq's answer). For chance-agreement, we only need the diagonal cells. In R

# generate the given data
> lvls <- factor(1:3, labels=letters[1:3])
> rtr1 <- rep(lvls, c(100, 60, 40))
> rtr2 <- rep(rep(lvls, nlevels(lvls)), c(88,10,2, 14,40,6, 18,10,12))
> cTab <- table(rtr1, rtr2)
> addmargins(cTab)
     rtr2
rtr1    a   b   c Sum
  a    88  10   2 100
  b    14  40   6  60
  c    18  10  12  40
  Sum 120  60  20 200

> library(irr)       # for kappa2()
> kappa2(cbind(rtr1, rtr2))
 Cohen's Kappa for 2 Raters (Weights: unweighted)
 Subjects = 200 
   Raters = 2 
    Kappa = 0.492 
        z = 9.46 
  p-value = 0 

# observed frequency of agreement (diagonal cells)
> fObs <- sum(diag(cTab)) / sum(cTab)

# frequency of agreement expected by chance (like chi^2)
> fExp <- sum(rowSums(cTab) * colSums(cTab)) / sum(cTab)^2
> (fObs-fExp) / (1-fExp)    # Cohen's kappa
[1] 0.4915254

Note that $\kappa$ is not universally accepted at doing a good job, see, e.g., here, or here, or the literature cited in the Wikipedia article.

Solved – Predictive Model for Attribution Model

I think they way you are thinking about it is correct, but I would add a little bit more related to how I think the company would use this information.

1.) Maximize your return on investment. You eluded to this concept in your question, but you missed a key concept: The cost of the variable. Let's consider a simple model of two advertising strategies, A and B, which cost 100\$ and 500\$ respectively. If you acquire 30 costumers for A and 60 customers from B who will pay a 10\$ subscription fee, strategy A would yield a return of 200\$ while A would yield 100\$. Thus, it would make more sense to shift resources to A from B, even though B get's you more customers.

2.) Better coordination between services. In point 1, we assumed uncorrelated events. However, it could very well be that the customer bought the product because A occurred before B and not when B occurred before A. Thus, you would also want to account for this in your return on investment. Finding correlations between marketing strategies could yield some very interesting and non-trivial ways to increase the chances of acquiring new customers.

3.) Gaining a better understanding of your customers psychology to create new marketing techniques.. These models are not just numbers, they are representations of complex thought processes across a wide range of individuals. To understand why your customer purchased your product may be the biggest gain from this model.

I'm not really sure if this constitutes an answer, but this is how I would leverage the data if I were in their position. I'm sure there are many other ways as well.

Best Answer

Related Solutions

Solved – Kappa for Predictive Model

Solved – Predictive Model for Attribution Model

Related Question