I used Logistic Regression. I have six features, I want to know the important features in this classifier that influence the result more than other features. I used Information Gain but it seems that it doesn't depend on the used classifier. Is there any method to rank the features according to their importance based on specific classifier (like Logistic Regression)? any help would be highly appreciated.
Solved – Ranking features in logistic regression
feature selectionlogisticrankingregression-strategies
Related Solutions
The absolute value of the coefficient is not proportional to the importance of the corresponding feature. There are two ways to assess the significance of a given feature in logistic regression (and more generally for Generalized Linear Models):
- Look at the p-value of this parameter in the output of the logistic regression
- or: run two models, one with all the features except the feature of interest (the one you want to assess the performance), and run a second model with all the features, including the feature of interest. You can then perform a Likelihood Ratio Test between the two models to see if this feature is significant in the prediction task
The second approach is more reliable.
Since you are interested in ranking the categories, you may want to re-code the categorical variables into a number of separate binary variables.
Example: Create a binary variable for express delivery- which would take the value 1 for express delivery cases and 0 otherwise. Similarly, a binary variable for standard delivery.
For each of these recoded binary variables you can calculate the marginal effects as indicated below:
Let me explain a bit on the above equation: lets say d is the re-coded binary variable for express delivery
is the probability of event evaluated at mean when d=1
is the probability of event evaluated at mean when d=0
Once you calculate the marginal effects for all the categories (re-coded binary variables) you can rank them.
Best Answer
I think the answer you are looking for might be the Boruta algorithm. This is a wrapper method that directly measures the importance of features in an "all relevance" sense and is implemented in an R package, which produces nice plots such as
where the importance of any feature is on the y-axis and is compared with a null plotted in blue here. This blog post describes the approach and I would recommend you read it as a very clear intro.