MATLAB: Does “feature selection” consider the interdependence between multiple parameters, e.g. quotient, difference, etc., as meaningful combinations for identification

featurejointpredictorsselectionStatistics and Machine Learning Toolbox

We want to identify important features in our data set. Using the Statistics and Machine Learning Toolbox R2018a we found suitable functions for feature selection. However, in the available feature selection methods, the following case is in our opinion not considered:
"2 Features A and B are not meaningful for distinguishing the important features individually, but the quotient A / B is very good."
Does feature selection consider the interdependence between multiple parameters, e.g. quotient, difference, etc., as meaningful combinations for identification?

Best Answer

I understand that you are interested in the interdependence or joint ability of predictors to influence classification even when individual predictors are not useful. If you want to generate specific additional predictors based on original predictors– such as A./B based on A and B – then you can do this easily at the command line. Possible methods are: FSCNCA and FSRNCA
  • Methods that select features using neighborhood component analysis (NCA).
  • They return a set of weights for each specified feature, and indicate the important features using NCA.
Functions fscnca and fsrnca can detect such interaction or interdependence – the exact form of useful interdependence (A/B, A-B etc.) does not need to be specified. An example to illustrate this concept is attached (see “testfscna” as ”.m” and “.html”).
FITLM, FITGLM and linear model variants (e.g. FITLME)
  • Methods that accept a formula through which one can specify checking for interactions
The Example “<http://www.mathworks.com/help/releases/R2018a/stats/fitlm.html#bunisqg-3 Fit Linear Regression Using Specified Model Formula>” gives a good overview on the required steps. Additional information on the formula for model specification can be found here.
Decision Trees with interaction test
  • Very good method for detecting bivariate combinations of predictors.
For instance, the following will find powerful predictors in input data X and class labels y:
>> bag = fitcensemble(X, y, 'Method', 'Bag', 'Learner', templateTree('PredictorSelection', 'interaction-curvature'))
>> oobPermutedPredictorImportance(bag)
If you are interested in regression, replace fitcensemble with fitrensemble. You can also learn a model using all variables, take out a specific combination of variables and see how much removing these variables affects the model accuracy. An exhaustive list of variable combinations can be generated using the combnk function.
FSCNCA and FSRNCA with manually augmented features
  • Method to manually handle formulas which can specify checking for interactions
Augment the features with a new feature that represents the interaction between other features (e.g. A/B) and pass this augmented set to fscnca to see the importance weight for that new feature.
You can use x2fx in creating this new augmented matrix, see https://www.mathworks.com/help/releases/R2018a/stats/x2fx.html
Note that feature selection only accepts a numeric predictor matrix, it does not accept categoricals. If we want to use categorical variables, we can specify them for x2fx and x2fx returns the correct matrix with dummy variables for categorical predictors.
Related Question