Solved – How to use Random Forest for categorical variables with missing value

categorical datamissing datapredictive-modelsrandom forest

I have a labelled dataset of 1M rows and 600 features. I am trying to build a supervised learning model for prediction. I am particularly working with Random forests in R. The data I have has following properties.

Most of the features are categorical in nature.
Each categorical variable has multiple levels ( some of them having 20 levels)
Some of the features have data missing

Can random forests work without imputation of these missing values? If no, then what is the best way to impute these missing categorical values? Any literature or R functionality which addresses this issue will be really helpful.

Best Answer

Off the top of my head, I would say that this shouldn't be an issue. The rf package in R implements random forests using CARTs. One of the nicest thing about trees is how they are "natively" capable of dealing with categorical and missing variables. Here is the package documentation; you can download the package itself from CRAN.

Chapter 8 in James, Witten, Hastie, & Tibshirani's Introduction to Statistical Learning with Applications in R offers a good introduction to tree methods and also covers random forests on page 328.

Imputing missing variables is a whole thing in and of itself and, depending on your needs and data, you might be able to get away with not having to do it. If you do have to perform imputation you might want to check here and here for some quick pointers, but you're probably just going to have to read up on imputation methods and make a judgement call on what to go with.

Related Solutions

Solved – How to impute a missing categorical predictor variable for a random forest model

I think you need an unsupervised imputing method. That is one which do not use the target values for imputation. If you only have few prediction feature vectors, it may be difficult to uncover a data structure. Instead you could mix your predictions with already imputed training feature vectors and use this structure to impute once again. Notice this procedure may violate assumptions of independence, therefore wrap the entire procedure in an outer cross-validation to check for serious overfitting.

I just learned about missForest from a comment to this question. missForest seems to do the trick. I simulated your problem on the iris data. (without outer cross-validation)

rm(list=ls())
data("iris")
set.seed(1234)
n.train = 100
train.index = sample(nrow(iris),n.train)
feature.train = as.matrix(iris[ train.index,1:4])
feature.test  = as.matrix(iris[-train.index,1:4])


#simulate 40 NAs in train
n.NAs = 40
NA.index = sample(length(feature.train),n.NAs)
NA.feature.train = feature.train; NA.feature.train[NA.index] = NA

#imputing 40 NAs unsupervised
library(missForest)
imp.feature.train = missForest(NA.feature.train)$ximp
#check how well imputation went, seems promsing for this data set
plot(    feature.train[NA.index],xlab="true value",
     imp.feature.train[NA.index],ylab="imp  value",)

#simulate random NAs in feature test
feature.test[sample(length(feature.test),20)] = NA

#mix feature.test with imp.feature.train
nrow.test = nrow(feature.test)
mix.feature = rbind(feature.test,imp.feature.train)
imp.feature.test = missForest(mix.feature)$ximp[1:nrow.test,]

#train RF and predict
library(randomForest)
rf = randomForest(imp.feature.train,iris$Species[train.index])
pred.test = predict(rf,imp.feature.test)
table(pred.test, iris$Species[-train.index])

Printing...
-----------------
pred.test    setosa versicolor virginica
  setosa         12          0         0
  versicolor      0         20         2
  virginica       0          1        15

Solved – building up a predictive model with lots of features and missing data

There are several possibilities for handling missing data. A typical easy one is imputing the median for continuous and the modus for discrete predictors. Other more sophisticated methods are available (like e.g. imputation with random forest, see here for some possibilites with R-package mlr: http://mlr-org.github.io/mlr-tutorial/devel/html/impute/index.html)
As most of the algorithms for predictive modeling cannot handle missing data, you should do imputation before building a model
Random Forest (randomForest or ranger in R) and linear model (lm in R) are good first options for regression problems. Better results with a bit of parameter tuning can usually be obtained by boosting methods (e.g. with the xgboost package), but a bit more experience is needed for this.

Best Answer

Related Solutions

Solved – How to impute a missing categorical predictor variable for a random forest model

Solved – building up a predictive model with lots of features and missing data

Related Question