Solved – Buiding Ensemble model

ensemble learningk nearest neighbourmachine learningnaive bayesr

I'm new to ensemble model. Suppose I've KNN models like this – (in R)
library(class)

pred.y1 <- knn(train.x,test.x,k=3)
pred.y2 <- knn(train.x,test.x,k=5)
pred.y3 <- knn(train.x,test.x,k=6)
pred.y4 <- knn(train.x,test.x,k=8)

Can I build an ensemble model out this different results out of these pred.y1, pred.y2, pred.y3, and pred.y4? If so how I can I achieve it?

Related to the same question, for the same dataset, I was building naiveBayes model with e1071 package, and it appears naiveBayes compensates what KNN lacks, and it may be ideal if I combine KNN and naiveBayes results with ensemble model.

Now, suppose, if I've something like this,

library(e1071)
model <- naiveBayes(train.x,train.y,laplace = 0.6)
pred.yNB <- predict(model,test.x)

Now, how do I build an ensemble model here with pred.yNB and pred.y1, pred.y2, pred.y3,pred.y4?

Are there any packages in R that can be used for building ensemble models independent of base (prior) modeling technique?

Thank you in advance.

Best Answer

Ensemble models in their simplest form are just averaging (regression) or voting (classification) a prediction using multiple models.

So for your example you could average the prediction vector of all 5 models. This is a valid ensemble model.

This paper has an example of using the package ipred with generic models to improve the performance of an ensemble by using Bagging.

You may want to look at Boosting which is the sort of optimization you describe when saying "naiveBayes compensates what KNN lacks".

Related Question