Solved – Is WEKA’s logistic regression tool slow? If yes, are there faster alternatives

javalogisticregressionweka

I have to run logistic regression several thousand times, within a Java/Scala program. A general spec for one
of the problems is:

~300 continuous attributes

~<10000 positives, and randomly pick double the number of negatives as positives

The attributes are the same for all problems, and the positives/negatives are always drawn from the same pool, of about 100,000.

I'm finding that naively plugging this into WEKA takes a few minutes per problem, which is unsuitable given the number of tasks which need to be performed. Is there a faster library I can use, or is there a way to parallelize these problems, or something else?

Best Answer

The fastest I know of is Vowpal Wabbit by John Langford and his teams, first in Yahoo! and then at Microsoft. The implementation tweaks and tricks in that code are exceptional.

Its implementation is in C++. So, one option is to call it as an external tool, since it accepts data as files from disk or from stdin. More interesting option is to use it as a web service since it provides a network interface for data input. Probably easiest is the first one, while the second could be way more efficient (no read/writing on disk) but needs some more time invested in programming if you haven't done something similar before. In either case you can first export some data in its format and check out if its speed is up to your expectations.

Related Solutions

Solved – Alternatives to logistic regression in R

Popular right now are randomForest and gbm (called MART or Gradient Boosting in machine learning literature), rpart for simple trees. Also popular is bayesglm, which uses MAP with priors for regularization.

install.packages(c("randomForest", "gbm", "rpart", "arm"))

library(randomForest)
library(gbm)
library(rpart)
library(arm)

r1 <- randomForest(y~x)
r2 <- gbm(y~x)
r3 <- rpart(y~x)
r4 <- bayesglm(y ~ x, family=binomial)

yy1 <- predict(r1, data.frame(x=xx))
yy2 <- predict(r2, data.frame(x=xx))
yy3 <- predict(r3, data.frame(x=xx))
yy4 <- predict(r4, data.frame(x=xx), type="response")

Solved – What are the practical & interpretation differences between alternatives and logistic regression

Disclaimer: It is certainly far from being a full answer to the question!

I think there are at least two levels to consider before establishing a distinction between all such methods:

whether a single model is fitted or not: This helps opposing methods like logistic regression vs. RF or Gradient Boosting (or more generally Ensemble methods), and also put emphasis on parameters estimation (with associated asymptotic or bootstrap confidence intervals) vs. classification or prediction accuracy computation;
whether all variables are considered or not: This is the basis of feature selection, in the sense that penalization or regularization allows to cope with "irregular" data sets (e.g., large $p$ and/or small $n$) and improve generalizability of the findings.

Here are few other points that I think are relevant to the question.

In case we consider several models--the same model is fitted on different subsets (individuals and/or variables) of the available data, or different competitive models are fitted on the same data set--, cross-validation can be used to avoid overfitting and perform model or feature selection, although CV is not limited to this particular cases (it can be used with GAMs or penalized GLMs, for instance). Also, there is the traditional interpretation issue: more complex models often implies more complex interpretation (more parameters, more stringent assumptions, etc.).

Gradient boosting and RFs overcome the limitations of a single decision tree, thanks to Boosting whose main idea is to combine the output of several weak learning algorithms in order to build a more accurate and stable decision rule, and Bagging where we "average" results over resampled data sets. Altogether, they are often viewed as some kind of black boxes in comparison to more "classical" models where clear specifications for the model are provided (I can think of three classes of models: parameteric, semi-parametric, non-parametric), but I think the discussion held under this other thread The Two Cultures: statistics vs. machine learning? provide interesting viewpoints.

Here are a couple of papers about feature selection and some ML techniques:

Saeys, Y, Inza, I, and Larrañaga, P. A review of feature selection techniques in bioinformatics, Bioinformatics (2007) 23(19): 2507-2517.
Dougherty, ER, Hua J, and Sima, C. Performance of Feature Selection Methods, Current Genomics (2009) 10(6): 365–374.
Boulesteix, A-L and Strobl, C. Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction, BMC Medical Research Methodology (2009) 9:85.
Caruana, R and Niculescu-Mizil, A. An Empirical Comparison of Supervised Learning Algorithms. Proceedings of the 23rd International Conference on Machine Learning (2006).
Friedman, J, Hastie, T, and Tibshirani, R. Additive logistic regression: A statistical view of boosting, Ann. Statist. (2000) 28(2):337-407. (With discussion)
Olden, JD, Lawler, JJ, and Poff, NL. Machine learning methods without tears: a primer for ecologists, Q Rev Biol. (2008) 83(2):171-93.

And of course, The Elements of Statistical Learning, by Hastie and coll., is full of illustrations and references. Also be sure to check the Statistical Data Mining Tutorials, from Andrew Moore.

Best Answer

Related Solutions

Solved – Alternatives to logistic regression in R

Solved – What are the practical & interpretation differences between alternatives and logistic regression

Related Question