classificationlibsvmlogisticregressionsvm
I have liblinear model file for a classifier learned using logistic regression. In the file, they say, the weight vector and intercept term. But when I simply plot it as
$$w^Tx + b$$
on the original data, it does not seems like the proper decision boundary. But in case of SVM, plotting
$$w^Tx+b$$ seems to be the correct way. Is there any difference in case of Logistic Regression with liblinear ?
Yes, Bayes Classifier is the one which produces the lowest possible test error rate. This is I think best illustrated through an example.
To simplify things a bit, let's say we have a simple two class classification problem. For example, we survey a group of students and collect their age, SAT scores and current GPA and want to predict whether they are going to fail a course or not. So in R it would be something like fail ~ age + sat.score + current.GPA
Bayes classifier works by just looking at the probabilities for each combination of the features and assigning each instance to the class which has the probability bigger than 50%.
Imagine that we do survey all the students that exist. Then in that case, the classifier would know correct probabilities for failing or not for all possible combination of features, and then it will give the best possible classification accuracy.
However, this does not mean that it will be able to classify all instances correctly (i.e., to have 0% error rate) as this is impossible in most of the cases.
In our example, it is very likely that some of the students would have the same values for all three features and yet some of them will fail while others won't. There is no single classifier which will give you 100% correct answer to this problem, as there is no way to differentiate between the students who failed or not (for a classifier, they look the same). Adding new features for example, previous knowledge or IQ etc might help, but then the problem definition would changed and that might improve the classification accuracy.
Thus, if for example for a given combination of features 80% of students pass and only 20% fail, well, then the Bayesian classifier will predict that the students with that combination of features will pass the course, as that is more likely.
This minimal possible error rate of the Bayesian classifier is called irreducible error and all classifiers exhibit it. Other classifiers besides this type of error also exhibit reducible error which can be described as resulting from good, but not perfect estimates of those probabilities. Given that most of the times we don't have this perfect information, the idea of different classification models is to make different assumption which will be good enough to produce high enough classification accuracy while not requiring to collect the data about the whole population.
Read the full explanation in the "Introduction to statistical learning" by Hastie and Tibshiani http://www-bcf.usc.edu/~gareth/ISL/ (Page 38)
I figured out what is needed to be done. Actually, it's something simple, but its seems I had a matlaboid bug... Here is the code and the resulting figure for the "XOR" binary classification problem.
gamma = getGamma();
b = getB();
points_x1 = linspace(xLimits(1), xLimits(2), 100);
points_x2 = linspace(yLimits(1), yLimits(2), 100);
[X1, X2] = meshgrid(points_x1, points_x2);
% Initialize f
f = ones(length(points_x1), length(points_x2))*rho;
% Iter. all SVs
for i=1:N_sv
alpha_i = getAlpha(i);
sv_i = getSV(i);
for j=1:length(points_x1)
for k=1:length(points_x2)
x = [points_x1(j);points_x2(k)];
f(j,k) = f(j,k) + alpha_i*y_i*kernel_func(gamma, x, sv_i);
end
end
end
surf(X1,X2,f);
shading interp;
lighting phong;
alpha(.6)
contourf(X1, X2, f, 1);
where the function
function k = kernel_func(gamma, x, x_i)
k = exp(-gamma*norm(x - x_i)^2);
end
just produces the kernel function (RBF kernel), $k(\mathbf{x},\mathbf{x}')=\operatorname{exp}\left(-\gamma\lVert\mathbf{x}-\mathbf{x}'\rVert^2\right)$.
Here is the result for the XOR problem. Here $\gamma=4$.
Best Answer
Logistic regression does not have decision boundaries. It is a method to estimate probabilities of events/class membership. Decisions are made in a separate step once you know the estimated risk along with utilities/costs/loss function, which is the way optimum decisions are made.